<font size=2 face="sans-serif">Hi,</font><br><br><font size=2 face="sans-serif">Not sure how will it work over the mailing
list...</font><br><font size=2 face="sans-serif">Since its a popular question, I've prepared
a slide explaining all of that - ( pasted/attached below, but I'll try
to explain in text as well...).</font><br><br><font size=2 face="sans-serif">On the right we can see the various
"layers":</font><br><font size=2 face="sans-serif">- OS disks ( what looks to the OS/GPFS
as a physical disk) - its properties are size, media, device name etc.
( we actually won't know what media means, but we don't really care)</font><br><font size=2 face="sans-serif">- NSD: When introduced to GPFS, so later
on we can use it for "something". Two interesting properties
at this stage: name and through which servers we can get to it...</font><br><font size=2 face="sans-serif">- FS disk: When NSD is being added to
a filesystem, then we start caring about stuff like type ( data, metadata,
data+metadata, desconly etc.), to what pool we add the disk, what failure
groups etc.</font><br><br><font size=2 face="sans-serif">That's true on a per filesystem basis.
With the exception that nsd name must be unique across the cluster. All
the rest is in a filesystem context. So:</font><br><font size=2 face="sans-serif">- Each filesystem will have its own
"system pool" which will store that filesystem metadata ( can
also store data - which of course belong to that filesystem..not others...not
the cluster)</font><br><font size=2 face="sans-serif">- Pool exist just because several filesystem
disks were told that they belong to that pool ( and hopefully there is
some policy that brings data to that pool). And since filesystem disks,
exist only in the context of their filesystem - a pool exist inside a single
filesystem only ( other filesystems might have their own pools of course).</font><br><br><br><img src=cid:_1_E26AD1E4E26ACD480059AF66C22583CA style="border:0px solid;"><br><font size=2 face="sans-serif"><br>Regards,<br><br>Tomer Perry<br>Scalable I/O Development (Spectrum Scale)<br>email: tomp@il.ibm.com<br>1 Azrieli Center, Tel Aviv 67021, Israel<br>Global Tel: +1 720 3422758<br>Israel Tel: +972 3 9188625<br>Mobile: +972 52 2554625<br></font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Stephen Ulmer <ulmer@ulmer.org></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug main discussion
list <gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">27/03/2019 17:53</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
GPFS v5: Blocksizes and subblocks</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><font size=3>Hmmm… I was going to ask what structures are actually
shared by "two" pools that are in different file systems, and
you provided the answer before I asked.</font><br><br><font size=3>So all disks which are labelled with a particular storage
pool name share some metadata: pool id, the name, possibly other items.
I was confused because the NSD is labelled with the pool when it’s added
to the file system — not when it’s created. So I thought that the pool
was a property of a disk+fs, not the NSD itself.</font><br><br><font size=3>The more I talk this out the more I think that pools aren’t
real, but just another label that happens to be orthogonal to all of the
other labels:</font><br><ul><li><font size=3>Only disks have pools — NSDs do not, because there is
no way to give them one at creation time.</font><li><font size=3>Disks are NSDs that are in file systems.</font><li><font size=3>A disk is in exactly one file system.</font><li><font size=3>All disks that have the same "pool name" will
have the same "pool id", and possibly other pool-related metadata.</font></ul><br><font size=3>It appears that the disks in a pool have absolutely nothing
in common other than that they have been labelled as being in the same
pool when added to a file system, right? I mean, literally everything but
the pool name/id could be different — or is there more there?</font><br><br><font size=3>Do we do <i>anything</i> to pools outside of the context
of a file system? Even when we list them we have to provide a file system.
Does GPFS keep statistics about pools that aren’t related to file systems?</font><br><br><font size=3>(I love learning things, even when I look like an idiot…)</font><br><br><font size=3>-- <br>Stephen<br><br></font><br><br><font size=3>On Mar 27, 2019, at 11:20 AM, J. Eric Wonderley <</font><a href="mailto:eric.wonderley@vt.edu"><font size=3 color=blue><u>eric.wonderley@vt.edu</u></font></a><font size=3>>
wrote:</font><br><br><font size=3>mmlspool might suggest there's only 1 system pool per
cluster. We have 2 clusters and it has id=0 on both.</font><br><br><font size=3>One of our clusters has 2 filesystems that have same id
for two different dataonly pools:</font><br><font size=3>[root@cl001 ~]# mmlspool home all</font><br><font size=3>Name Id</font><br><font size=3>system 0
</font><br><font size=3>fc_8T 65537
</font><br><font size=3>fc_ssd400G 65538 </font><br><font size=3>[root@cl001 ~]# mmlspool work all</font><br><font size=3>Name Id</font><br><font size=3>system 0
</font><br><font size=3>sas_6T 65537
</font><br><br><font size=3>I know md lives in the system pool and if you do encryption
you can forget about putting data into you inodes for small files</font><br><br><br><br><font size=3>On Wed, Mar 27, 2019 at 10:57 AM Stephen Ulmer <</font><a href="mailto:ulmer@ulmer.org"><font size=3 color=blue><u>ulmer@ulmer.org</u></font></a><font size=3>>
wrote:</font><br><font size=3>This presentation contains lots of good information about
file system structure in general, and GPFS in specific, and I appreciate
that and enjoyed reading it.</font><br><br><font size=3>However, it states outright (both graphically and in text)
that storage pools are a feature of the cluster, not of a file system —
which I believe to be completely incorrect. For example, it states that
there is "only one system pool per cluster", rather than one
per file system.</font><br><br><font size=3>Given that this was written by IBMers and presented at
an actual users’ group, can someone please weigh in on this? I’m asking
because it represents a fundamental misunderstanding of a very basic GPFS
concept, which makes me wonder how authoritative the rest of it is...</font><br><br><font size=3>-- <br>Stephen<br><br></font><br><br><font size=3>On Mar 26, 2019, at 12:27 PM, Dorigo Alvise (PSI) <</font><a href="mailto:alvise.dorigo@psi.ch" target="_blank"><font size=3 color=blue><u>alvise.dorigo@psi.ch</u></font></a><font size=3>>
wrote:</font><br><br><font size=2 face="Tahoma">Hi Marc,</font><br><font size=2 face="Tahoma">"Indirect block size" is well
explained in this presentation: </font><br><br><a href="http://files.gpfsug.org/presentations/2016/south-bank/D2_P2_A_spectrum_scale_metadata_dark_V2a.pdf" target="_blank"><font size=2 color=blue face="Tahoma"><u>http://files.gpfsug.org/presentations/2016/south-bank/D2_P2_A_spectrum_scale_metadata_dark_V2a.pdf</u></font></a><br><br><font size=2 face="Tahoma">pages 37-41</font><br><br><font size=2 face="Tahoma">Cheers,</font><br><br><font size=2 face="Tahoma"> Alvise</font><br><br><hr><br><font size=2 face="Tahoma"><b>From:</b> </font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank"><font size=2 color=blue face="Tahoma"><u>gpfsug-discuss-bounces@spectrumscale.org</u></font></a><font size=2 face="Tahoma">[</font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank"><font size=2 color=blue face="Tahoma"><u>gpfsug-discuss-bounces@spectrumscale.org</u></font></a><font size=2 face="Tahoma">]
on behalf of Caubet Serrabou Marc (PSI) [</font><a href="mailto:marc.caubet@psi.ch" target="_blank"><font size=2 color=blue face="Tahoma"><u>marc.caubet@psi.ch</u></font></a><font size=2 face="Tahoma">]<b><br>Sent:</b> Tuesday, March 26, 2019 4:39 PM<b><br>To:</b> gpfsug main discussion list<b><br>Subject:</b> [gpfsug-discuss] GPFS v5: Blocksizes and subblocks</font><font size=3 face="Times New Roman"><br></font><br><font size=2 face="Tahoma">Hi all,</font><br><br><font size=2 face="Tahoma">according to several GPFS presentations
as well as according to the man pages:</font><br><br><font size=2 face="Courier New"> Table
1. Block sizes and subblock sizes<br><br>+-------------------------------+-------------------------------+<br>| Block size
| Subblock size
|<br>+-------------------------------+-------------------------------+<br>| 64 KiB
| 2 KiB
|<br>+-------------------------------+-------------------------------+<br>| 128 KiB
| 4 KiB
|<br>+-------------------------------+-------------------------------+<br>| 256 KiB, 512 KiB, 1 MiB, 2 | 8 KiB
|<br>| MiB, 4 MiB
|
|<br>+-------------------------------+-------------------------------+<br>| 8 MiB, 16 MiB
| 16 KiB
|<br>+-------------------------------+-------------------------------+</font><font size=2 face="Tahoma"><br></font><br><font size=2 face="Tahoma">A block size of 8MiB or 16MiB should contain
subblocks of 16KiB.</font><br><br><font size=2 face="Tahoma">However, when creating a new filesystem
with 16MiB blocksize, looks like is using 128KiB subblocks:</font><br><br><font size=2 face="Courier New">[root@merlindssio01 ~]# mmlsfs merlin<br>flag value
description<br>------------------- ------------------------ -----------------------------------<br> -f 8192
Minimum
fragment (subblock) size in bytes (system pool)<br> 131072
Minimum
fragment (subblock) size in bytes (other pools)<br> -i 4096
Inode size
in bytes<br> -I 32768
Indirect
block size in bytes<br>.</font><br><font size=2 face="Courier New">.</font><br><font size=2 face="Courier New">.</font><br><font size=2 face="Courier New"> -n
128
Estimated number of nodes that will mount
file system<br> -B 1048576
Block size (system
pool)<br> 16777216
Block size (other
pools)<br>.</font><br><font size=2 face="Courier New">.</font><br><font size=2 face="Courier New">.</font><br><br><font size=2 face="Tahoma">What am I missing? According to documentation,
I expect this to be a fixed value, or it isn't at all?</font><br><br><font size=2 face="Tahoma">On the other hand, I don't really understand
the concept 'Indirect block size in bytes', can somebody clarify or provide
some details about this setting?</font><br><br><font size=2 face="Tahoma">Thanks a lot and best regards,</font><br><font size=2 face="Tahoma">Marc
</font><br><font size=2 face="Tahoma">_________________________________________<br>Paul Scherrer Institut <br>High Performance Computing<br>Marc Caubet Serrabou<br>Building/Room: WHGA/019A</font><br><font size=2 face="Tahoma">Forschungsstrasse, 111</font><br><font size=2 face="Tahoma">5232 Villigen PSI<br>Switzerland<br><br>Telephone: +41 56 310 46 67<br>E-Mail: </font><a href="mailto:marc.caubet@psi.ch" target="_blank"><font size=2 color=blue face="Tahoma"><u>marc.caubet@psi.ch</u></font></a><br><font size=1>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="http://spectrumscale.org/" target="_blank"><font size=1 color=blue><u>spectrumscale.org</u></font></a><font size=1 color=blue><u><br></u></font><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><font size=1 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></a><br><br><font size=3>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="http://spectrumscale.org/" target="_blank"><font size=3 color=blue><u>spectrumscale.org</u></font></a><font size=3 color=blue><u><br></u></font><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><font size=3 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></a><br><font size=3>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="http://spectrumscale.org"><font size=3 color=blue><u>spectrumscale.org</u></font></a><font size=3 color=blue><u><br></u></font><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><font size=3 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></a><br><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br></font></tt><br><br><BR>