<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hmmm… I was going to ask what structures are actually shared by "two" pools that are in different file systems, and you provided the answer before I asked.<div class=""><br class=""></div><div class="">So all disks which are labelled with a particular storage pool name share some metadata: pool id, the name, possibly other items. I was confused because the NSD is labelled with the pool when it’s added to the file system — not when it’s created. So I thought that the pool was a property of a disk+fs, not the NSD itself.</div><div class=""><br class=""></div><div class="">The more I talk this out the more I think that pools aren’t real, but just another label that happens to be orthogonal to all of the other labels:</div><div class=""><br class=""></div><div class=""><ul class="MailOutline"><li class="">Only disks have pools — NSDs do not, because there is no way to give them one at creation time.</li><li class="">Disks are NSDs that are in file systems.</li><li class="">A disk is in exactly one file system.</li><li class="">All disks that have the same "pool name" will have the same "pool id", and possibly other pool-related metadata.</li></ul><div class=""><br class=""></div></div><div class="">It appears that the disks in a pool have absolutely nothing in common other than that they have been labelled as being in the same pool when added to a file system, right? I mean, literally everything but the pool name/id could be different — or is there more there?</div><div class=""><br class=""></div><div class="">Do we do <i class="">anything</i> to pools outside of the context of a file system? Even when we list them we have to provide a file system. Does GPFS keep statistics about pools that aren’t related to file systems?</div><div class=""><div class=""><br class=""></div><div class="">(I love learning things, even when I look like an idiot…)<div class=""><div class=""><br class=""><div class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">-- <br class="">Stephen<br class=""><br class=""><br class=""></div>
</div>
<div><br class=""><blockquote type="cite" class=""><div class="">On Mar 27, 2019, at 11:20 AM, J. Eric Wonderley <<a href="mailto:eric.wonderley@vt.edu" class="">eric.wonderley@vt.edu</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div dir="ltr" class="">mmlspool might suggest there's only 1 system pool per cluster. We have 2 clusters and it has id=0 on both.<div class=""><br class=""></div><div class="">One of our clusters has 2 filesystems that have same id for two different dataonly pools:<br class=""><div class=""><div class="">[root@cl001 ~]# mmlspool home all</div><div class="">Name Id</div><div class="">system 0 </div><div class="">fc_8T 65537 </div><div class="">fc_ssd400G 65538 </div><div class="">[root@cl001 ~]# mmlspool work all</div><div class="">Name Id</div><div class="">system 0 </div><div class="">sas_6T 65537 </div></div><div class=""><br class=""></div><div class="">I know md lives in the system pool and if you do encryption you can forget about putting data into you inodes for small files</div><div class=""><br class=""></div><div class=""><br class=""></div></div></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Mar 27, 2019 at 10:57 AM Stephen Ulmer <<a href="mailto:ulmer@ulmer.org" class="">ulmer@ulmer.org</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" class=""><div class="">This presentation contains lots of good information about file system structure in general, and GPFS in specific, and I appreciate that and enjoyed reading it.</div><div class=""><br class=""></div>However, it states outright (both graphically and in text) that storage pools are a feature of the cluster, not of a file system — which I believe to be completely incorrect. For example, it states that there is "only one system pool per cluster", rather than one per file system.<div class=""><br class=""></div><div class="">Given that this was written by IBMers and presented at an actual users’ group, can someone please weigh in on this? I’m asking because it represents a fundamental misunderstanding of a very basic GPFS concept, which makes me wonder how authoritative the rest of it is...</div><div class=""><div class=""><br class=""><div class="">
<div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class="">-- <br class="">Stephen<br class=""><br class=""><br class=""></div>
</div>
<div class=""><br class=""><blockquote type="cite" class=""><div class="">On Mar 26, 2019, at 12:27 PM, Dorigo Alvise (PSI) <<a href="mailto:alvise.dorigo@psi.ch" target="_blank" class="">alvise.dorigo@psi.ch</a>> wrote:</div><br class="gmail-m_871237091420363041Apple-interchange-newline"><div class=""><div style="font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;direction:ltr;font-family:Tahoma;font-size:10pt" class=""><div class="">Hi Marc,<br class=""></div><div class="">"Indirect block size" is well explained in this presentation:<span class="gmail-m_871237091420363041Apple-converted-space"> </span><br class=""></div><div class=""><br class=""></div><div class=""><a href="http://files.gpfsug.org/presentations/2016/south-bank/D2_P2_A_spectrum_scale_metadata_dark_V2a.pdf" target="_blank" class="">http://files.gpfsug.org/presentations/2016/south-bank/D2_P2_A_spectrum_scale_metadata_dark_V2a.pdf</a></div><div class=""><br class=""></div><div class="">pages 37-41<br class=""></div><div class=""><br class=""></div><div class="">Cheers,</div><div class=""><br class=""></div><div class=""> Alvise<br class=""></div><div class=""><br class=""></div><div style="font-family:"Times New Roman";font-size:16px" class=""><hr class=""><div id="gmail-m_871237091420363041divRpF933280" style="direction:ltr" class=""><font size="2" face="Tahoma" class=""><b class="">From:</b><span class="gmail-m_871237091420363041Apple-converted-space"> </span><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank" class="">gpfsug-discuss-bounces@spectrumscale.org</a><span class="gmail-m_871237091420363041Apple-converted-space"> </span>[<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank" class="">gpfsug-discuss-bounces@spectrumscale.org</a>] on behalf of Caubet Serrabou Marc (PSI) [<a href="mailto:marc.caubet@psi.ch" target="_blank" class="">marc.caubet@psi.ch</a>]<br class=""><b class="">Sent:</b><span class="gmail-m_871237091420363041Apple-converted-space"> </span>Tuesday, March 26, 2019 4:39 PM<br class=""><b class="">To:</b><span class="gmail-m_871237091420363041Apple-converted-space"> </span>gpfsug main discussion list<br class=""><b class="">Subject:</b><span class="gmail-m_871237091420363041Apple-converted-space"> </span>[gpfsug-discuss] GPFS v5: Blocksizes and subblocks<br class=""></font><br class=""></div><div class=""></div><div class=""><div style="direction:ltr;font-family:Tahoma;font-size:10pt" class=""><div class="">Hi all,</div><div class=""><br class=""></div><div class="">according to several GPFS presentations as well as according to the man pages:</div><div class=""><br class=""></div><div class=""><font face="Courier New" class=""> Table 1. Block sizes and subblock sizes<br class=""><br class="">+-------------------------------+-------------------------------+<br class="">| Block size | Subblock size |<br class="">+-------------------------------+-------------------------------+<br class="">| 64 KiB | 2 KiB |<br class="">+-------------------------------+-------------------------------+<br class="">| 128 KiB | 4 KiB |<br class="">+-------------------------------+-------------------------------+<br class="">| 256 KiB, 512 KiB, 1 MiB, 2 | 8 KiB |<br class="">| MiB, 4 MiB | |<br class="">+-------------------------------+-------------------------------+<br class="">| 8 MiB, 16 MiB | 16 KiB |<br class="">+-------------------------------+-------------------------------+</font><br class=""><br class=""></div><div class="">A block size of 8MiB or 16MiB should contain subblocks of 16KiB.</div><div class=""><br class=""></div><div class="">However, when creating a new filesystem with 16MiB blocksize, looks like is using 128KiB subblocks:</div><div class=""><br class=""></div><div class=""><font face="Courier New" class="">[root@merlindssio01 ~]# mmlsfs merlin<br class="">flag value description<br class="">------------------- ------------------------ -----------------------------------<br class=""> -f 8192 Minimum fragment (subblock) size in bytes (system pool)<br class=""> 131072 Minimum fragment (subblock) size in bytes (other pools)<br class=""> -i 4096 Inode size in bytes<br class=""> -I 32768 Indirect block size in bytes<br class="">.</font></div><div class=""><font face="Courier New" class="">.</font></div><div class=""><font face="Courier New" class="">.<br class=""></font></div><div class=""><font face="Courier New" class=""> -n 128 Estimated number of nodes that will mount file system<br class=""> -B 1048576 Block size (system pool)<br class=""> 16777216 Block size (other pools)<br class="">.</font></div><div class=""><font face="Courier New" class="">.</font></div><div class=""><font face="Courier New" class="">.</font><br class=""></div><div class=""><br class=""></div><div class="">What am I missing? According to documentation, I expect this to be a fixed value, or it isn't at all?<br class=""></div><div class=""><br class=""></div><div class="">On the other hand, I don't really understand the concept 'Indirect block size in bytes', can somebody clarify or provide some details about this setting?</div><div class=""><br class=""></div><div class="">Thanks a lot and best regards,</div><div class="">Marc <span class="gmail-m_871237091420363041Apple-converted-space"> </span><br class=""><div style="font-family:Tahoma;font-size:13px" class=""><div style="font-family:Tahoma;font-size:13px" class=""><div class="gmail-m_871237091420363041BodyFragment"><font size="2" class=""><span style="font-size:10pt" class=""><div class="gmail-m_871237091420363041PlainText">_________________________________________<br class="">Paul Scherrer Institut<span class="gmail-m_871237091420363041Apple-converted-space"> </span><br class="">High Performance Computing<br class="">Marc Caubet Serrabou<br class="">Building/Room: WHGA/019A</div><div class="gmail-m_871237091420363041PlainText"><font size="2" class=""><span style="font-size:10pt" class="">Forschungsstrasse, 111</span></font></div><div class="gmail-m_871237091420363041PlainText">5232 Villigen PSI<br class="">Switzerland<br class=""><br class="">Telephone: +41 56 310 46 67<br class="">E-Mail:<span class="gmail-m_871237091420363041Apple-converted-space"> </span><a href="mailto:marc.caubet@psi.ch" target="_blank" class="">marc.caubet@psi.ch</a></div></span></font></div></div></div></div></div></div></div></div><span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;float:none;display:inline" class="">_______________________________________________</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class=""><span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;float:none;display:inline" class="">gpfsug-discuss mailing list</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class=""><span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;float:none;display:inline" class="">gpfsug-discuss at<span class="gmail-m_871237091420363041Apple-converted-space"> </span></span><a href="http://spectrumscale.org/" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank" class="">spectrumscale.org</a><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class=""><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></div></blockquote></div><br class=""></div></div></div>_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="http://spectrumscale.org/" rel="noreferrer" target="_blank" class="">spectrumscale.org</a><br class="">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="">
</blockquote></div>
_______________________________________________<br class="">gpfsug-discuss mailing list<br class="">gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a><br class=""><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class=""></div></blockquote></div><br class=""></div></div></div></div></body></html>