<div dir="ltr">
<span style="color:rgb(34,34,34);font-family:sans-serif;font-size:13px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Thanks Olaf and Sven,</span><div style="color:rgb(34,34,34);font-family:sans-serif;font-size:13px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:sans-serif;font-size:13px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">It looks like a lot of advice from the wiki (<a href="https://www.ibm.com/developerworks/community/wikis/home?lang=en-us#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/Data%20and%20Metadata">https://www.ibm.com/developerworks/community/wikis/home?lang=en-us#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/Data%20and%20Metadata</a>) is no longer relevant for version 5. Any idea if its likely to be updated soon?</div><div style="color:rgb(34,34,34);font-family:sans-serif;font-size:13px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:sans-serif;font-size:13px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">The new subblock changes appear to have removed a lot of reasons for using smaller block sizes. In broad terms there any situations where you would recommend using less than the new default block size? </div><br class="gmail-Apple-interchange-newline">
Cheers,<div><br></div><div>Carl.</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Mon, 2 Jul 2018 at 17:55, Sven Oehme <<a href="mailto:oehmes@gmail.com">oehmes@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Olaf, he is talking about indirect size not subblock size .<div><br></div><div>Carl, </div><div><br></div><div>here is a screen shot of a 4mb filesystem :</div><div><br></div><div><div>[root@p8n15hyp ~]# mmlsfs all_local</div><div><br></div><div>File system attributes for /dev/fs2-4m-07:</div><div>==========================================</div><div>flag value description</div><div>------------------- ------------------------ -----------------------------------</div><div> -f 8192 Minimum fragment (subblock) size in bytes</div><div> -i 4096 Inode size in bytes</div><div> -I 32768 Indirect block size in bytes</div><div> -m 1 Default number of metadata replicas</div><div> -M 2 Maximum number of metadata replicas</div><div> -r 1 Default number of data replicas</div><div> -R 2 Maximum number of data replicas</div><div> -j scatter Block allocation type</div><div> -D nfs4 File locking semantics in effect</div><div> -k all ACL semantics in effect</div><div> -n 512 Estimated number of nodes that will mount file system</div><div> -B 4194304 Block size</div><div> -Q none Quotas accounting enabled</div><div> none Quotas enforced</div><div> none Default quotas enabled</div><div> --perfileset-quota No Per-fileset quota enforcement</div><div> --filesetdf No Fileset df enabled?</div><div> -V 19.01 (5.0.1.0) File system version</div><div> --create-time Mon Jun 18 12:30:54 2018 File system creation time</div><div> -z No Is DMAPI enabled?</div><div> -L 33554432 Logfile size</div><div> -E Yes Exact mtime mount option</div><div> -S relatime Suppress atime mount option</div><div> -K whenpossible Strict replica allocation option</div><div> --fastea Yes Fast external attributes enabled?</div><div> --encryption No Encryption enabled?</div><div> --inode-limit 4000000000 Maximum number of inodes</div><div> --log-replicas 0 Number of log replicas</div><div> --is4KAligned Yes is4KAligned?</div><div> --rapid-repair Yes rapidRepair enabled?</div><div> --write-cache-threshold 0 HAWC Threshold (max 65536)</div><div> --subblocks-per-full-block 512 Number of subblocks per full block</div><div> -P system Disk storage pools in file system</div><div> --file-audit-log No File Audit Logging enabled?</div><div> --maintenance-mode No Maintenance Mode enabled?</div><div> -d RG001VS001;RG002VS001;RG003VS002;RG004VS002 Disks in file system</div><div> -A no Automatic mount option</div><div> -o none Additional mount options</div><div> -T /gpfs/fs2-4m-07 Default mount point</div><div> --mount-priority 0 Mount priority</div></div><div><br></div><div>as you can see indirect size is 32k</div><div><br></div><div>sven</div></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Jul 2, 2018 at 9:46 AM Olaf Weiser <<a href="mailto:olaf.weiser@de.ibm.com" target="_blank">olaf.weiser@de.ibm.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><font size="2" face="sans-serif">HI Carl, </font><br><font size="2" face="sans-serif">8k for 4 M Blocksize</font><br><font size="2" face="sans-serif">files < ~3,x KB fits into the inode
, for "larger" files (> 3,x KB) at least one
"subblock" be allocated .. </font><br><br><font size="2" face="sans-serif">in R < 5.x ... it was fixed 1/32
from blocksize so subblocksize is retrieved from the blocksize ...
</font><br><font size="2" face="sans-serif">since R >5 (so new created file systems)
.. the new default block size is 4 MB, fragment size is 8k (512 subblocks)
</font><br><font size="2" face="sans-serif">for even larger block sizes ... more
subblocks are available per block </font><br><font size="2" face="sans-serif">so e.g. </font><br><font size="2" face="sans-serif">8M .... 1024 subblocks (fragment size
is 8 k again)</font><br><br><font size="2" face="sans-serif">@Sven.. correct me, if I'm wrong ...
</font><br><br><br><div><br><br><br><br><font size="1" color="#5f5f5f" face="sans-serif">From:
</font><font size="1" face="sans-serif">Carl <<a href="mailto:mutantllama@gmail.com" target="_blank">mutantllama@gmail.com</a>></font></div><div><br><font size="1" color="#5f5f5f" face="sans-serif">To:
</font><font size="1" face="sans-serif">gpfsug main discussion
list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a>></font><br></div><div><font size="1" color="#5f5f5f" face="sans-serif">Date:
</font><font size="1" face="sans-serif">07/02/2018 08:55 AM</font><br><font size="1" color="#5f5f5f" face="sans-serif">Subject:
</font><font size="1" face="sans-serif">Re: [gpfsug-discuss]
subblock sanity check in 5.0</font><br><font size="1" color="#5f5f5f" face="sans-serif">Sent by:
</font><font size="1" face="sans-serif"><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a></font><br><hr noshade><br><br></div><div><br><font size="3">Hi Sven,</font><br><br><font size="3">What is the resulting indirect-block size with a
4mb metadata block size? </font><br><br><font size="3">Does the new sub-block magic mean that it will take up
32k, or will it occupy 128k?</font><br><br><font size="3">Cheers,</font><br><br><font size="3">Carl.</font><br><br><br><font size="3">On Mon, 2 Jul 2018 at 15:26, Sven Oehme <</font><a href="mailto:oehmes@gmail.com" target="_blank"><font size="3" color="blue"><u>oehmes@gmail.com</u></font></a><font size="3">>
wrote:</font><br><font size="3">Hi,</font><br><br><font size="3" color="#2f2f2f">most traditional raid controllers can't
deal well with blocksizes above 4m, which is why the new default is 4m
and i would leave it at that unless you know for sure you get better performance
with 8mb which typically requires your raid controller volume full block
size to be 8mb with maybe a 8+2p @1mb strip size (many people confuse strip
size with full track size) .</font><br><font size="3" color="#2f2f2f">if you don't have dedicated SSDs for metadata
i would recommend to just use a 4mb blocksize with mixed data and metadata
disks, if you have a reasonable number of SSD's put them in a raid 1 or
raid 10 and use them as dedicated metadata and the other disks as dataonly
, but i would not use the --metadata-block-size parameter as it prevents
the datapool to use large number of subblocks.</font><br><font size="3" color="#2f2f2f">as long as your SSDs are on raid 1 or 10
there is no read/modify/write penalty, so using them with the 4mb blocksize
has no real negative impact at least on controllers i have worked with.</font><br><br><font size="3" color="#2f2f2f">hope this helps.</font><br><br><font size="3">On Tue, Jun 26, 2018 at 5:18 PM Joseph Mendoza <</font><a href="mailto:jam@ucar.edu" target="_blank"><font size="3" color="blue"><u>jam@ucar.edu</u></font></a><font size="3">>
wrote:</font><br><font size="3">Hi, it's for a traditional NSD setup.</font><p><font size="3">--Joey</font><p><br><font size="3">On 6/26/18 12:21 AM, Sven Oehme wrote:</font><br><font size="3">Joseph, </font><br><br><font size="3">the subblocksize will be derived from the smallest blocksize
in the filesytem, given you specified a metadata block size of 512k thats
what will be used to calculate the number of subblocks, even your data
pool is 4mb. </font><br><font size="3">is this setup for a traditional NSD Setup or for GNR as
the recommendations would be different. </font><br><br><font size="3">sven<br></font><br><font size="3">On Tue, Jun 26, 2018 at 2:59 AM Joseph Mendoza <</font><a href="mailto:jam@ucar.edu" target="_blank"><font size="3" color="blue"><u>jam@ucar.edu</u></font></a><font size="3">>
wrote:</font><br><font size="3">Quick question, anyone know why GPFS wouldn't respect
the default for<br>the subblocks-per-full-block parameter when creating a new filesystem? <br>I'd expect it to be set to 512 for an 8MB block size but my guess is<br>that also specifying a metadata-block-size is interfering with it (by<br>being too small). This was a parameter recommended by the vendor
for a<br>4.2 installation with metadata on dedicated SSDs in the system pool, any<br>best practices for 5.0? I'm guessing I'd have to bump it up to at
least<br>4MB to get 512 subblocks for both pools.<br><br>fs1 created with:<br># mmcrfs fs1 -F fs1_ALL -A no -B 8M -i 4096 -m 2 -M 2 -r 1 -R 2 -j<br>cluster -n 9000 --metadata-block-size 512K --perfileset-quota<br>--filesetdf -S relatime -Q yes --inode-limit 20000000:10000000 -T /gpfs/fs1<br><br># mmlsfs fs1<br><snipped><br><br>flag
value
description<br>------------------- ------------------------<br>-----------------------------------<br> -f
8192
Minimum fragment (subblock)<br>size in bytes (system pool)<br>
131072
Minimum fragment (subblock)<br>size in bytes (other pools)<br> -i
4096
Inode size in bytes<br> -I
32768
Indirect block size in bytes<br><br> -B
524288
Block size (system pool)<br>
8388608
Block size (other pools)<br><br> -V
19.01 (5.0.1.0) File
system version<br><br> --subblocks-per-full-block 64
Number of subblocks per<br>full block<br> -P
system;DATA
Disk storage pools in file<br>system<br><br><br>Thanks!<br>--Joey Mendoza<br>NCAR<br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="http://spectrumscale.org" target="_blank"><font size="3" color="blue"><u>spectrumscale.org</u></font></a><font size="3" color="blue"><u><br></u></font><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><font size="3" color="blue"><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></a><br><font size="3"><br></font><br><tt><font size="3">_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font></tt><a href="http://spectrumscale.org" target="_blank"><tt><font size="3" color="blue"><u>spectrumscale.org</u></font></tt></a><tt><font size="3"><br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><tt><font size="3" color="blue"><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></tt></a><tt><font size="3"><br></font></tt><br><br><font size="3">_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="http://spectrumscale.org" target="_blank"><font size="3" color="blue"><u>spectrumscale.org</u></font></a><font size="3" color="blue"><u><br></u></font><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><font size="3" color="blue"><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></a><tt><font size="2">_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><tt><font size="2">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size="2"><br></font></tt><br><br></p></p></div><br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
</blockquote></div>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
</blockquote></div>