<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi All,
<div class=""><br class="">
</div>
<div class="">Our production cluster is still on GPFS 4.2.3.x, but in preparation for moving to GPFS 5 I have upgraded our small (7 node) test cluster to GPFS 5.0.1-1. I am setting up a new filesystem there using hardware that we recently life-cycled out of
our production environment.</div>
<div class=""><br class="">
</div>
<div class="">I “successfully” created a filesystem but I believe the sub-block size is wrong. I’m using a 4 MB filesystem block size, so according to the mmcrfs man page the sub-block size should be 8K:</div>
<div class=""><br class="">
</div>
<div class="">
<div class=""> Table 1. Block sizes and subblock sizes</div>
<div class=""><br class="">
</div>
<div class="">+‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+</div>
<div class="">| Block size | Subblock size |</div>
<div class="">+‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+</div>
<div class="">| 64 KiB | 2 KiB |</div>
<div class="">+‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+</div>
<div class="">| 128 KiB | 4 KiB |</div>
<div class="">+‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+</div>
<div class="">| 256 KiB, 512 KiB, 1 MiB, 2 | 8 KiB |</div>
<div class="">| MiB, 4 MiB | |</div>
<div class="">+‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+</div>
<div class="">| 8 MiB, 16 MiB | 16 KiB |</div>
<div class="">+‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+</div>
<div class=""><br class="">
</div>
<div class="">However, it appears that it’s 8K for the system pool but 32K for the other pools:</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">flag value description</div>
<div class="">------------------- ------------------------ -----------------------------------</div>
<div class=""> -f 8192 Minimum fragment (subblock) size in bytes (system pool)</div>
<div class=""> 32768 Minimum fragment (subblock) size in bytes (other pools)</div>
<div class=""> -i 4096 Inode size in bytes</div>
<div class=""> -I 32768 Indirect block size in bytes</div>
<div class=""> -m 2 Default number of metadata replicas</div>
<div class=""> -M 3 Maximum number of metadata replicas</div>
<div class=""> -r 1 Default number of data replicas</div>
<div class=""> -R 3 Maximum number of data replicas</div>
<div class=""> -j scatter Block allocation type</div>
<div class=""> -D nfs4 File locking semantics in effect</div>
<div class=""> -k all ACL semantics in effect</div>
<div class=""> -n 32 Estimated number of nodes that will mount file system</div>
<div class=""> -B 1048576 Block size (system pool)</div>
<div class=""> 4194304 Block size (other pools)</div>
<div class=""> -Q user;group;fileset Quotas accounting enabled</div>
<div class=""> user;group;fileset Quotas enforced</div>
<div class=""> none Default quotas enabled</div>
<div class=""> --perfileset-quota No Per-fileset quota enforcement</div>
<div class=""> --filesetdf No Fileset df enabled?</div>
<div class=""> -V 19.01 (5.0.1.0) File system version</div>
<div class=""> --create-time Wed Aug 1 11:39:39 2018 File system creation time</div>
<div class=""> -z No Is DMAPI enabled?</div>
<div class=""> -L 33554432 Logfile size</div>
<div class=""> -E Yes Exact mtime mount option</div>
<div class=""> -S relatime Suppress atime mount option</div>
<div class=""> -K whenpossible Strict replica allocation option</div>
<div class=""> --fastea Yes Fast external attributes enabled?</div>
<div class=""> --encryption No Encryption enabled?</div>
<div class=""> --inode-limit 101095424 Maximum number of inodes</div>
<div class=""> --log-replicas 0 Number of log replicas</div>
<div class=""> --is4KAligned Yes is4KAligned?</div>
<div class=""> --rapid-repair Yes rapidRepair enabled?</div>
<div class=""> --write-cache-threshold 0 HAWC Threshold (max 65536)</div>
<div class=""> --subblocks-per-full-block 128 Number of subblocks per full block</div>
<div class=""> -P system;raid1;raid6 Disk storage pools in file system</div>
<div class=""> --file-audit-log No File Audit Logging enabled?</div>
<div class=""> --maintenance-mode No Maintenance Mode enabled?</div>
<div class=""> -d test21A3nsd;test21A4nsd;test21B3nsd;test21B4nsd;test23Ansd;test23Bnsd;test23Cnsd;test24Ansd;test24Bnsd;test24Cnsd;test25Ansd;test25Bnsd;test25Cnsd Disks in file system</div>
<div class=""> -A yes Automatic mount option</div>
<div class=""> -o none Additional mount options</div>
<div class=""> -T /gpfs5 Default mount point</div>
<div class=""> --mount-priority 0 Mount priority</div>
</div>
<div class=""><br class="">
</div>
<div class="">Output of mmcrfs:</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">mmcrfs gpfs5 -F ~/gpfs/gpfs5.stanza -A yes -B 4M -E yes -i 4096 -j scatter -k all -K whenpossible -m 2 -M 3 -n 32 -Q yes -r 1 -R 3 -T /gpfs5 -v yes --nofilesetdf --metadata-block-size 1M</div>
<div class=""><br class="">
</div>
<div class="">The following disks of gpfs5 will be formatted on node testnsd3:</div>
<div class=""> test21A3nsd: size 953609 MB</div>
<div class=""> test21A4nsd: size 953609 MB</div>
<div class=""> test21B3nsd: size 953609 MB</div>
<div class=""> test21B4nsd: size 953609 MB</div>
<div class=""> test23Ansd: size 15259744 MB</div>
<div class=""> test23Bnsd: size 15259744 MB</div>
<div class=""> test23Cnsd: size 1907468 MB</div>
<div class=""> test24Ansd: size 15259744 MB</div>
<div class=""> test24Bnsd: size 15259744 MB</div>
<div class=""> test24Cnsd: size 1907468 MB</div>
<div class=""> test25Ansd: size 15259744 MB</div>
<div class=""> test25Bnsd: size 15259744 MB</div>
<div class=""> test25Cnsd: size 1907468 MB</div>
<div class="">Formatting file system ...</div>
<div class="">Disks up to size 8.29 TB can be added to storage pool system.</div>
<div class="">Disks up to size 16.60 TB can be added to storage pool raid1.</div>
<div class="">Disks up to size 132.62 TB can be added to storage pool raid6.</div>
<div class="">Creating Inode File</div>
<div class=""> 8 % complete on Wed Aug 1 11:39:19 2018</div>
<div class=""> 18 % complete on Wed Aug 1 11:39:24 2018</div>
<div class=""> 27 % complete on Wed Aug 1 11:39:29 2018</div>
<div class=""> 37 % complete on Wed Aug 1 11:39:34 2018</div>
<div class=""> 48 % complete on Wed Aug 1 11:39:39 2018</div>
<div class=""> 60 % complete on Wed Aug 1 11:39:44 2018</div>
<div class=""> 72 % complete on Wed Aug 1 11:39:49 2018</div>
<div class=""> 83 % complete on Wed Aug 1 11:39:54 2018</div>
<div class=""> 95 % complete on Wed Aug 1 11:39:59 2018</div>
<div class=""> 100 % complete on Wed Aug 1 11:40:01 2018</div>
<div class="">Creating Allocation Maps</div>
<div class="">Creating Log Files</div>
<div class=""> 3 % complete on Wed Aug 1 11:40:07 2018</div>
<div class=""> 28 % complete on Wed Aug 1 11:40:14 2018</div>
<div class=""> 53 % complete on Wed Aug 1 11:40:19 2018</div>
<div class=""> 78 % complete on Wed Aug 1 11:40:24 2018</div>
<div class=""> 100 % complete on Wed Aug 1 11:40:25 2018</div>
<div class="">Clearing Inode Allocation Map</div>
<div class="">Clearing Block Allocation Map</div>
<div class="">Formatting Allocation Map for storage pool system</div>
<div class=""> 85 % complete on Wed Aug 1 11:40:32 2018</div>
<div class=""> 100 % complete on Wed Aug 1 11:40:33 2018</div>
<div class="">Formatting Allocation Map for storage pool raid1</div>
<div class=""> 53 % complete on Wed Aug 1 11:40:38 2018</div>
<div class=""> 100 % complete on Wed Aug 1 11:40:42 2018</div>
<div class="">Formatting Allocation Map for storage pool raid6</div>
<div class=""> 20 % complete on Wed Aug 1 11:40:47 2018</div>
<div class=""> 39 % complete on Wed Aug 1 11:40:52 2018</div>
<div class=""> 60 % complete on Wed Aug 1 11:40:57 2018</div>
<div class=""> 79 % complete on Wed Aug 1 11:41:02 2018</div>
<div class=""> 100 % complete on Wed Aug 1 11:41:08 2018</div>
<div class="">Completed creation of file system /dev/gpfs5.</div>
<div class="">mmcrfs: Propagating the cluster configuration data to all</div>
<div class=""> affected nodes. This is an asynchronous process.</div>
</div>
<div class=""><br class="">
</div>
<div class="">And contents of stanza file:</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">%nsd:</div>
<div class=""> nsd=test21A3nsd</div>
<div class=""> usage=metadataOnly</div>
<div class=""> failureGroup=210</div>
<div class=""> pool=system</div>
<div class=""> servers=testnsd3,testnsd1,testnsd2</div>
<div class=""> device=dm-15</div>
<div class=""><br class="">
</div>
<div class="">%nsd:</div>
<div class=""> nsd=test21A4nsd</div>
<div class=""> usage=metadataOnly</div>
<div class=""> failureGroup=210</div>
<div class=""> pool=system</div>
<div class=""> servers=testnsd1,testnsd2,testnsd3</div>
<div class=""> device=dm-14</div>
<div class=""><br class="">
</div>
<div class="">%nsd:</div>
<div class=""> nsd=test21B3nsd</div>
<div class=""> usage=metadataOnly</div>
<div class=""> failureGroup=211</div>
<div class=""> pool=system</div>
<div class=""> servers=testnsd1,testnsd2,testnsd3</div>
<div class=""> device=dm-17</div>
<div class=""><br class="">
</div>
<div class="">%nsd:</div>
<div class=""> nsd=test21B4nsd</div>
<div class=""> usage=metadataOnly</div>
<div class=""> failureGroup=211</div>
<div class=""> pool=system</div>
<div class=""> servers=testnsd2,testnsd3,testnsd1</div>
<div class=""> device=dm-16</div>
<div class=""><br class="">
</div>
<div class="">%nsd:</div>
<div class=""> nsd=test23Ansd</div>
<div class=""> usage=dataOnly</div>
<div class=""> failureGroup=23</div>
<div class=""> pool=raid6</div>
<div class=""> servers=testnsd2,testnsd3,testnsd1</div>
<div class=""> device=dm-10</div>
<div class=""><br class="">
</div>
<div class="">%nsd:</div>
<div class=""> nsd=test23Bnsd</div>
<div class=""> usage=dataOnly</div>
<div class=""> failureGroup=23</div>
<div class=""> pool=raid6</div>
<div class=""> servers=testnsd3,testnsd1,testnsd2</div>
<div class=""> device=dm-9</div>
<div class=""><br class="">
</div>
<div class="">%nsd:</div>
<div class=""> nsd=test23Cnsd</div>
<div class=""> usage=dataOnly</div>
<div class=""> failureGroup=23</div>
<div class=""> pool=raid1</div>
<div class=""> servers=testnsd1,testnsd2,testnsd3</div>
<div class=""> device=dm-5</div>
<div class=""><br class="">
</div>
<div class="">%nsd:</div>
<div class=""> nsd=test24Ansd</div>
<div class=""> usage=dataOnly</div>
<div class=""> failureGroup=24</div>
<div class=""> pool=raid6</div>
<div class=""> servers=testnsd3,testnsd1,testnsd2</div>
<div class=""> device=dm-6</div>
<div class=""><br class="">
</div>
<div class="">%nsd:</div>
<div class=""> nsd=test24Bnsd</div>
<div class=""> usage=dataOnly</div>
<div class=""> failureGroup=24</div>
<div class=""> pool=raid6</div>
<div class=""> servers=testnsd1,testnsd2,testnsd3</div>
<div class=""> device=dm-0</div>
<div class=""><br class="">
</div>
<div class="">%nsd:</div>
<div class=""> nsd=test24Cnsd</div>
<div class=""> usage=dataOnly</div>
<div class=""> failureGroup=24</div>
<div class=""> pool=raid1</div>
<div class=""> servers=testnsd2,testnsd3,testnsd1</div>
<div class=""> device=dm-2</div>
<div class=""><br class="">
</div>
<div class="">%nsd:</div>
<div class=""> nsd=test25Ansd</div>
<div class=""> usage=dataOnly</div>
<div class=""> failureGroup=25</div>
<div class=""> pool=raid6</div>
<div class=""> servers=testnsd1,testnsd2,testnsd3</div>
<div class=""> device=dm-6</div>
<div class=""><br class="">
</div>
<div class="">%nsd:</div>
<div class=""> nsd=test25Bnsd</div>
<div class=""> usage=dataOnly</div>
<div class=""> failureGroup=25</div>
<div class=""> pool=raid6</div>
<div class=""> servers=testnsd2,testnsd3,testnsd1</div>
<div class=""> device=dm-6</div>
<div class=""><br class="">
</div>
<div class="">%nsd:</div>
<div class=""> nsd=test25Cnsd</div>
<div class=""> usage=dataOnly</div>
<div class=""> failureGroup=25</div>
<div class=""> pool=raid1</div>
<div class=""> servers=testnsd3,testnsd1,testnsd2</div>
<div class=""> device=dm-3</div>
<div class=""><br class="">
</div>
<div class="">%pool:</div>
<div class=""> pool=system</div>
<div class=""> blockSize=1M</div>
<div class=""> usage=metadataOnly</div>
<div class=""> layoutMap=scatter</div>
<div class=""> allowWriteAffinity=no</div>
<div class=""><br class="">
</div>
<div class="">%pool:</div>
<div class=""> pool=raid6</div>
<div class=""> blockSize=4M</div>
<div class=""> usage=dataOnly</div>
<div class=""> layoutMap=scatter</div>
<div class=""> allowWriteAffinity=no</div>
<div class=""><br class="">
</div>
<div class="">%pool:</div>
<div class=""> pool=raid1</div>
<div class=""> blockSize=4M</div>
<div class=""> usage=dataOnly</div>
<div class=""> layoutMap=scatter</div>
<div class=""> allowWriteAffinity=no</div>
</div>
<div class=""><br class="">
</div>
<div class="">What am I missing or what have I done wrong? Thanks…</div>
<div class=""><br class="">
</div>
<div class="">Kevin</div>
<div class="">
<div class="">—</div>
<div class="">Kevin Buterbaugh - Senior System Administrator</div>
<div class="">Vanderbilt University - Advanced Computing Center for Research and Education</div>
<div class=""><a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" class="">Kevin.Buterbaugh@vanderbilt.edu</a> - (615)875-9633</div>
<div class=""><br class="">
</div>
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</body>
</html>