<span style=" font-size:10pt;font-family:sans-serif">I haven't looked
into all the details but here's a clue -- notice there is only one "subblocks-per-full-block"
parameter. </span><br><br><span style=" font-size:10pt;font-family:sans-serif">And it is the
same for both metadata blocks and datadata blocks.</span><br><span style=" font-size:10pt;font-family:sans-serif"><br>So maybe (MAYBE) that is a constraint somewhere...</span><br><br><span style=" font-size:10pt;font-family:sans-serif">Certainly, in
the currently supported code, that's what you get.<br></span><br><br><br><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">From:
</span><span style=" font-size:9pt;font-family:sans-serif">"Buterbaugh,
Kevin L" <Kevin.Buterbaugh@Vanderbilt.Edu></span><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">To:
</span><span style=" font-size:9pt;font-family:sans-serif">gpfsug
main discussion list <gpfsug-discuss@spectrumscale.org></span><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Date:
</span><span style=" font-size:9pt;font-family:sans-serif">08/01/2018
12:55 PM</span><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Subject:
</span><span style=" font-size:9pt;font-family:sans-serif">[gpfsug-discuss]
Sub-block size wrong on GPFS 5 filesystem?</span><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Sent
by: </span><span style=" font-size:9pt;font-family:sans-serif">gpfsug-discuss-bounces@spectrumscale.org</span><br><hr noshade><br><br><br><span style=" font-size:12pt">Hi All, </span><br><br><span style=" font-size:12pt">Our production cluster is still on GPFS
4.2.3.x, but in preparation for moving to GPFS 5 I have upgraded our small
(7 node) test cluster to GPFS 5.0.1-1. I am setting up a new filesystem
there using hardware that we recently life-cycled out of our production
environment.</span><br><br><span style=" font-size:12pt">I $B!H(Bsuccessfully$B!I(B created a filesystem
but I believe the sub-block size is wrong. I$B!G(Bm using a 4 MB filesystem
block size, so according to the mmcrfs man page the sub-block size should
be 8K:</span><br><br><span style=" font-size:12pt"> Table
1. Block sizes and subblock sizes</span><br><br><span style=" font-size:12pt">+$B!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>(B+$B!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>(B+</span><br><span style=" font-size:12pt">| Block size
| Subblock size
|</span><br><span style=" font-size:12pt">+$B!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>(B+$B!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>(B+</span><br><span style=" font-size:12pt">| 64 KiB
| 2 KiB
|</span><br><span style=" font-size:12pt">+$B!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>(B+$B!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>(B+</span><br><span style=" font-size:12pt">| 128 KiB
| 4 KiB
|</span><br><span style=" font-size:12pt">+$B!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>(B+$B!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>(B+</span><br><span style=" font-size:12pt">| 256 KiB, 512 KiB, 1 MiB, 2 |
8 KiB
|</span><br><span style=" font-size:12pt">| MiB, 4 MiB
|
|</span><br><span style=" font-size:12pt">+$B!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>(B+$B!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>(B+</span><br><span style=" font-size:12pt">| 8 MiB, 16 MiB
| 16 KiB
|</span><br><span style=" font-size:12pt">+$B!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>(B+$B!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>!>(B+</span><br><br><span style=" font-size:12pt">However, it appears that it$B!G(Bs 8K for
the system pool but 32K for the other pools:</span><br><br><span style=" font-size:12pt">flag
value
description</span><br><span style=" font-size:12pt">------------------- ------------------------
-----------------------------------</span><br><span style=" font-size:12pt"> -f
8192
Minimum fragment (subblock) size in bytes (system
pool)</span><br><span style=" font-size:12pt">
32768
Minimum fragment (subblock) size in bytes (other
pools)</span><br><span style=" font-size:12pt"> -i
4096
Inode size in bytes</span><br><span style=" font-size:12pt"> -I
32768
Indirect block size in bytes</span><br><span style=" font-size:12pt"> -m
2
Default number of metadata replicas</span><br><span style=" font-size:12pt"> -M
3
Maximum number of metadata replicas</span><br><span style=" font-size:12pt"> -r
1
Default number of data replicas</span><br><span style=" font-size:12pt"> -R
3
Maximum number of data replicas</span><br><span style=" font-size:12pt"> -j
scatter
Block allocation type</span><br><span style=" font-size:12pt"> -D
nfs4
File locking semantics in effect</span><br><span style=" font-size:12pt"> -k
all
ACL semantics in effect</span><br><span style=" font-size:12pt"> -n
32
Estimated number of nodes that will
mount file system</span><br><span style=" font-size:12pt"> -B
1048576
Block size (system pool)</span><br><span style=" font-size:12pt">
4194304
Block size (other pools)</span><br><span style=" font-size:12pt"> -Q
user;group;fileset Quotas
accounting enabled</span><br><span style=" font-size:12pt">
user;group;fileset Quotas
enforced</span><br><span style=" font-size:12pt">
none
Default quotas enabled</span><br><span style=" font-size:12pt"> --perfileset-quota No
Per-fileset
quota enforcement</span><br><span style=" font-size:12pt"> --filesetdf
No
Fileset df enabled?</span><br><span style=" font-size:12pt"> -V
19.01 (5.0.1.0)
File system version</span><br><span style=" font-size:12pt"> --create-time Wed
Aug 1 11:39:39 2018 File system creation time</span><br><span style=" font-size:12pt"> -z
No
Is DMAPI enabled?</span><br><span style=" font-size:12pt"> -L
33554432
Logfile size</span><br><span style=" font-size:12pt"> -E
Yes
Exact mtime mount option</span><br><span style=" font-size:12pt"> -S
relatime
Suppress atime mount option</span><br><span style=" font-size:12pt"> -K
whenpossible
Strict replica allocation option</span><br><span style=" font-size:12pt"> --fastea
Yes
Fast external attributes enabled?</span><br><span style=" font-size:12pt"> --encryption
No
Encryption enabled?</span><br><span style=" font-size:12pt"> --inode-limit 101095424
Maximum number of
inodes</span><br><span style=" font-size:12pt"> --log-replicas 0
Number of log replicas</span><br><span style=" font-size:12pt"> --is4KAligned Yes
is4KAligned?</span><br><span style=" font-size:12pt"> --rapid-repair Yes
rapidRepair
enabled?</span><br><span style=" font-size:12pt"> --write-cache-threshold 0
HAWC Threshold
(max 65536)</span><br><span style=" font-size:12pt"> --subblocks-per-full-block 128
Number of subblocks per
full block</span><br><span style=" font-size:12pt"> -P
system;raid1;raid6 Disk
storage pools in file system</span><br><span style=" font-size:12pt"> --file-audit-log No
File
Audit Logging enabled?</span><br><span style=" font-size:12pt"> --maintenance-mode No
Maintenance
Mode enabled?</span><br><span style=" font-size:12pt"> -d
test21A3nsd;test21A4nsd;test21B3nsd;test21B4nsd;test23Ansd;test23Bnsd;test23Cnsd;test24Ansd;test24Bnsd;test24Cnsd;test25Ansd;test25Bnsd;test25Cnsd
Disks in file system</span><br><span style=" font-size:12pt"> -A
yes
Automatic mount option</span><br><span style=" font-size:12pt"> -o
none
Additional mount options</span><br><span style=" font-size:12pt"> -T
/gpfs5
Default mount point</span><br><span style=" font-size:12pt"> --mount-priority 0
Mount
priority</span><br><br><span style=" font-size:12pt">Output of mmcrfs:</span><br><br><span style=" font-size:12pt">mmcrfs gpfs5 -F ~/gpfs/gpfs5.stanza -A
yes -B 4M -E yes -i 4096 -j scatter -k all -K whenpossible -m 2 -M 3 -n
32 -Q yes -r 1 -R 3 -T /gpfs5 -v yes --nofilesetdf --metadata-block-size
1M</span><br><br><span style=" font-size:12pt">The following disks of gpfs5 will be
formatted on node testnsd3:</span><br><span style=" font-size:12pt"> test21A3nsd: size 953609
MB</span><br><span style=" font-size:12pt"> test21A4nsd: size 953609
MB</span><br><span style=" font-size:12pt"> test21B3nsd: size 953609
MB</span><br><span style=" font-size:12pt"> test21B4nsd: size 953609
MB</span><br><span style=" font-size:12pt"> test23Ansd: size 15259744
MB</span><br><span style=" font-size:12pt"> test23Bnsd: size 15259744
MB</span><br><span style=" font-size:12pt"> test23Cnsd: size 1907468
MB</span><br><span style=" font-size:12pt"> test24Ansd: size 15259744
MB</span><br><span style=" font-size:12pt"> test24Bnsd: size 15259744
MB</span><br><span style=" font-size:12pt"> test24Cnsd: size 1907468
MB</span><br><span style=" font-size:12pt"> test25Ansd: size 15259744
MB</span><br><span style=" font-size:12pt"> test25Bnsd: size 15259744
MB</span><br><span style=" font-size:12pt"> test25Cnsd: size 1907468
MB</span><br><span style=" font-size:12pt">Formatting file system ...</span><br><span style=" font-size:12pt">Disks up to size 8.29 TB can be added
to storage pool system.</span><br><span style=" font-size:12pt">Disks up to size 16.60 TB can be added
to storage pool raid1.</span><br><span style=" font-size:12pt">Disks up to size 132.62 TB can be added
to storage pool raid6.</span><br><span style=" font-size:12pt">Creating Inode File</span><br><span style=" font-size:12pt"> 8 % complete on Wed Aug
1 11:39:19 2018</span><br><span style=" font-size:12pt"> 18 % complete on Wed Aug 1
11:39:24 2018</span><br><span style=" font-size:12pt"> 27 % complete on Wed Aug 1
11:39:29 2018</span><br><span style=" font-size:12pt"> 37 % complete on Wed Aug 1
11:39:34 2018</span><br><span style=" font-size:12pt"> 48 % complete on Wed Aug 1
11:39:39 2018</span><br><span style=" font-size:12pt"> 60 % complete on Wed Aug 1
11:39:44 2018</span><br><span style=" font-size:12pt"> 72 % complete on Wed Aug 1
11:39:49 2018</span><br><span style=" font-size:12pt"> 83 % complete on Wed Aug 1
11:39:54 2018</span><br><span style=" font-size:12pt"> 95 % complete on Wed Aug 1
11:39:59 2018</span><br><span style=" font-size:12pt"> 100 % complete on Wed Aug 1
11:40:01 2018</span><br><span style=" font-size:12pt">Creating Allocation Maps</span><br><span style=" font-size:12pt">Creating Log Files</span><br><span style=" font-size:12pt"> 3 % complete on Wed Aug
1 11:40:07 2018</span><br><span style=" font-size:12pt"> 28 % complete on Wed Aug 1
11:40:14 2018</span><br><span style=" font-size:12pt"> 53 % complete on Wed Aug 1
11:40:19 2018</span><br><span style=" font-size:12pt"> 78 % complete on Wed Aug 1
11:40:24 2018</span><br><span style=" font-size:12pt"> 100 % complete on Wed Aug 1
11:40:25 2018</span><br><span style=" font-size:12pt">Clearing Inode Allocation Map</span><br><span style=" font-size:12pt">Clearing Block Allocation Map</span><br><span style=" font-size:12pt">Formatting Allocation Map for storage
pool system</span><br><span style=" font-size:12pt"> 85 % complete on Wed Aug 1
11:40:32 2018</span><br><span style=" font-size:12pt"> 100 % complete on Wed Aug 1
11:40:33 2018</span><br><span style=" font-size:12pt">Formatting Allocation Map for storage
pool raid1</span><br><span style=" font-size:12pt"> 53 % complete on Wed Aug 1
11:40:38 2018</span><br><span style=" font-size:12pt"> 100 % complete on Wed Aug 1
11:40:42 2018</span><br><span style=" font-size:12pt">Formatting Allocation Map for storage
pool raid6</span><br><span style=" font-size:12pt"> 20 % complete on Wed Aug 1
11:40:47 2018</span><br><span style=" font-size:12pt"> 39 % complete on Wed Aug 1
11:40:52 2018</span><br><span style=" font-size:12pt"> 60 % complete on Wed Aug 1
11:40:57 2018</span><br><span style=" font-size:12pt"> 79 % complete on Wed Aug 1
11:41:02 2018</span><br><span style=" font-size:12pt"> 100 % complete on Wed Aug 1
11:41:08 2018</span><br><span style=" font-size:12pt">Completed creation of file system /dev/gpfs5.</span><br><span style=" font-size:12pt">mmcrfs: Propagating the cluster configuration
data to all</span><br><span style=" font-size:12pt"> affected nodes. This is
an asynchronous process.</span><br><br><span style=" font-size:12pt">And contents of stanza file:</span><br><br><span style=" font-size:12pt">%nsd:</span><br><span style=" font-size:12pt"> nsd=test21A3nsd</span><br><span style=" font-size:12pt"> usage=metadataOnly</span><br><span style=" font-size:12pt"> failureGroup=210</span><br><span style=" font-size:12pt"> pool=system</span><br><span style=" font-size:12pt"> servers=testnsd3,testnsd1,testnsd2</span><br><span style=" font-size:12pt"> device=dm-15</span><br><br><span style=" font-size:12pt">%nsd:</span><br><span style=" font-size:12pt"> nsd=test21A4nsd</span><br><span style=" font-size:12pt"> usage=metadataOnly</span><br><span style=" font-size:12pt"> failureGroup=210</span><br><span style=" font-size:12pt"> pool=system</span><br><span style=" font-size:12pt"> servers=testnsd1,testnsd2,testnsd3</span><br><span style=" font-size:12pt"> device=dm-14</span><br><br><span style=" font-size:12pt">%nsd:</span><br><span style=" font-size:12pt"> nsd=test21B3nsd</span><br><span style=" font-size:12pt"> usage=metadataOnly</span><br><span style=" font-size:12pt"> failureGroup=211</span><br><span style=" font-size:12pt"> pool=system</span><br><span style=" font-size:12pt"> servers=testnsd1,testnsd2,testnsd3</span><br><span style=" font-size:12pt"> device=dm-17</span><br><br><span style=" font-size:12pt">%nsd:</span><br><span style=" font-size:12pt"> nsd=test21B4nsd</span><br><span style=" font-size:12pt"> usage=metadataOnly</span><br><span style=" font-size:12pt"> failureGroup=211</span><br><span style=" font-size:12pt"> pool=system</span><br><span style=" font-size:12pt"> servers=testnsd2,testnsd3,testnsd1</span><br><span style=" font-size:12pt"> device=dm-16</span><br><br><span style=" font-size:12pt">%nsd:</span><br><span style=" font-size:12pt"> nsd=test23Ansd</span><br><span style=" font-size:12pt"> usage=dataOnly</span><br><span style=" font-size:12pt"> failureGroup=23</span><br><span style=" font-size:12pt"> pool=raid6</span><br><span style=" font-size:12pt"> servers=testnsd2,testnsd3,testnsd1</span><br><span style=" font-size:12pt"> device=dm-10</span><br><br><span style=" font-size:12pt">%nsd:</span><br><span style=" font-size:12pt"> nsd=test23Bnsd</span><br><span style=" font-size:12pt"> usage=dataOnly</span><br><span style=" font-size:12pt"> failureGroup=23</span><br><span style=" font-size:12pt"> pool=raid6</span><br><span style=" font-size:12pt"> servers=testnsd3,testnsd1,testnsd2</span><br><span style=" font-size:12pt"> device=dm-9</span><br><br><span style=" font-size:12pt">%nsd:</span><br><span style=" font-size:12pt"> nsd=test23Cnsd</span><br><span style=" font-size:12pt"> usage=dataOnly</span><br><span style=" font-size:12pt"> failureGroup=23</span><br><span style=" font-size:12pt"> pool=raid1</span><br><span style=" font-size:12pt"> servers=testnsd1,testnsd2,testnsd3</span><br><span style=" font-size:12pt"> device=dm-5</span><br><br><span style=" font-size:12pt">%nsd:</span><br><span style=" font-size:12pt"> nsd=test24Ansd</span><br><span style=" font-size:12pt"> usage=dataOnly</span><br><span style=" font-size:12pt"> failureGroup=24</span><br><span style=" font-size:12pt"> pool=raid6</span><br><span style=" font-size:12pt"> servers=testnsd3,testnsd1,testnsd2</span><br><span style=" font-size:12pt"> device=dm-6</span><br><br><span style=" font-size:12pt">%nsd:</span><br><span style=" font-size:12pt"> nsd=test24Bnsd</span><br><span style=" font-size:12pt"> usage=dataOnly</span><br><span style=" font-size:12pt"> failureGroup=24</span><br><span style=" font-size:12pt"> pool=raid6</span><br><span style=" font-size:12pt"> servers=testnsd1,testnsd2,testnsd3</span><br><span style=" font-size:12pt"> device=dm-0</span><br><br><span style=" font-size:12pt">%nsd:</span><br><span style=" font-size:12pt"> nsd=test24Cnsd</span><br><span style=" font-size:12pt"> usage=dataOnly</span><br><span style=" font-size:12pt"> failureGroup=24</span><br><span style=" font-size:12pt"> pool=raid1</span><br><span style=" font-size:12pt"> servers=testnsd2,testnsd3,testnsd1</span><br><span style=" font-size:12pt"> device=dm-2</span><br><br><span style=" font-size:12pt">%nsd:</span><br><span style=" font-size:12pt"> nsd=test25Ansd</span><br><span style=" font-size:12pt"> usage=dataOnly</span><br><span style=" font-size:12pt"> failureGroup=25</span><br><span style=" font-size:12pt"> pool=raid6</span><br><span style=" font-size:12pt"> servers=testnsd1,testnsd2,testnsd3</span><br><span style=" font-size:12pt"> device=dm-6</span><br><br><span style=" font-size:12pt">%nsd:</span><br><span style=" font-size:12pt"> nsd=test25Bnsd</span><br><span style=" font-size:12pt"> usage=dataOnly</span><br><span style=" font-size:12pt"> failureGroup=25</span><br><span style=" font-size:12pt"> pool=raid6</span><br><span style=" font-size:12pt"> servers=testnsd2,testnsd3,testnsd1</span><br><span style=" font-size:12pt"> device=dm-6</span><br><br><span style=" font-size:12pt">%nsd:</span><br><span style=" font-size:12pt"> nsd=test25Cnsd</span><br><span style=" font-size:12pt"> usage=dataOnly</span><br><span style=" font-size:12pt"> failureGroup=25</span><br><span style=" font-size:12pt"> pool=raid1</span><br><span style=" font-size:12pt"> servers=testnsd3,testnsd1,testnsd2</span><br><span style=" font-size:12pt"> device=dm-3</span><br><br><span style=" font-size:12pt">%pool:</span><br><span style=" font-size:12pt"> pool=system</span><br><span style=" font-size:12pt"> blockSize=1M</span><br><span style=" font-size:12pt"> usage=metadataOnly</span><br><span style=" font-size:12pt"> layoutMap=scatter</span><br><span style=" font-size:12pt"> allowWriteAffinity=no</span><br><br><span style=" font-size:12pt">%pool:</span><br><span style=" font-size:12pt"> pool=raid6</span><br><span style=" font-size:12pt"> blockSize=4M</span><br><span style=" font-size:12pt"> usage=dataOnly</span><br><span style=" font-size:12pt"> layoutMap=scatter</span><br><span style=" font-size:12pt"> allowWriteAffinity=no</span><br><br><span style=" font-size:12pt">%pool:</span><br><span style=" font-size:12pt"> pool=raid1</span><br><span style=" font-size:12pt"> blockSize=4M</span><br><span style=" font-size:12pt"> usage=dataOnly</span><br><span style=" font-size:12pt"> layoutMap=scatter</span><br><span style=" font-size:12pt"> allowWriteAffinity=no</span><br><br><span style=" font-size:12pt">What am I missing or what have I done
wrong? Thanks$B!D(B</span><br><br><span style=" font-size:12pt">Kevin</span><br><span style=" font-size:12pt">—</span><br><span style=" font-size:12pt">Kevin Buterbaugh - Senior System Administrator</span><br><span style=" font-size:12pt">Vanderbilt University - Advanced Computing
Center for Research and Education</span><br><a href="mailto:Kevin.Buterbaugh@vanderbilt.edu"><span style=" font-size:12pt;color:blue"><u>Kevin.Buterbaugh@vanderbilt.edu</u></span></a><span style=" font-size:12pt">- (615)875-9633</span><br><br><br><tt><span style=" font-size:10pt">_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></span></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><span style=" font-size:10pt">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></tt></a><tt><span style=" font-size:10pt"><br></span></tt><br><br><BR>