<span style=" font-size:10pt;font-family:sans-serif">Hello Walter,</span><br><br><span style=" font-size:10pt;font-family:sans-serif">If you anticipate
that the number of clients accessing this file system may grow as high
as 5000, then that is probably the value you should use when creating the
file system. The data structures (regions for example) are allocated at
file system creation time (more precisely at storage pool creation time)
and are not changed later.</span><br><br><span style=" font-size:10pt;font-family:sans-serif">The mmcrfs doc
explains this:</span><br><br><a href="https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.4/com.ibm.spectrum.scale.v5r04.doc/bl1adm_mmcrfs.htm"><span style=" font-size:10pt;color:blue;font-family:sans-serif">https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.4/com.ibm.spectrum.scale.v5r04.doc/bl1adm_mmcrfs.htm</span></a><br><br><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;color:#2f2f2f;font-family:Courier">-n </span><span style=" font-size:12pt;color:#5f5f5f;font-family:Courier"><i>NumNodes</i></span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;color:#2f2f2f;font-family:Arial">The
estimated number of nodes that will mount the file system in the local
cluster and all remote clusters. This is used as a best guess for the initial
size of some file system data structures. The default is 32. This value
can be changed after the file system has been created but it does not change
the existing data structures. Only the newly created data structure is
affected by the new value. For example, </span><span style=" font-size:12pt;color:#5f5f5f;font-family:Arial"><i>new
storage pool</i></span><span style=" font-size:12pt;color:#2f2f2f;font-family:Arial">.</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;color:#2f2f2f;font-family:Arial">When
you create a GPFS file system, you might want to overestimate the number
of nodes that will mount the file system. GPFS uses this information for
creating data structures that are essential for achieving maximum parallelism
in file system operations (For more information, see </span><span style=" font-size:12pt;color:#5f5f5f;font-family:Arial"><i>GPFS
architecture</i></span><span style=" font-size:12pt;color:#2f2f2f;font-family:Arial"> ).
If you are sure there will never be more than 64 nodes, allow the default
value to be applied. If you are planning to add nodes to your system, you
should specify a number larger than the default.</span><span style=" font-size:10pt;font-family:sans-serif"><br></span></p><br><span style=" font-size:10pt;font-family:sans-serif">Thanks,</span><br><span style=" font-size:9pt;font-family:Arial"><br></span><table width=650 style="border-collapse:collapse;"><tr height=8><td width=650 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:12pt;color:#8f8f8f;font-family:Arial"><b>Nate
Falk</b></span><span style=" font-size:9pt;font-family:Arial"><br>IBM Spectrum Scale Level 2 Support<br>Software Defined Infrastructure, IBM Systems</span></table><p style="margin-top:0px;margin-Bottom:0px"></p><table width=650 style="border-collapse:collapse;"><tr height=8><td width=650 colspan=2 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><hr><tr valign=top height=8><td width=363 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:7pt;color:#4181c0;font-family:Arial"><b>Phone:</b></span><span style=" font-size:7pt;color:#5f5f5f;font-family:Arial">1-720-349-9538</span><span style=" font-size:7pt;color:#4181c0;font-family:Arial">| <b>Mobile:</b></span><span style=" font-size:7pt;color:#5f5f5f;font-family:Arial">1-845-546-4930</span><span style=" font-size:7pt;color:#4181c0;font-family:Arial"><b><br>E-mail:</b></span><span style=" font-size:7pt;color:#5f5f5f;font-family:Arial"></span><a href="mailto:nfalk@us.ibm.com" target="_blank"><span style=" font-size:7pt;color:#5f5f5f;font-family:Arial"><u>nfalk@us.ibm.com</u></span></a><span style=" font-size:7pt;color:#4181c0;font-family:Arial"><b><br>Find me on:</b></span><span style=" font-size:7pt;color:#5f5f5f;font-family:Arial"></span><a href="https://www.linkedin.com/in/nathan-falk-078ba5125" target="_blank"><img src=http://www.ibm.com/webaccessories/emailsig/i/LinkedIn.jpg alt="LinkedIn: https://www.linkedin.com/in/nathan-falk-078ba5125" style="border:0px solid;"></a><span style=" font-size:7pt;color:#5f5f5f;font-family:Arial"> </span><a href="https://twitter.com/natefalk922" target="_blank"><img src=http://www.ibm.com/webaccessories/emailsig/i/Twitter.jpg alt="Twitter: https://twitter.com/natefalk922" style="border:0px solid;"></a><span style=" font-size:7pt;color:#5f5f5f;font-family:Arial"></span><td width=286 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><div align=right><img align=bottom src=http://www.ibm.com/webaccessories/emailsig/i/ibm2.gif width=83 height=30 alt=IBM style="border:0px solid;"></div></table><p style="margin-top:0px;margin-Bottom:0px"></p><br><br><br><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">From:
</span><span style=" font-size:9pt;font-family:sans-serif">Walter
Sklenka <Walter.Sklenka@EDV-Design.at></span><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">To:
</span><span style=" font-size:9pt;font-family:sans-serif">gpfsug
main discussion list <gpfsug-discuss@spectrumscale.org></span><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Date:
</span><span style=" font-size:9pt;font-family:sans-serif">02/09/2020
04:59 AM</span><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Subject:
</span><span style=" font-size:9pt;font-family:sans-serif">[EXTERNAL]
Re: [gpfsug-discuss] mmdf and maybe other commands long running // influence
of n and B on number of regions</span><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Sent
by: </span><span style=" font-size:9pt;font-family:sans-serif">gpfsug-discuss-bounces@spectrumscale.org</span><br><hr noshade><br><br><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri">Hi!</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri">At
the time of writing we set N to 1200 , but we are not sure if it would
be better to set to overestimated 5000 ? </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri">We
use 6 backend nodes </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri">The
backend storage is a Flash9100 for metadata and 6x Lenovo DE6000H . We
will finally use 2 filesystems : data and home</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri">Fs
“data” consist of 12 metadada-nsd and 72 dataonly nsds </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri">We
have enough space to add nsds (finally the fs </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">[root@nsd75-01
~]# mmlspool data</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">Storage
pools in file system at '/gpfs/data':</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">Name
Id
BlkSize Data Meta Total Data in (KB) Free Data in (KB)
Total Meta in (KB) Free Meta in (KB)</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">system
0
4 MB no yes
0 0 ( 0%)
12884901888 12800315392 ( 99%)</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">saspool
65537 4
MB yes no 1082331758592 1082326446080 (100%)
0
0 ( 0%)</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">[root@nsd75-01
~]# mmlsfs data</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">flag
value
description</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-------------------
------------------------ -----------------------------------</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-f
8192
Minimum fragment
(subblock) size in bytes</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-i
4096
Inode size in bytes</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-I
32768
Indirect block size
in bytes</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-m
1
Default
number of metadata replicas</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-M
2
Maximum
number of metadata replicas</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-r
1
Default
number of data replicas</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-R
2
Maximum
number of data replicas</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-j
scatter
Block allocation
type</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-D
nfs4
File locking semantics
in effect</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-k
all
ACL semantics
in effect</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-n
1200
Estimated number
of nodes that will mount file system</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-B
4194304
Block size</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-Q
user;group;fileset
Quotas accounting enabled</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">
user;group;fileset
Quotas enforced</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">
fileset
Default quotas
enabled</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">--perfileset-quota
Yes
Per-fileset quota enforcement</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">--filesetdf
Yes
Fileset df enabled?</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-V
21.00 (5.0.3.0)
File system version</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">--create-time
Fri Feb 7 15:32:05 2020 File system creation
time</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-z
No
Is DMAPI
enabled?</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-L
33554432
Logfile size</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-E
Yes
Exact mtime
mount option</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-S
relatime
Suppress atime mount option</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-K
whenpossible
Strict replica allocation option</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">--fastea
Yes
Fast external attributes enabled?</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">--encryption
No
Encryption enabled?</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">--inode-limit
1342177280
Maximum number of inodes</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">--log-replicas
0
Number of log replicas</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">--is4KAligned
Yes
is4KAligned?</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">--rapid-repair
Yes
rapidRepair enabled?</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">--write-cache-threshold
0 HAWC Threshold
(max 65536)</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">--subblocks-per-full-block
512 Number of subblocks
per full block</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-P
system;saspool
Disk storage pools in file system</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">--file-audit-log
No
File Audit Logging enabled?</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">--maintenance-mode
No
Maintenance Mode enabled?</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-d
de750101vol01;de750101vol02;de750101vol03;de750101vol04;de750101vol05;de750101vol06;de750102vol01;de750102vol02;de750102vol03;de750102vol04;de750102vol05;de750102vol06;</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-d
de750201vol01;de750201vol02;de750201vol03;de750201vol04;de750201vol05;de750201vol06;de750202vol01;de750202vol02;de750202vol03;de750202vol04;de750202vol05;de750202vol06;</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-d
de760101vol01;de760101vol02;de760101vol03;de760101vol04;de760101vol05;de760101vol06;de760102vol01;de760102vol02;de760102vol03;de760102vol04;de760102vol05;de760102vol06;</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-d
de760201vol01;de760201vol02;de760201vol03;de760201vol04;de760201vol05;de760201vol06;de760202vol01;de760202vol02;de760202vol03;de760202vol04;de760202vol05;de760202vol06;</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-d
de770101vol01;de770101vol02;de770101vol03;de770101vol04;de770101vol05;de770101vol06;de770102vol01;de770102vol02;de770102vol03;de770102vol04;de770102vol05;de770102vol06;</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-d
de770201vol01;de770201vol02;de770201vol03;de770201vol04;de770201vol05;de770201vol06;de770202vol01;de770202vol02;de770202vol03;de770202vol04;de770202vol05;de770202vol06;</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-d
globalmeta0;globalmeta1;globalmeta2;globalmeta3;globalmeta4;globalmeta5;globalmeta6;globalmeta7;globalmeta8;globalmeta9;globalmeta10;globalmeta11
Disks in file system</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-A
yes
Automatic
mount option</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-o
none
Additional mount
options</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-T
/gpfs/data
Default mount point</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">--mount-priority
0
Mount priority</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri">##
</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri">For
fs Home we use 24 dataAdnMetadata disks only on flash </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">[root@nsd75-01
~]# mmlspool home</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">Storage
pools in file system at '/gpfs/home':</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">Name
Id
BlkSize Data Meta Total Data in (KB) Free Data in (KB)
Total Meta in (KB) Free Meta in (KB)</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">system
0
1024 KB yes yes 25769803776 25722931200
(100%) 25769803776 25722981376 (100%)</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">[root@nsd75-01
~]#</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">[root@nsd75-01
~]# mmlsfs home</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">flag
value
description</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-------------------
------------------------ -----------------------------------</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-f
8192
Minimum fragment
(subblock) size in bytes</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-i
4096
Inode size in bytes</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-I
32768
Indirect block size
in bytes</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-m
1
Default
number of metadata replicas</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-M
2
Maximum
number of metadata replicas</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-r
1
Default
number of data replicas</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-R
2
Maximum
number of data replicas</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-j
scatter
Block allocation
type</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-D
nfs4
File locking semantics
in effect</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-k
all
ACL semantics
in effect</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-n
1200
Estimated number
of nodes that will mount file system</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-B
1048576
Block size</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-Q
user;group;fileset
Quotas accounting enabled</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">
user;group;fileset
Quotas enforced</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">
fileset
Default quotas
enabled</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">--perfileset-quota
Yes
Per-fileset quota enforcement</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">--filesetdf
Yes
Fileset df enabled?</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-V
21.00 (5.0.3.0)
File system version</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">--create-time
Fri Feb 7 15:31:28 2020 File system creation
time</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-z
No
Is DMAPI
enabled?</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-L
33554432
Logfile size</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-E
Yes
Exact mtime
mount option</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-S
relatime
Suppress atime mount option</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-K
whenpossible
Strict replica allocation option</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">--fastea
Yes
Fast external attributes enabled?</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">--encryption
No
Encryption enabled?</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">--inode-limit
25166080
Maximum number of inodes</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">--log-replicas
0
Number of log replicas</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">--is4KAligned
Yes
is4KAligned?</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">--rapid-repair
Yes
rapidRepair enabled?</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">--write-cache-threshold
0 HAWC Threshold
(max 65536)</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">--subblocks-per-full-block
128 Number of subblocks
per full block</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-P
system
Disk storage pools in
file system</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">--file-audit-log
No
File Audit Logging enabled?</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">--maintenance-mode
No
Maintenance Mode enabled?</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-d
home0;home10;home11;home12;home13;home14;home15;home16;home17;home18;home19;home1;home20;home21;home22;home23;home2;home3;home4;home5;home6;home7;home8;home9
Disks in file system</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-A
yes
Automatic
mount option</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-o
none
Additional mount
options</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">-T
/gpfs/home
Default mount point</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">--mount-priority
0
Mount priority</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">[root@nsd75-01
~]#</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:10pt;font-family:Arial">Mit
freundlichen Grüßen<b><i><br>Walter Sklenka</i></b></span><span style=" font-size:8pt;color:#808080;font-family:Arial"><b><i><br>Technical Consultant</i></b></span><span style=" font-size:11pt;font-family:Calibri"></span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:Times New Roman"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:8pt;font-family:Arial">EDV-Design
Informationstechnologie GmbH<br>Giefinggasse 6/1/2, A-1210 Wien<br>Tel: +43 1 29 22 165-31<br>Fax: +43 1 29 22 165-90<br>E-Mail: </span><a href="mailto:sklenka@edv-design.at" target="_blank"><span style=" font-size:8pt;color:#0082bf;font-family:Arial"><u>sklenka@edv-design.at</u></span></a><span style=" font-size:8pt;font-family:Arial"><br>Internet: </span><a href="http://www.edv-design.at/" target="_blank"><span style=" font-size:8pt;color:#0082bf;font-family:Arial"><u>www.edv-design.at</u></span></a><span style=" font-size:8pt;font-family:Calibri"><i><br></i></span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri"><b>Von:</b>gpfsug-discuss-bounces@spectrumscale.org <gpfsug-discuss-bounces@spectrumscale.org>
<b>Im Auftrag von </b>José Filipe Higino<b><br>Gesendet:</b> Saturday, February 8, 2020 1:00 PM<b><br>An:</b> gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><b><br>Betreff:</b> Re: [gpfsug-discuss] mmdf and maybe other commands long running
// influence of n and B on number of regions</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri">How
many back end nodes for that cluster? and how many filesystems for that
same access... and how many pools for the same data access type (12 ndisks
sounds very LOW to me, for that size of a cluster, probably no other filesystem
can do more than that). On GPFS there are so many different ways to access
the data, that is sometimes hard to start a conversation. And you did a
very great job of introducing it. =)</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri">We
(I am a customer too) do not have that many nodes, but from experience,
I know some clusters (and also multicluster configs) depend mostly on how
much metadata you can service in the network and how fast (latency wise)
you can do it, to accommodate such amount of nodes. There is never design
by the book that can safely tell something will work 100% times. But the
beauty of it is that GPFS allows lots of aspects to be resized at your
convenience to facilitate what you need most the system to do.</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri">Let
us know more...</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri">On
Sun, 9 Feb 2020 at 00:40, Walter Sklenka <</span><a href="mailto:Walter.Sklenka@edv-design.at"><span style=" font-size:11pt;color:blue;font-family:Calibri"><u>Walter.Sklenka@edv-design.at</u></span></a><span style=" font-size:11pt;font-family:Calibri">>
wrote:</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri">Hello!</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri">We
are designing two fs where we cannot anticipate if there will be
3000, or maybe 5000 or more nodes totally accessing these filesystems</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri">What
we saw, was that execution time of mmdf can last 5-7min </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri">We
openend a case and they said, that during such commands like mmdf or also
mmfsck, mmdefragfs,mmresripefs all regions must be scanned at this is the
reason why it takes so long</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri">The
technichian also said, that it is “rule of thumb” that there should be
</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri">(-n)*32
regions , this would then be enough ( N=5000 </span><span style=" font-size:11pt;font-family:Wingdings">à</span><span style=" font-size:11pt;font-family:Calibri">160000 regions per pool ?) </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri">(also
Block size has influence on regions ?) </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri">#mmfsadm
saferdump stripe </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Courier New">Gives
the regions number </span></p><br><span style=" font-size:11pt;font-family:Courier New"> storage
pools: max 8</span><br><span style=" font-size:11pt;font-family:Courier New"> </span><br><span style=" font-size:11pt;font-family:Courier New">
alloc map type 'scatter'</span><br><span style=" font-size:11pt;font-family:Courier New"> </span><br><span style=" font-size:11pt;font-family:Courier New">
0: name 'system' Valid nDisks 12 nInUse 12 id 0 poolFlags 0 thinProvision
reserved inode -1, reserved nBlocks 0</span><br><span style=" font-size:11pt;font-family:Courier New"> </span><br><span style=" font-size:11pt;font-family:Courier New">
<u>regns 170413</u> segs 1 size 4096 FBlks 0 MBlks
3145728 subblock size 8192</span><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri">We
also saw when creating the filesystem with a speciicic (-n) very
high (5000) (where mmdf execution time was some minutes) and then
changing (-n) to a lower value this does not influence the behavior any
more</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri">My
question is: Is the rule (Number of Nodes)x5000 for number of regios in
a pool an good estimation , </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri">Is
it better to overestimate the number of Nodes (lnger running commands)
or is it unrealistic to get into problems when not reaching the regions
number calculated ?</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri">Does
anybody have experience with high number of nodes (>>3000)
and how to design the filesystems for such large clusters ? </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri">Thank
you very much in advance !</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:10pt;font-family:Arial">Mit
freundlichen Grüßen<b><i><br>Walter Sklenka</i></b></span><span style=" font-size:8pt;color:#808080;font-family:Arial"><b><i><br>Technical Consultant</i></b></span><span style=" font-size:11pt;font-family:Calibri"></span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:Times New Roman"> </span></p><p style="margin-top:0px;margin-Bottom:240px"><span style=" font-size:8pt;font-family:Arial">EDV-Design
Informationstechnologie GmbH<br>Giefinggasse 6/1/2, A-1210 Wien<br>Tel: +43 1 29 22 165-31<br>Fax: +43 1 29 22 165-90<br>E-Mail: </span><a href="mailto:sklenka@edv-design.at" target="_blank"><span style=" font-size:8pt;color:#0082bf;font-family:Arial"><u>sklenka@edv-design.at</u></span></a><span style=" font-size:8pt;font-family:Arial"><br>Internet: </span><a href="http://www.edv-design.at/" target="_blank"><span style=" font-size:8pt;color:#0082bf;font-family:Arial"><u>www.edv-design.at</u></span></a></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:11pt;font-family:Calibri">_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </span><a href="http://spectrumscale.org" target="_blank"><span style=" font-size:11pt;color:blue;font-family:Calibri"><u>spectrumscale.org</u></span></a><span style=" font-size:11pt;color:blue;font-family:Calibri"><u><br></u></span><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><span style=" font-size:11pt;color:blue;font-family:Calibri"><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></span></a></p><br><tt><span style=" font-size:10pt">_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></span></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><span style=" font-size:10pt">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></tt></a><tt><span style=" font-size:10pt"><br></span></tt><br><br><BR>