<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Courier;
panose-1:2 7 4 9 2 2 5 2 4 4;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
tt
{mso-style-priority:99;
font-family:"Courier New";}
span.E-MailFormatvorlage21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hello Nate!<o:p></o:p></p>
<p class="MsoNormal">Thank you very much for the response<o:p></o:p></p>
<p class="MsoNormal">Do you know if the rule of thumb for “enough regions =N*32 per pool
<o:p></o:p></p>
<p class="MsoNormal">And isn´t there an other way to increate the number of regions? (mybe by reducing block-size ?
<o:p></o:p></p>
<p class="MsoNormal">It´s only because the commands excetuin time of a couple of minutes make me nervous , or is the reason more a poor metadata perf for the long running command?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">But if you say so we will change it to N=5000 <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Best regards<o:p></o:p></p>
<p class="MsoNormal">Walter <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span lang="DE-AT" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">Mit freundlichen Grüßen<br>
<b><i>Walter Sklenka</i></b><br>
</span><b><i><span lang="DE-AT" style="font-size:8.0pt;font-family:"Arial",sans-serif;color:gray">Technical Consultant</span></i></b><span lang="DE-AT" style="color:black">
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="DE-AT" style="font-size:12.0pt;font-family:"Times New Roman",serif;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="DE-AT" style="font-size:8.0pt;font-family:"Arial",sans-serif;color:black">EDV-Design Informationstechnologie GmbH<br>
Giefinggasse 6/1/2, A-1210 Wien<br>
Tel: +43 1 29 22 165-31<br>
Fax: +43 1 29 22 165-90<br>
E-Mail: <a href="mailto:sklenka@edv-design.at" target="_blank"><span style="color:#0563C1">sklenka@edv-design.at</span></a></span><span lang="DE-AT" style="font-size:8.0pt;color:black"><br>
</span><span lang="DE-AT" style="font-size:8.0pt;font-family:"Arial",sans-serif;color:black">Internet:
<a href="http://www.edv-design.at/" target="_blank"><span style="color:#0563C1">www.edv-design.at</span></a></span><i><span lang="DE-AT" style="font-size:8.0pt;color:black"><br>
<br>
</span></i><span lang="DE"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="DE"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="DE">Von:</span></b><span lang="DE"> gpfsug-discuss-bounces@spectrumscale.org <gpfsug-discuss-bounces@spectrumscale.org>
<b>Im Auftrag von </b>Nathan Falk<br>
<b>Gesendet:</b> Monday, February 10, 2020 3:57 PM<br>
<b>An:</b> gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
<b>Betreff:</b> Re: [gpfsug-discuss] mmdf and maybe other commands long running // influence of n and B on number of regions<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Hello Walter,</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif">If you anticipate that the number of clients accessing this file system may grow as high as 5000, then that is probably the value you should use when creating the file system. The data structures
(regions for example) are allocated at file system creation time (more precisely at storage pool creation time) and are not changed later.</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif">The mmcrfs doc explains this:</span><br>
<br>
<a href="https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.4/com.ibm.spectrum.scale.v5r04.doc/bl1adm_mmcrfs.htm"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.4/com.ibm.spectrum.scale.v5r04.doc/bl1adm_mmcrfs.htm</span></a><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:12.0pt;font-family:Courier;color:#2F2F2F">-n </span><i><span style="font-size:12.0pt;font-family:Courier;color:#5F5F5F">NumNodes</span></i><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#2F2F2F">The estimated number of nodes that will mount the file system in the local cluster and all remote clusters. This is used as a best guess
for the initial size of some file system data structures. The default is 32. This value can be changed after the file system has been created but it does not change the existing data structures. Only the newly created data structure is affected by the new
value. For example, </span><i><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#5F5F5F">new storage pool</span></i><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#2F2F2F">.</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#2F2F2F">When you create a GPFS file system, you might want to overestimate the number of nodes that will mount the file system. GPFS uses this information
for creating data structures that are essential for achieving maximum parallelism in file system operations (For more information, see </span><i><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#5F5F5F">GPFS architecture</span></i><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#2F2F2F"> ).
If you are sure there will never be more than 64 nodes, allow the default value to be applied. If you are planning to add nodes to your system, you should specify a number larger than the default.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif">Thanks,</span><o:p></o:p></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="650" style="width:487.5pt;border-collapse:collapse">
<tbody>
<tr style="height:6.0pt">
<td width="650" style="width:487.5pt;padding:0in 0in 0in 0in;height:6.0pt">
<p class="MsoNormal"><b><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#8F8F8F">Nate Falk</span></b><span style="font-size:9.0pt;font-family:"Arial",sans-serif"><br>
IBM Spectrum Scale Level 2 Support<br>
Software Defined Infrastructure, IBM Systems</span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><span style="display:none"><o:p> </o:p></span></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="650" style="width:487.5pt;border-collapse:collapse">
<tbody>
<tr style="height:6.0pt">
<td width="650" colspan="2" style="width:487.5pt;padding:0in 0in 0in 0in;height:6.0pt">
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" align="center">
</div>
</td>
</tr>
<tr style="height:6.0pt">
<td width="363" valign="top" style="width:272.25pt;padding:0in 0in 0in 0in;height:6.0pt">
<p class="MsoNormal"><b><span style="font-size:7.0pt;font-family:"Arial",sans-serif;color:#4181C0">Phone:</span></b><span style="font-size:7.0pt;font-family:"Arial",sans-serif;color:#5F5F5F">1-720-349-9538</span><span style="font-size:7.0pt;font-family:"Arial",sans-serif;color:#4181C0">|
<b>Mobile:</b></span><span style="font-size:7.0pt;font-family:"Arial",sans-serif;color:#5F5F5F">1-845-546-4930</span><b><span style="font-size:7.0pt;font-family:"Arial",sans-serif;color:#4181C0"><br>
E-mail:</span></b><a href="mailto:nfalk@us.ibm.com" target="_blank"><span style="font-size:7.0pt;font-family:"Arial",sans-serif;color:#5F5F5F">nfalk@us.ibm.com</span></a><b><span style="font-size:7.0pt;font-family:"Arial",sans-serif;color:#4181C0"><br>
Find me on:</span></b><a href="https://www.linkedin.com/in/nathan-falk-078ba5125" target="_blank"><span style="text-decoration:none"><img border="0" id="_x0000_i1026" src="http://www.ibm.com/webaccessories/emailsig/i/LinkedIn.jpg" alt="LinkedIn: https://www.linkedin.com/in/nathan-falk-078ba5125"></span></a><span style="font-size:7.0pt;font-family:"Arial",sans-serif;color:#5F5F5F"> </span><a href="https://twitter.com/natefalk922" target="_blank"><span style="text-decoration:none"><img border="0" id="_x0000_i1027" src="http://www.ibm.com/webaccessories/emailsig/i/Twitter.jpg" alt="Twitter: https://twitter.com/natefalk922"></span></a><o:p></o:p></p>
</td>
<td width="286" valign="top" style="width:214.5pt;padding:0in 0in 0in 0in;height:6.0pt">
<p class="MsoNormal" align="right" style="text-align:right"><img border="0" width="83" height="30" style="width:.8645in;height:.3125in" id="_x0000_i1028" src="http://www.ibm.com/webaccessories/emailsig/i/ibm2.gif" alt="IBM"><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><br>
<br>
<br>
<br>
<span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#5F5F5F">From: </span><span style="font-size:9.0pt;font-family:"Arial",sans-serif">Walter Sklenka <<a href="mailto:Walter.Sklenka@EDV-Design.at">Walter.Sklenka@EDV-Design.at</a>></span><br>
<span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#5F5F5F">To: </span><span style="font-size:9.0pt;font-family:"Arial",sans-serif">gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.org</a>></span><br>
<span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#5F5F5F">Date: </span><span style="font-size:9.0pt;font-family:"Arial",sans-serif">02/09/2020 04:59 AM</span><br>
<span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#5F5F5F">Subject: </span><span style="font-size:9.0pt;font-family:"Arial",sans-serif">[EXTERNAL] Re: [gpfsug-discuss] mmdf and maybe other commands long running // influence of n and B
on number of regions</span><br>
<span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#5F5F5F">Sent by: </span><span style="font-size:9.0pt;font-family:"Arial",sans-serif"><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">gpfsug-discuss-bounces@spectrumscale.org</a></span><o:p></o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" noshade="" style="color:#A0A0A0" align="center">
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
<p style="margin:0in;margin-bottom:.0001pt">Hi!<o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt">At the time of writing we set N to 1200 , but we are not sure if it would be better to set to overestimated 5000 ?
<o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt">We use 6 backend nodes <o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt">The backend storage is a Flash9100 for metadata and 6x Lenovo DE6000H . We will finally use 2 filesystems : data and home<o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt">Fs “data” consist of 12 metadada-nsd and 72 dataonly nsds
<o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"> <o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt">We have enough space to add nsds (finally the fs
<o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">[root@nsd75-01 ~]# mmlspool data</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">Storage pools in file system at '/gpfs/data':</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">Name Id BlkSize Data Meta Total Data in (KB) Free Data in (KB) Total Meta in (KB) Free Meta in (KB)</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">system 0 4 MB no yes 0 0 ( 0%) 12884901888 12800315392 ( 99%)</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">saspool 65537 4 MB yes no 1082331758592 1082326446080 (100%) 0 0 ( 0%)</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">[root@nsd75-01 ~]# mmlsfs data</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">flag value description</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">------------------- ------------------------ -----------------------------------</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-f 8192 Minimum fragment (subblock) size in bytes</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-i 4096 Inode size in bytes</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-I 32768 Indirect block size in bytes</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-m 1 Default number of metadata replicas</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-M 2 Maximum number of metadata replicas</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-r 1 Default number of data replicas</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-R 2 Maximum number of data replicas</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-j scatter Block allocation type</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-D nfs4 File locking semantics in effect</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-k all ACL semantics in effect</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-n 1200 Estimated number of nodes that will mount file system</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-B 4194304 Block size</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-Q user;group;fileset Quotas accounting enabled</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New""> user;group;fileset Quotas enforced</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New""> fileset Default quotas enabled</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">--perfileset-quota Yes Per-fileset quota enforcement</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">--filesetdf Yes Fileset df enabled?</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-V 21.00 (5.0.3.0) File system version</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">--create-time Fri Feb 7 15:32:05 2020 File system creation time</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-z No Is DMAPI enabled?</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-L 33554432 Logfile size</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-E Yes Exact mtime mount option</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-S relatime Suppress atime mount option</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-K whenpossible Strict replica allocation option</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">--fastea Yes Fast external attributes enabled?</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">--encryption No Encryption enabled?</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">--inode-limit 1342177280 Maximum number of inodes</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">--log-replicas 0 Number of log replicas</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">--is4KAligned Yes is4KAligned?</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">--rapid-repair Yes rapidRepair enabled?</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">--write-cache-threshold 0 HAWC Threshold (max 65536)</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">--subblocks-per-full-block 512 Number of subblocks per full block</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-P system;saspool Disk storage pools in file system</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">--file-audit-log No File Audit Logging enabled?</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">--maintenance-mode No Maintenance Mode enabled?</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-d de750101vol01;de750101vol02;de750101vol03;de750101vol04;de750101vol05;de750101vol06;de750102vol01;de750102vol02;de750102vol03;de750102vol04;de750102vol05;de750102vol06;</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-d de750201vol01;de750201vol02;de750201vol03;de750201vol04;de750201vol05;de750201vol06;de750202vol01;de750202vol02;de750202vol03;de750202vol04;de750202vol05;de750202vol06;</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-d de760101vol01;de760101vol02;de760101vol03;de760101vol04;de760101vol05;de760101vol06;de760102vol01;de760102vol02;de760102vol03;de760102vol04;de760102vol05;de760102vol06;</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-d de760201vol01;de760201vol02;de760201vol03;de760201vol04;de760201vol05;de760201vol06;de760202vol01;de760202vol02;de760202vol03;de760202vol04;de760202vol05;de760202vol06;</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-d de770101vol01;de770101vol02;de770101vol03;de770101vol04;de770101vol05;de770101vol06;de770102vol01;de770102vol02;de770102vol03;de770102vol04;de770102vol05;de770102vol06;</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-d de770201vol01;de770201vol02;de770201vol03;de770201vol04;de770201vol05;de770201vol06;de770202vol01;de770202vol02;de770202vol03;de770202vol04;de770202vol05;de770202vol06;</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-d globalmeta0;globalmeta1;globalmeta2;globalmeta3;globalmeta4;globalmeta5;globalmeta6;globalmeta7;globalmeta8;globalmeta9;globalmeta10;globalmeta11 Disks in
file system</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-A yes Automatic mount option</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-o none Additional mount options</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-T /gpfs/data Default mount point</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">--mount-priority 0 Mount priority</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"> <o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"> <o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt">## <o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt">For fs Home we use 24 dataAdnMetadata disks only on flash
<o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"> <o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">[root@nsd75-01 ~]# mmlspool home</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">Storage pools in file system at '/gpfs/home':</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">Name Id BlkSize Data Meta Total Data in (KB) Free Data in (KB) Total Meta in (KB) Free Meta in (KB)</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">system 0 1024 KB yes yes 25769803776 25722931200 (100%) 25769803776 25722981376 (100%)</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">[root@nsd75-01 ~]#</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">[root@nsd75-01 ~]# mmlsfs home</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">flag value description</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">------------------- ------------------------ -----------------------------------</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-f 8192 Minimum fragment (subblock) size in bytes</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-i 4096 Inode size in bytes</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-I 32768 Indirect block size in bytes</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-m 1 Default number of metadata replicas</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-M 2 Maximum number of metadata replicas</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-r 1 Default number of data replicas</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-R 2 Maximum number of data replicas</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-j scatter Block allocation type</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-D nfs4 File locking semantics in effect</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-k all ACL semantics in effect</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-n 1200 Estimated number of nodes that will mount file system</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-B 1048576 Block size</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-Q user;group;fileset Quotas accounting enabled</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New""> user;group;fileset Quotas enforced</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New""> fileset Default quotas enabled</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">--perfileset-quota Yes Per-fileset quota enforcement</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">--filesetdf Yes Fileset df enabled?</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-V 21.00 (5.0.3.0) File system version</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">--create-time Fri Feb 7 15:31:28 2020 File system creation time</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-z No Is DMAPI enabled?</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-L 33554432 Logfile size</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-E Yes Exact mtime mount option</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-S relatime Suppress atime mount option</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-K whenpossible Strict replica allocation option</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">--fastea Yes Fast external attributes enabled?</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">--encryption No Encryption enabled?</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">--inode-limit 25166080 Maximum number of inodes</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">--log-replicas 0 Number of log replicas</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">--is4KAligned Yes is4KAligned?</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">--rapid-repair Yes rapidRepair enabled?</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">--write-cache-threshold 0 HAWC Threshold (max 65536)</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">--subblocks-per-full-block 128 Number of subblocks per full block</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-P system Disk storage pools in file system</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">--file-audit-log No File Audit Logging enabled?</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">--maintenance-mode No Maintenance Mode enabled?</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-d home0;home10;home11;home12;home13;home14;home15;home16;home17;home18;home19;home1;home20;home21;home22;home23;home2;home3;home4;home5;home6;home7;home8;home9
Disks in file system</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-A yes Automatic mount option</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-o none Additional mount options</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">-T /gpfs/home Default mount point</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">--mount-priority 0 Mount priority</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">[root@nsd75-01 ~]#</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"> <o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"> <o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"> <o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Mit freundlichen Grüßen<b><i><br>
Walter Sklenka</i></b></span><b><i><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:gray"><br>
Technical Consultant</span></i></b><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"> </span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:8.0pt;font-family:"Arial",sans-serif">EDV-Design Informationstechnologie GmbH<br>
Giefinggasse 6/1/2, A-1210 Wien<br>
Tel: +43 1 29 22 165-31<br>
Fax: +43 1 29 22 165-90<br>
E-Mail: </span><a href="mailto:sklenka@edv-design.at" target="_blank"><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#0082BF">sklenka@edv-design.at</span></a><span style="font-size:8.0pt;font-family:"Arial",sans-serif"><br>
Internet: </span><a href="http://www.edv-design.at/" target="_blank"><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#0082BF">www.edv-design.at</span></a><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"> <o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><b>Von:</b>gpfsug-discuss-bounces@spectrumscale.org <<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">gpfsug-discuss-bounces@spectrumscale.org</a>>
<b>Im Auftrag von </b>José Filipe Higino<b><br>
Gesendet:</b> Saturday, February 8, 2020 1:00 PM<b><br>
An:</b> gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.org</a>><b><br>
Betreff:</b> Re: [gpfsug-discuss] mmdf and maybe other commands long running // influence of n and B on number of regions<o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"> <o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt">How many back end nodes for that cluster? and how many filesystems for that same access... and how many pools for the same data access type (12 ndisks sounds very LOW to me, for that size of a cluster, probably no
other filesystem can do more than that). On GPFS there are so many different ways to access the data, that is sometimes hard to start a conversation. And you did a very great job of introducing it. =)<o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"> <o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt">We (I am a customer too) do not have that many nodes, but from experience, I know some clusters (and also multicluster configs) depend mostly on how much metadata you can service in the network and how fast (latency
wise) you can do it, to accommodate such amount of nodes. There is never design by the book that can safely tell something will work 100% times. But the beauty of it is that GPFS allows lots of aspects to be resized at your convenience to facilitate what you
need most the system to do.<o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"> <o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt">Let us know more...<o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"> <o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt">On Sun, 9 Feb 2020 at 00:40, Walter Sklenka <<a href="mailto:Walter.Sklenka@edv-design.at">Walter.Sklenka@edv-design.at</a>> wrote:<o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt">Hello!<o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt">We are designing two fs where we cannot anticipate if there will be 3000, or maybe 5000 or more nodes totally accessing these filesystems<o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt">What we saw, was that execution time of mmdf can last 5-7min <o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt">We openend a case and they said, that during such commands like mmdf or also mmfsck, mmdefragfs,mmresripefs all regions must be scanned at this is the reason why it takes so long<o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt">The technichian also said, that it is “rule of thumb” that there should be
<o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt">(-n)*32 regions , this would then be enough ( N=5000
<span style="font-family:Wingdings">à</span>160000 regions per pool ?) <o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt">(also Block size has influence on regions ?)
<o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"> <o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt">#mmfsadm saferdump stripe <o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-family:"Courier New"">Gives the regions number
</span><o:p></o:p></p>
<p class="MsoNormal"><br>
<span style="font-family:"Courier New""> storage pools: max 8</span><br>
<span style="font-family:"Courier New""> </span><br>
<span style="font-family:"Courier New""> alloc map type 'scatter'</span><br>
<span style="font-family:"Courier New""> </span><br>
<span style="font-family:"Courier New""> 0: name 'system' Valid nDisks 12 nInUse 12 id 0 poolFlags 0 thinProvision reserved inode -1, reserved nBlocks 0</span><br>
<span style="font-family:"Courier New""> </span><br>
<span style="font-family:"Courier New""> <u>regns 170413</u> segs 1 size 4096 FBlks 0 MBlks 3145728 subblock size 8192</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"> <o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"> <o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"> <o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"> <o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"> <o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt">We also saw when creating the filesystem with a speciicic (-n) very high (5000) (where mmdf execution time was some minutes) and then changing (-n) to a lower value this does not influence the behavior any more<o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"> <o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt">My question is: Is the rule (Number of Nodes)x5000 for number of regios in a pool an good estimation , <o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt">Is it better to overestimate the number of Nodes (lnger running commands) or is it unrealistic to get into problems when not reaching the regions number calculated ?<o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"> <o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt">Does anybody have experience with high number of nodes (>>3000) and how to design the filesystems for such large clusters ?
<o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"> <o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt">Thank you very much in advance !<o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"> <o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"> <o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"> <o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Mit freundlichen Grüßen<b><i><br>
Walter Sklenka</i></b></span><b><i><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:gray"><br>
Technical Consultant</span></i></b><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"> </span><o:p></o:p></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:2.5in;margin-left:0in">
<span style="font-size:8.0pt;font-family:"Arial",sans-serif">EDV-Design Informationstechnologie GmbH<br>
Giefinggasse 6/1/2, A-1210 Wien<br>
Tel: +43 1 29 22 165-31<br>
Fax: +43 1 29 22 165-90<br>
E-Mail: </span><a href="mailto:sklenka@edv-design.at" target="_blank"><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#0082BF">sklenka@edv-design.at</span></a><span style="font-size:8.0pt;font-family:"Arial",sans-serif"><br>
Internet: </span><a href="http://www.edv-design.at/" target="_blank"><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#0082BF">www.edv-design.at</span></a><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"> <o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt">_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><u><span style="color:blue"><br>
</span></u><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
<tt><span style="font-size:10.0pt">_______________________________________________</span></tt><span style="font-size:10.0pt;font-family:"Courier New""><br>
<tt>gpfsug-discuss mailing list</tt><br>
<tt>gpfsug-discuss at spectrumscale.org</tt><br>
</span><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><span style="font-size:10.0pt">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></tt></a><span style="font-size:10.0pt;font-family:"Courier New""><br>
</span><br>
<br>
<o:p></o:p></p>
</div>
</body>
</html>