<div dir="ltr">How many back end nodes for that cluster? and how many filesystems for that same access... and how many pools for the same data access type (12 ndisks sounds very LOW to me, for that size of a cluster, probably no other filesystem can do more than that). On GPFS there are so many different ways to access the data, that is sometimes hard to start a conversation. And you did a very great job of introducing it. =)<div><br></div><div>We (I am a customer too) do not have that many nodes, but from experience, I know some clusters (and also multicluster configs) depend mostly on how much metadata you can service in the network and how fast (latency wise) you can do it, to accommodate such amount of nodes. There is never design by the book that can safely tell something will work 100% times. But the beauty of it is that GPFS allows lots of aspects to be resized at your convenience to facilitate what you need most the system to do.</div><div><br></div><div>Let us know more...</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, 9 Feb 2020 at 00:40, Walter Sklenka <<a href="mailto:Walter.Sklenka@edv-design.at">Walter.Sklenka@edv-design.at</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_-1312250387704777376WordSection1">
<p class="MsoNormal"><span lang="DE">Hello!<u></u><u></u></span></p>
<p class="MsoNormal">We are designing two fs where we cannot anticipate if there will be 3000, or maybe 5000 or more nodes totally accessing these filesystems<u></u><u></u></p>
<p class="MsoNormal">What we saw, was that execution time of mmdf can last 5-7min <u></u><u></u></p>
<p class="MsoNormal">We openend a case and they said, that during such commands like mmdf or also mmfsck, mmdefragfs,mmresripefs all regions must be scanned at this is the reason why it takes so long<u></u><u></u></p>
<p class="MsoNormal">The technichian also said, that it is “rule of thumb” that there should be
<u></u><u></u></p>
<p class="MsoNormal">(-n)*32 regions , this would then be enough ( N=5000 <span style="font-family:Wingdings">
à</span> 160000 regions per pool ?) <u></u><u></u></p>
<p class="MsoNormal">(also Block size has influence on regions ?) <u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">#mmfsadm saferdump stripe <u></u><u></u></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">Gives the regions number
<u></u><u></u></span></p>
<p class="gmail-m_-1312250387704777376MsoPlainText"><span style="font-family:"Courier New""> storage pools: max 8<u></u><u></u></span></p>
<p class="gmail-m_-1312250387704777376MsoPlainText"><span style="font-family:"Courier New""><u></u> <u></u></span></p>
<p class="gmail-m_-1312250387704777376MsoPlainText"><span style="font-family:"Courier New""> alloc map type 'scatter'<u></u><u></u></span></p>
<p class="gmail-m_-1312250387704777376MsoPlainText"><span style="font-family:"Courier New""><u></u> <u></u></span></p>
<p class="gmail-m_-1312250387704777376MsoPlainText"><span style="font-family:"Courier New""> 0: name 'system' Valid nDisks 12 nInUse 12 id 0 poolFlags 0 thinProvision reserved inode -1, reserved nBlocks 0<u></u><u></u></span></p>
<p class="gmail-m_-1312250387704777376MsoPlainText"><span style="font-family:"Courier New""><u></u> <u></u></span></p>
<p class="gmail-m_-1312250387704777376MsoPlainText"><span style="font-family:"Courier New""> <u>regns 170413</u> segs 1 size 4096 FBlks 0 MBlks 3145728 subblock size 8192<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">We also saw when creating the filesystem with a speciicic (-n) very high (5000) (where mmdf execution time was some minutes) and then changing (-n) to a lower value this does not influence the behavior any more<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">My question is: Is the rule (Number of Nodes)x5000 for number of regios in a pool an good estimation , <u></u><u></u></p>
<p class="MsoNormal">Is it better to overestimate the number of Nodes (lnger running commands) or is it unrealistic to get into problems when not reaching the regions number calculated ?<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Does anybody have experience with high number of nodes (>>3000) and how to design the filesystems for such large clusters ?
<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thank you very much in advance !<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span lang="DE-AT" style="font-size:10pt;font-family:Arial,sans-serif;color:black">Mit freundlichen Grüßen<br>
<b><i>Walter Sklenka</i></b><br>
</span><b><i><span lang="DE-AT" style="font-size:8pt;font-family:Arial,sans-serif;color:gray">Technical Consultant</span></i></b><span lang="DE-AT" style="color:black">
<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="DE-AT" style="font-size:12pt;font-family:"Times New Roman",serif;color:black"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="DE-AT" style="font-size:8pt;font-family:Arial,sans-serif;color:black">EDV-Design Informationstechnologie GmbH<br>
Giefinggasse 6/1/2, A-1210 Wien<br>
Tel: +43 1 29 22 165-31<br>
Fax: +43 1 29 22 165-90<br>
E-Mail: <a href="mailto:sklenka@edv-design.at" target="_blank"><span style="color:rgb(5,99,193)">sklenka@edv-design.at</span></a></span><span lang="DE-AT" style="font-size:8pt;color:black"><br>
</span><span lang="DE-AT" style="font-size:8pt;font-family:Arial,sans-serif;color:black">Internet:
<a href="http://www.edv-design.at/" target="_blank"><span style="color:rgb(5,99,193)">www.edv-design.at</span></a></span><i><span lang="DE-AT" style="font-size:8pt;color:black"><br>
<br>
</span></i><span lang="DE"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="DE"><u></u> <u></u></span></p>
</div>
</div>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
</blockquote></div>