<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div dir="ltr" >Perhaps this presentation can help you:</div>
<div dir="ltr" ><a href="https://pt.slideshare.net/tomerperry/ibm-spectrum-scale-memory-usage">https://pt.slideshare.net/tomerperry/ibm-spectrum-scale-memory-usage</a></div>
<div dir="ltr" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div dir="ltr" ><div> </div>
<table cellpadding="0" cellspacing="0" style="border:0;table-layout:auto;white-space:normal;color:#5f5f5f;width:650px;font-size:7.2pt" width="650px" > <tbody> <tr> <td style="vertical-align: bottom;" > <div style="font-size:9pt" ><span style="font-size:1.000em;" >Abraços / Regards / Saludos,</span></div>
<p> </p>
<div class="vcard" style="margin:0 0 8px 0" ><span style="font-size:1.000em;" ><b class="fn n" style="color:#888888;font-size:12pt" ><span class="given-name" >Anderson</span> <span class="family-name" >Nobre</span></b><br> Power and Storage Consultant<br> IBM Systems Hardware Client Technical Team – IBM Systems Lab Services<br> <br> <img alt="community_general_lab_services" class="org" height="120" src="http://esign.webmaster.ibm.com/img/signature_lab_services.jpg" width="333" ></span></div> </td> <td> </td> </tr> </tbody></table>
<table cellpadding="0" cellspacing="0" style="border:0;table-layout:auto;white-space:normal;color:#5f5f5f;width:650px;font-size:7.2pt" width="650px" > <tbody> <tr> <td colspan="3" > <hr style="border:0;color:#696969;background-color:#696969;height:1px;" ></td> </tr> <tr> <td style="vertical-align: top;font-family:arial,helvetica,sans-serif" width="400px" ><span style="font-size:1.000em;" ><span class="tel" ><span style="color:#466BB0;font-weight:bold" ><abbr class="type" title="work" >Phone:</abbr></span> <span class="value" >55-19-2132-4317</span></span><br> <span style="color:#466BB0;font-weight:bold;font-style:7.2pt" >E-mail:</span> <a class="email" href="mailto:anobre@br.ibm.com" target="_blank"><span class="value" style="color:#5f5f5f" >anobre@br.ibm.com</span></a></span></td> <td align="right" style="vertical-align: top;font-family:arial,helvetica,sans-serif" width="150px" ><span style="font-size:1.000em;" ><img alt="IBM" class="org" src="http://esign.webmaster.ibm.com/img/ibm3.gif" ></span></td> </tr> </tbody></table></div></div></div></div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Original message -----<br>From: Jim Doherty <jjdoherty@yahoo.com><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Cc:<br>Subject: [EXTERNAL] Re: [gpfsug-discuss] advanced filecache math<br>Date: Thu, May 9, 2019 6:08 PM<br>
<div style="font-family: Helvetica Neue, Helvetica, Arial, sans-serif; font-size: 13px;" ><div id="ydp650340feyiv2554603969" ><div><div style="font-family:Helvetica Neue, Helvetica, Arial, sans-serif;font-size:13px;" ><div> </div>
<div> </div>
<div>A couple of observations on memory, a maxFilesToCache object takes anwhere from 6-10K, so 1 million =~ 6-10 Gig. Memory utilized in the mmfsd comes from either the pagepool, the shared memory segment used by MFTC objects, the token memory segment used to track MFTC objects, and newer is memory used by AFM. If the memory resources are in the mmfsd address space then this will show in the RSS size of the mmfsd. Ignore the VMM size, since the glibc change awhile back to allocate a heap for each thread VMM has become an imaginary number for a large multi-threaded application. </div>
<div> </div>
<div>There have been some memory leaks fixed in Ganesha that will be in 4.2.3 PTF15 which is available on fixcentral</div>
<div> </div>
<div> </div>
<div>Jim Doherty</div>
<div> </div></div>
<div id="ydp650340feyiv2554603969yahoo_quoted_8190556182" ><div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;" > </div></div></div></div></div>
<div id="yiv2554603969yqt98669" ><div>On Thursday, May 9, 2019, 1:25:03 PM EDT, Sven Oehme <oehmes@gmail.com> wrote:</div>
<div> </div>
<div> </div>
<div><div id="yiv2554603969" ><div><div><div>Unfortunate more complicated :)
<div>
<div>The consumption here is an estimate based on 512b inodes, which no newly created filesystem has as all new default to 4k. So if you have 4k inodes you could easily need 2x of the estimated value.</div>
<div>Then there are extended attributes, also not added here, etc .</div>
<div>So don't take this number as usage, it's really just a rough estimate.</div>
<div> </div>
<div>Sven</div>
<div> </div></div>
<div><div id="yiv2554603969yqtfd60423" ><div dir="ltr" >On Thu, May 9, 2019, 5:53 PM Achim Rehor <<a href="mailto:Achim.Rehor@de.ibm.com" rel="nofollow" shape="rect" target="_blank" ymailto="mailto:Achim.Rehor@de.ibm.com">Achim.Rehor@de.ibm.com</a>> wrote:</div>
<blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;" ><font face="sans-serif" size="2" >Sorry for my fast ( and not well thought) answer, before. You obviously are correct, there is no relation between the setting of maxFilesToCache, and the </font><br clear="none" ><br clear="none" ><font size="3" >fileCacheMem 38359956 k = 11718554 * 3352 bytes (inode size 512 + 2840)</font><br clear="none" ><br clear="none" ><font face="sans-serif" size="2" >usage. it is rather a statement of how many metadata may fit in the remaining structures outside the pagepool. this value does not change at all, when you modify your mFtC setting. </font><br clear="none" ><br clear="none" ><font face="sans-serif" size="2" >There is a really good presentation by Tomer Perry on the User Group meetings, referring about memory footprint of GPFS under various conditions.</font><br clear="none" ><br clear="none" ><font face="sans-serif" size="2" >In your case, you may very well hit the CES nodes memleak you just pointed out. </font><br clear="none" ><br clear="none" ><font face="sans-serif" size="2" >Sorry for my hasty reply earlier ;) </font><br clear="none" ><br clear="none" ><font face="sans-serif" size="2" >Achim</font><br clear="none" ><br clear="none" ><br clear="none" ><br clear="none" ><font color="#5f5f5f" face="sans-serif" size="1" >From: </font><font face="sans-serif" size="1" >Stijn De Weirdt <<a href="mailto:stijn.deweirdt@ugent.be" rel="nofollow" shape="rect" target="_blank" ymailto="mailto:stijn.deweirdt@ugent.be">stijn.deweirdt@ugent.be</a>></font><br clear="none" ><font color="#5f5f5f" face="sans-serif" size="1" >To: </font><font face="sans-serif" size="1" ><a href="mailto:gpfsug-discuss@spectrumscale.org" rel="nofollow" shape="rect" target="_blank" ymailto="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.org</a></font><br clear="none" ><font color="#5f5f5f" face="sans-serif" size="1" >Date: </font><font face="sans-serif" size="1" >09/05/2019 16:48</font><br clear="none" ><font color="#5f5f5f" face="sans-serif" size="1" >Subject: </font><font face="sans-serif" size="1" >Re: [gpfsug-discuss] advanced filecache math</font><br clear="none" ><font color="#5f5f5f" face="sans-serif" size="1" >Sent by: </font><font face="sans-serif" size="1" ><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" rel="nofollow" shape="rect" target="_blank" ymailto="mailto:gpfsug-discuss-bounces@spectrumscale.org">gpfsug-discuss-bounces@spectrumscale.org</a></font>
<hr noshade="noshade" ><br clear="none" ><br clear="none" ><br clear="none" ><tt><font face="" size="3" >seems like we are suffering from</font></tt><br clear="none" ><tt><font face="" size="3" ><a href="http://www-01.ibm.com/support/docview.wss?uid=isg1IJ12737" target="_blank">http://www-01.ibm.com/support/docview.wss?uid=isg1IJ12737</a></font></tt><br clear="none" ><br clear="none" ><tt><font face="" size="3" >as these are ces nodes, we susepcted something wrong the caches, but it<br clear="none" >looks like a memleak instead.<br clear="none" ><br clear="none" >sorry for the noise (as usual you find the solution right after sending<br clear="none" >the mail ;)<br clear="none" ><br clear="none" >stijn<br clear="none" ><br clear="none" >On 5/9/19 4:38 PM, Stijn De Weirdt wrote:<br clear="none" >> hi achim,<br clear="none" >><br clear="none" >>> you just misinterpreted the term fileCacheLimit.<br clear="none" >>> This is not in byte, but specifies the maxFilesToCache setting :<br clear="none" >> i understand that, but how does the fileCacheLimit relate to the<br clear="none" >> fileCacheMem number?<br clear="none" >><br clear="none" >><br clear="none" >><br clear="none" >> (we have a 32GB pagepool, and mmfsd is using 80GB RES (101 VIRT), so we<br clear="none" >> are looking for large numbers that might explain wtf is going on<br clear="none" >> (pardon my french ;)<br clear="none" >><br clear="none" >> stijn<br clear="none" >><br clear="none" >>><br clear="none" >>> UMALLOC limits:<br clear="none" >>> bufferDescLimit 40000 desired 40000<br clear="none" >>> fileCacheLimit 4000 desired 4000 <=== mFtC<br clear="none" >>> statCacheLimit 1000 desired 1000 <=== mSC<br clear="none" >>> diskAddrBuffLimit 200 desired 200<br clear="none" >>><br clear="none" >>> # mmfsadm dump config | grep -E "maxFilesToCache|maxStatCache"<br clear="none" >>> maxFilesToCache 4000<br clear="none" >>> maxStatCache 1000<br clear="none" >>><br clear="none" >>> Mit freundlichen Grüßen / Kind regards<br clear="none" >>><br clear="none" >>> *Achim Rehor*<br clear="none" >>><br clear="none" >>> --------------------------------------------------------------------------------<br clear="none" >>> Software Technical Support Specialist AIX/ Emea HPC Support <br clear="none" >>> IBM Certified Advanced Technical Expert - Power Systems with AIX<br clear="none" >>> TSCC Software Service, Dept. 7922<br clear="none" >>> Global Technology Services<br clear="none" >>> --------------------------------------------------------------------------------<br clear="none" >>> Phone: +49-7034-274-7862 IBM Deutschland<br clear="none" >>> E-Mail: <a href="mailto:Achim.Rehor@de.ibm.com" rel="nofollow" shape="rect" target="_blank" ymailto="mailto:Achim.Rehor@de.ibm.com">Achim.Rehor@de.ibm.com</a> Am Weiher 24<br clear="none" >>> 65451 Kelsterbach<br clear="none" >>> Germany<br clear="none" >>> <br clear="none" >>> --------------------------------------------------------------------------------<br clear="none" >>> IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter<br clear="none" >>> Geschäftsführung: Martin Hartmann (Vorsitzender), Norbert Janzen, Stefan Lutz,<br clear="none" >>> Nicole Reimer, Dr. Klaus Seifert, Wolfgang Wendt<br clear="none" >>> Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB<br clear="none" >>> 14562 WEEE-Reg.-Nr. DE 99369940<br clear="none" >>><br clear="none" >>><br clear="none" >>><br clear="none" >>><br clear="none" >>><br clear="none" >>><br clear="none" >>> From: Stijn De Weirdt <<a href="mailto:stijn.deweirdt@ugent.be" rel="nofollow" shape="rect" target="_blank" ymailto="mailto:stijn.deweirdt@ugent.be">stijn.deweirdt@ugent.be</a>><br clear="none" >>> To: gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" rel="nofollow" shape="rect" target="_blank" ymailto="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.org</a>><br clear="none" >>> Date: 09/05/2019 16:21<br clear="none" >>> Subject: [gpfsug-discuss] advanced filecache math<br clear="none" >>> Sent by: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" rel="nofollow" shape="rect" target="_blank" ymailto="mailto:gpfsug-discuss-bounces@spectrumscale.org">gpfsug-discuss-bounces@spectrumscale.org</a><br clear="none" >>><br clear="none" >>> --------------------------------------------------------------------------------<br clear="none" >>><br clear="none" >>><br clear="none" >>><br clear="none" >>> hi all,<br clear="none" >>><br clear="none" >>> we are looking into some memory issues with gpfs 5.0.2.2, and found<br clear="none" >>> following in mmfsadm dump fs:<br clear="none" >>><br clear="none" >>> > fileCacheLimit 1000000 desired 1000000<br clear="none" >>> ...<br clear="none" >>> > fileCacheMem 38359956 k = 11718554 * 3352 bytes (inode size 512 + 2840)<br clear="none" >>><br clear="none" >>> the limit is 1M (we configured that), however, the fileCacheMem mentions<br clear="none" >>> 11.7M?<br clear="none" >>><br clear="none" >>> this is also reported right after a mmshutdown/startup.<br clear="none" >>><br clear="none" >>> how do these 2 relate (again?)?<br clear="none" >>><br clear="none" >>> mnay thanks,<br clear="none" >>><br clear="none" >>> stijn<br clear="none" >>> _______________________________________________<br clear="none" >>> gpfsug-discuss mailing list<br clear="none" >>> gpfsug-discuss at <a href="http://spectrumscale.org" rel="nofollow" shape="rect" target="_blank">spectrumscale.org</a><br clear="none" >>> </font></tt><tt><font face="" size="3" ><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></tt><br clear="none" ><tt><font face="" size="3" >>><br clear="none" >>><br clear="none" >>><br clear="none" >>><br clear="none" >>><br clear="none" >>> _______________________________________________<br clear="none" >>> gpfsug-discuss mailing list<br clear="none" >>> gpfsug-discuss at <a href="http://spectrumscale.org" rel="nofollow" shape="rect" target="_blank">spectrumscale.org</a><br clear="none" >>> </font></tt><tt><font face="" size="3" ><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></tt><br clear="none" ><tt><font face="" size="3" >>><br clear="none" >> _______________________________________________<br clear="none" >> gpfsug-discuss mailing list<br clear="none" >> gpfsug-discuss at <a href="http://spectrumscale.org" rel="nofollow" shape="rect" target="_blank">spectrumscale.org</a><br clear="none" >> </font></tt><tt><font face="" size="3" ><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></tt><br clear="none" ><tt><font face="" size="3" >><br clear="none" >_______________________________________________<br clear="none" >gpfsug-discuss mailing list<br clear="none" >gpfsug-discuss at <a href="http://spectrumscale.org" rel="nofollow" shape="rect" target="_blank">spectrumscale.org</a></font></tt><br clear="none" ><tt><font face="" size="3" ><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></tt><br clear="none" ><br clear="none" ><br clear="none" ><br clear="none" ><br clear="none" >_______________________________________________<br clear="none" >gpfsug-discuss mailing list<br clear="none" >gpfsug-discuss at <a href="http://spectrumscale.org" rel="nofollow" shape="rect" target="_blank">spectrumscale.org</a><br clear="none" ><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="nofollow" shape="rect" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></blockquote></div></div></div></div></div></div>
<div id="yiv2554603969yqtfd02092" >_______________________________________________<br clear="none" >gpfsug-discuss mailing list<br clear="none" >gpfsug-discuss at spectrumscale.org<br clear="none" ><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="nofollow" shape="rect" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></div></div></div>
<div><font face="Default Monospace,Courier New,Courier,monospace" size="2" >_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></div></blockquote>
<div dir="ltr" > </div></div><BR>