<div dir="auto"><div>Unfortunate more complicated :)<div dir="auto"><br><div dir="auto">The consumption here is an estimate based on 512b inodes, which no newly created filesystem has as all new default to 4k. So if you have 4k inodes you could easily need 2x of the estimated value.</div><div dir="auto">Then there are extended attributes, also not added here, etc .</div><div dir="auto">So don't take this number as usage, it's really just a rough estimate.</div><div dir="auto"><br></div><div dir="auto">Sven</div><div dir="auto"><br></div></div><br><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, May 9, 2019, 5:53 PM Achim Rehor <<a href="mailto:Achim.Rehor@de.ibm.com">Achim.Rehor@de.ibm.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><font size="2" face="sans-serif">Sorry for my fast ( and not well thought)
answer, before. You obviously are correct, there is no relation between
the setting of maxFilesToCache, and the </font><br><br><font size="3">fileCacheMem 38359956 k = 11718554
* 3352 bytes (inode size 512 + 2840)</font><br><br><font size="2" face="sans-serif">usage. it is rather a statement of how
many metadata may fit in the remaining structures outside the pagepool.
this value does not change at all, when you modify your mFtC setting. </font><br><br><font size="2" face="sans-serif">There is a really good presentation
by Tomer Perry on the User Group meetings, referring about memory footprint
of GPFS under various conditions.</font><br><br><font size="2" face="sans-serif">In your case, you may very well hit
the CES nodes memleak you just pointed out. </font><br><br><font size="2" face="sans-serif">Sorry for my hasty reply earlier ;)
</font><br><br><font size="2" face="sans-serif">Achim</font><br><br><br><br><font size="1" color="#5f5f5f" face="sans-serif">From:
</font><font size="1" face="sans-serif">Stijn De Weirdt <<a href="mailto:stijn.deweirdt@ugent.be" target="_blank" rel="noreferrer">stijn.deweirdt@ugent.be</a>></font><br><font size="1" color="#5f5f5f" face="sans-serif">To:
</font><font size="1" face="sans-serif"><a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank" rel="noreferrer">gpfsug-discuss@spectrumscale.org</a></font><br><font size="1" color="#5f5f5f" face="sans-serif">Date:
</font><font size="1" face="sans-serif">09/05/2019 16:48</font><br><font size="1" color="#5f5f5f" face="sans-serif">Subject:
</font><font size="1" face="sans-serif">Re: [gpfsug-discuss]
advanced filecache math</font><br><font size="1" color="#5f5f5f" face="sans-serif">Sent by:
</font><font size="1" face="sans-serif"><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank" rel="noreferrer">gpfsug-discuss-bounces@spectrumscale.org</a></font><br><hr noshade><br><br><br><tt><font size="2">seems like we are suffering from<br></font></tt><a href="http://www-01.ibm.com/support/docview.wss?uid=isg1IJ12737" target="_blank" rel="noreferrer"><tt><font size="2">http://www-01.ibm.com/support/docview.wss?uid=isg1IJ12737</font></tt></a><tt><font size="2"><br><br>as these are ces nodes, we susepcted something wrong the caches, but it<br>looks like a memleak instead.<br><br>sorry for the noise (as usual you find the solution right after sending<br>the mail ;)<br><br>stijn<br><br>On 5/9/19 4:38 PM, Stijn De Weirdt wrote:<br>> hi achim,<br>> <br>>> you just misinterpreted the term fileCacheLimit.<br>>> This is not in byte, but specifies the maxFilesToCache setting
:<br>> i understand that, but how does the fileCacheLimit relate to the<br>> fileCacheMem number?<br>> <br>> <br>> <br>> (we have a 32GB pagepool, and mmfsd is using 80GB RES (101 VIRT),
so we<br>> are looking for large numbers that might explain wtf is going on<br>> (pardon my french ;)<br>> <br>> stijn<br>> <br>>><br>>> UMALLOC limits:<br>>> bufferDescLimit 40000
desired 40000<br>>> fileCacheLimit 4000 desired
4000 <=== mFtC<br>>> statCacheLimit 1000 desired
1000 <=== mSC<br>>> diskAddrBuffLimit 200
desired 200<br>>><br>>> # mmfsadm dump config | grep -E "maxFilesToCache|maxStatCache"<br>>> maxFilesToCache 4000<br>>> maxStatCache 1000<br>>><br>>> Mit freundlichen Grüßen / Kind regards<br>>><br>>> *Achim Rehor*<br>>><br>>> --------------------------------------------------------------------------------<br>>> Software Technical Support Specialist AIX/ Emea HPC Support
<br>>> IBM Certified Advanced Technical Expert - Power Systems with AIX<br>>> TSCC Software Service, Dept. 7922<br>>> Global Technology Services<br>>> --------------------------------------------------------------------------------<br>>> Phone:
+49-7034-274-7862
IBM Deutschland<br>>> E-Mail:
<a href="mailto:Achim.Rehor@de.ibm.com" target="_blank" rel="noreferrer">Achim.Rehor@de.ibm.com</a>
Am Weiher 24<br>>>
65451 Kelsterbach<br>>>
Germany<br>>>
<br>>> --------------------------------------------------------------------------------<br>>> IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin
Jetter<br>>> Geschäftsführung: Martin Hartmann (Vorsitzender), Norbert Janzen,
Stefan Lutz, <br>>> Nicole Reimer, Dr. Klaus Seifert, Wolfgang Wendt<br>>> Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht
Stuttgart, HRB <br>>> 14562 WEEE-Reg.-Nr. DE 99369940<br>>><br>>><br>>><br>>><br>>><br>>><br>>> From: Stijn De Weirdt <<a href="mailto:stijn.deweirdt@ugent.be" target="_blank" rel="noreferrer">stijn.deweirdt@ugent.be</a>><br>>> To: gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank" rel="noreferrer">gpfsug-discuss@spectrumscale.org</a>><br>>> Date: 09/05/2019 16:21<br>>> Subject: [gpfsug-discuss] advanced filecache math<br>>> Sent by: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank" rel="noreferrer">gpfsug-discuss-bounces@spectrumscale.org</a><br>>><br>>> --------------------------------------------------------------------------------<br>>><br>>><br>>><br>>> hi all,<br>>><br>>> we are looking into some memory issues with gpfs 5.0.2.2, and
found<br>>> following in mmfsadm dump fs:<br>>><br>>> > fileCacheLimit 1000000
desired 1000000<br>>> ...<br>>> > fileCacheMem 38359956 k
= 11718554 * 3352 bytes (inode size 512 + 2840)<br>>><br>>> the limit is 1M (we configured that), however, the fileCacheMem
mentions<br>>> 11.7M?<br>>><br>>> this is also reported right after a mmshutdown/startup.<br>>><br>>> how do these 2 relate (again?)?<br>>><br>>> mnay thanks,<br>>><br>>> stijn<br>>> _______________________________________________<br>>> gpfsug-discuss mailing list<br>>> gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank" rel="noreferrer">spectrumscale.org</a><br>>> </font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank" rel="noreferrer"><tt><font size="2">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size="2"><br>>><br>>><br>>><br>>><br>>><br>>> _______________________________________________<br>>> gpfsug-discuss mailing list<br>>> gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank" rel="noreferrer">spectrumscale.org</a><br>>> </font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank" rel="noreferrer"><tt><font size="2">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size="2"><br>>><br>> _______________________________________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank" rel="noreferrer">spectrumscale.org</a><br>> </font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank" rel="noreferrer"><tt><font size="2">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size="2"><br>> <br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank" rel="noreferrer">spectrumscale.org</a><br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank" rel="noreferrer"><tt><font size="2">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size="2"><br><br></font></tt><br><br><br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
</blockquote></div></div></div>