[gpfsug-discuss] frequent OOM killer due to high memory usage of mmfsd

[Stephan Graf]

Hi

in the past we had issues with the mmdf heap memory. Due to special 
workload it increased and took GB of memory, but after usage it was not 
freed again.
we had long discussions with IBM about it and it ends up in a 
Development User Story (261213) which was realized in 5.1.2:
---
In this story, the InodeAllocSegment object will be allocated when accessed.
For commands that iterates all InodeAllocSegment, we will release the object
immediately after use.
An undocumented configuration "!maxIAllocSegmentsToCache" is provided to 
control
the upper limit of the count of InodeAllocSegment objects. When the count
approaches the limit, a pre stealing thread will be started to steal and
release some InodeAllocSegment objects. Its default value is 1000,000.
---

since than we are fine so far. But this was on plain GPFS clients, no 
CES node where the service like NFS comes into play.

You can monitor the heap memory usage by using "mmdiag --memory"

@IBM colleagues: If there is something wrong in my explanation please 
correct me.

Stephan

On 9/6/23 20:55, Christoph Martin wrote:
> Hi all,
> 
> on a three node GPFS cluster with CES enabled and AFM-DR mirroring to a 
> second cluster we see frequent OOM killer events due to a constantly 
> growing mmfsd.
> The machines have 256G memory. The pagepool is configured to 16G.
> The GPFS version is 5.1.6-1.
> After a restart mmfsd rapidly grows to about 100G usage and grows over 
> some days up to 250G virtual and 220G physical memory usage.
> OOMkiller tries kill process like pmcollector or others and sometime 
> kills mmfsd.
> 
> Does anybody see a similar behavior?
> Any guess what could help with this problem?
> 
> Regards
> Christoph Martin
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org

-- 
Stephan Graf
Juelich Supercomputing Centre

Phone:  +49-2461-61-6578
Fax:    +49-2461-61-6656
E-mail: st.graf at fz-juelich.de
WWW:    http://www.fz-juelich.de/jsc/
---------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Dr. Astrid Lambrecht,
Prof. Dr. Frauke Melchior
---------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5938 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230907/79f8b629/attachment.bin>