<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
smem is recommended here
<div class=""><br class="">
</div>
<div class="">Cheers,</div>
<div class="">Adam</div>
<div class=""><br class="">
<div class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
--<br class="">
<br class="">
Adam Huffman<br class="">
Senior HPC and Cloud Systems Engineer<br class="">
The Francis Crick Institute<br class="">
1 Midland Road<br class="">
London NW1 1AT<br class="">
<br class="">
T: 020 3796 1175<br class="">
E: <a href="mailto:adam.huffman@crick.ac.uk" class="">adam.huffman@crick.ac.uk</a><br class="">
W: <a href="http://www.crick.ac.uk" class="">www.crick.ac.uk</a><br class="">
<br class="">
<br class="">
<br class="">
<br class="">
</div>
</div>
</div>
</div>
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On 24 Jul 2017, at 15:21, Peter Childs <<a href="mailto:p.childs@qmul.ac.uk" class="">p.childs@qmul.ac.uk</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">
<div class=""><br class="">
</div>
<div class="">top</div>
<div class=""><br class="">
</div>
<div class="">but ps gives the same value.</div>
<div class=""><br class="">
</div>
<div class="">[<a href="mailto:root@dn29" class="">root@dn29</a> ~]# ps auww -q 4444</div>
<div class="">USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND</div>
<div class="">root 4444 2.7 22.3 10537600 5472580 ? S<Ll Jul12 466:13 /usr/lpp/mmfs/bin/mmfsd</div>
<div class=""></div>
<div class=""><br class="">
</div>
<div class="">Thanks for the help</div>
<div class=""><br class="">
</div>
<div class="">Peter.</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">On Mon, 2017-07-24 at 14:10 +0000, Jim Doherty wrote:</div>
<blockquote type="cite" class="">
<div style="background-color: rgb(255, 255, 255); font-family: 'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif; font-size: 13px;" class="">
How are you identifying the high memory usage? <br class="">
<div id="yui_3_16_0_ym19_1_1500905216021_5198" class=""><span class=""></span></div>
<div class="qtdSeparateBR"><br class="">
<br class="">
</div>
<div class="yahoo_quoted" style="display: block;">
<div style="font-family: Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 13px;" class="">
<div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 16px;" class="">
<div dir="ltr" class=""><font size="2" face="Arial" class="">On Monday, July 24, 2017 9:30 AM, Peter Childs <<a href="mailto:p.childs@qmul.ac.uk" class="">p.childs@qmul.ac.uk</a>> wrote:<br class="">
</font></div>
<br class="">
<br class="">
<div class="y_msg_container">
<div id="yiv0922426330" class="">
<div class="">
<div class="">I've had a look at mmfsadm dump malloc and it looks to agree with the output from mmdiag --memory. and does not seam to account for the excessive memory usage.</div>
<div class=""><br clear="none" class="">
</div>
<div class="">The new machines do have idleSocketTimout set to 0 from what your saying it could be related to keeping that many connections between nodes working.</div>
<div class=""><br clear="none" class="">
</div>
<div class="">Thanks in advance</div>
<div class=""><br clear="none" class="">
</div>
<div class="">Peter.</div>
<div class=""><br clear="none" class="">
</div>
<div class=""><br clear="none" class="">
</div>
<div class=""><br clear="none" class="">
</div>
<div class=""><br clear="none" class="">
</div>
<div class="">[<a rel="nofollow" shape="rect" ymailto="mailto:root@dn29" target="_blank" href="mailto:root@dn29" class="">root@dn29</a> ~]# mmdiag --memory</div>
<div class=""><br clear="none" class="">
</div>
<div class="">=== mmdiag: memory ===</div>
<div class="">mmfsd heap size: 2039808 bytes</div>
<div class=""><br clear="none" class="">
</div>
<div class=""><br clear="none" class="">
</div>
<div class="">Statistics for MemoryPool id 1 ("Shared Segment (EPHEMERAL)")</div>
<div class=""> 128 bytes in use</div>
<div class=""> 17500049370 hard limit on memory usage</div>
<div class=""> 1048576 bytes committed to regions</div>
<div class=""> 1 number of regions</div>
<div class=""> 555 allocations</div>
<div class=""> 555 frees</div>
<div class=""> 0 allocation failures</div>
<div class=""><br clear="none" class="">
</div>
<div class=""><br clear="none" class="">
</div>
<div class="">Statistics for MemoryPool id 2 ("Shared Segment")</div>
<div class=""> 42179592 bytes in use</div>
<div class=""> 17500049370 hard limit on memory usage</div>
<div class=""> 56623104 bytes committed to regions</div>
<div class=""> 9 number of regions</div>
<div class=""> 100027 allocations</div>
<div class=""> 79624 frees</div>
<div class=""> 0 allocation failures</div>
<div class=""><br clear="none" class="">
</div>
<div class=""><br clear="none" class="">
</div>
<div class="">Statistics for MemoryPool id 3 ("Token Manager")</div>
<div class=""> 2099520 bytes in use</div>
<div class=""> 17500049370 hard limit on memory usage</div>
<div class=""> 16778240 bytes committed to regions</div>
<div class=""> 1 number of regions</div>
<div class=""> 4 allocations</div>
<div class=""> 0 frees</div>
<div class=""> 0 allocation failures</div>
<div class=""><br clear="none" class="">
</div>
<div class=""><br clear="none" class="">
</div>
<div class=""></div>
<div class="yiv0922426330yqt1081544495" id="yiv0922426330yqt39566">
<div class="">On Mon, 2017-07-24 at 13:11 +0000, Jim Doherty wrote:</div>
<blockquote type="cite" class="">
<div style="background-color: rgb(255, 255, 255); font-family: 'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif; font-size: 13px;" class="">
<div id="yiv0922426330" class="">
<div id="yiv0922426330yui_3_16_0_ym19_1_1500900502762_16662" class="">
<div id="yiv0922426330yui_3_16_0_ym19_1_1500900502762_16661" style="background-color: rgb(255, 255, 255); font-family: 'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif; font-size: 13px;" class="">
<div id="yiv0922426330yui_3_16_0_ym19_1_1500900502762_12131" class="">There are 3 places that the GPFS mmfsd uses memory the pagepool plus 2 shared memory segments. To see the memory utilization of the shared memory segments run the command mmfsadm dump
malloc . The statistics for memory pool id 2 is where maxFilesToCache/maxStatCache objects are and the manager nodes use memory pool id 3 to track the MFTC/MSC objects.
<br clear="none" class="">
</div>
<div id="yiv0922426330yui_3_16_0_ym19_1_1500900502762_13732" class=""><br clear="none" class="">
</div>
<div id="yiv0922426330yui_3_16_0_ym19_1_1500900502762_13733" class="">You might want to upgrade to later PTF as there was a PTF to fix a memory leak that occurred in tscomm associated with network connection drops.
<br clear="none" class="">
</div>
<div id="yiv0922426330yui_3_16_0_ym19_1_1500900502762_7195" class=""><span class=""></span></div>
<div class="yiv0922426330qtdSeparateBR" id="yiv0922426330yui_3_16_0_ym19_1_1500900502762_16707">
<br clear="none" class="">
<br clear="none" class="">
</div>
<div class="yiv0922426330yqt2983717346" id="yiv0922426330yqt44105"></div>
</div>
</div>
</div>
<div class="">
<div style="font-family:Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:13px;" class="">
<div style="font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:16px;" class="">
<div dir="ltr" class=""><font size="2" face="Arial" class="">On Monday, July 24, 2017 5:29 AM, Peter Childs <<a href="mailto:p.childs@qmul.ac.uk" class="">p.childs@qmul.ac.uk</a>> wrote:<br clear="none" class="">
</font></div>
<br clear="none" class="">
<br clear="none" class="">
<div class="yiv0922426330y_msg_container">
<div dir="ltr" class="">We have two GPFS clusters.<br clear="none" class="">
</div>
<div dir="ltr" class=""><br clear="none" class="">
</div>
<div dir="ltr" class="">One is fairly old and running 4.2.1-2 and non CCR and the nodes run<br clear="none" class="">
</div>
<div dir="ltr" class="">fine using up about 1.5G of memory and is consistent (GPFS pagepool is<br clear="none" class="">
</div>
<div dir="ltr" class="">set to 1G, so that looks about right.)<br clear="none" class="">
</div>
<div dir="ltr" class=""><br clear="none" class="">
</div>
<div dir="ltr" class="">The other one is "newer" running 4.2.1-3 with CCR and the nodes keep<br clear="none" class="">
</div>
<div dir="ltr" class="">increasing in there memory usage, starting at about 1.1G and are find<br clear="none" class="">
</div>
<div dir="ltr" class="">for a few days however after a while they grow to 4.2G which when the<br clear="none" class="">
</div>
<div dir="ltr" class="">node need to run real work, means the work can't be done.<br clear="none" class="">
</div>
<div dir="ltr" class=""><br clear="none" class="">
</div>
<div dir="ltr" class="">I'm losing track of what maybe different other than CCR, and I'm trying<br clear="none" class="">
</div>
<div dir="ltr" class="">to find some more ideas of where to look.<br clear="none" class="">
</div>
<div dir="ltr" class=""><br clear="none" class="">
</div>
<div dir="ltr" class="">I'm checked all the standard things like pagepool and maxFilesToCache<br clear="none" class="">
</div>
<div dir="ltr" class="">(set to the default of 4000), workerThreads is set to 128 on the new<br clear="none" class="">
</div>
<div dir="ltr" class="">gpfs cluster (against default 48 on the old) <br clear="none" class="">
</div>
<div dir="ltr" class=""><br clear="none" class="">
</div>
<div dir="ltr" class="">I'm not sure what else to look at on this one hence why I'm asking the<br clear="none" class="">
</div>
<div dir="ltr" class="">community.<br clear="none" class="">
</div>
<div dir="ltr" class=""><br clear="none" class="">
</div>
<div dir="ltr" class="">Thanks in advance<br clear="none" class="">
</div>
<div dir="ltr" class=""><br clear="none" class="">
</div>
<div dir="ltr" class="">Peter Childs<br clear="none" class="">
</div>
<div dir="ltr" class="">ITS Research Storage<br clear="none" class="">
</div>
<div dir="ltr" class="">Queen Mary University of London.<br clear="none" class="">
</div>
<div dir="ltr" class="">_______________________________________________<br clear="none" class="">
</div>
<div dir="ltr" class="">gpfsug-discuss mailing list<br clear="none" class="">
</div>
<div dir="ltr" class="">gpfsug-discuss at <a href="http://spectrumscale.org" class="">
spectrumscale.org</a><br clear="none" class="">
</div>
<div dir="ltr" class=""><a rel="nofollow" shape="rect" target="_blank" href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br clear="none" class="">
</div>
<br clear="none" class="">
<br clear="none" class="">
</div>
</div>
</div>
</div>
</div>
<pre class="">_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a>
<a rel="nofollow" shape="rect" target="_blank" href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a>
</pre>
</blockquote>
</div>
<div class=""><span class=""></span>
<pre class="">-- <br clear="none" class=""></pre>
<div style="" class="">Peter Childs</div>
<div style="" class="">ITS Research Storage</div>
<div style="" class="">Queen Mary, University of London</div>
<div style="" class=""><br clear="none" class="">
</div>
</div>
</div>
</div>
<div class="yqt1081544495" id="yqt96229">_______________________________________________<br clear="none" class="">
gpfsug-discuss mailing list<br clear="none" class="">
gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a><br clear="none" class="">
<a shape="rect" href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br clear="none" class="">
</div>
<br class="">
<br class="">
</div>
</div>
</div>
</div>
</div>
<pre class="">_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a>
</pre>
</blockquote>
<div class=""><span class="">
<pre class="">-- <br class=""></pre>
<div style="width: 71ch;" class="">Peter Childs</div>
<div style="width: 71ch;" class="">ITS Research Storage</div>
<div style="width: 71ch;" class="">Queen Mary, University of London</div>
<div style="width: 71ch;" class=""><br class="">
</div>
</span></div>
</div>
_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a><br class="">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
<p style="color:rgb(112,113,115);font-family: 'Trebuchet MS', 'Lucida Grande'; font-style: italic; font-size: 10pt;">
The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no. 06885462, with its registered office at 1 Midland Road London NW1 1AT</p>
</body>
</html>