<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<div><br>
</div>
<div>top</div>
<div><br>
</div>
<div>but ps gives the same value.</div>
<div><br>
</div>
<div>[<a href="mailto:root@dn29">root@dn29</a> ~]# ps auww -q 4444</div>
<div>USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND</div>
<div>root 4444 2.7 22.3 10537600 5472580 ? S<Ll Jul12 466:13 /usr/lpp/mmfs/bin/mmfsd</div>
<div></div>
<div><br>
</div>
<div>Thanks for the help</div>
<div><br>
</div>
<div>Peter.</div>
<div><br>
</div>
<div><br>
</div>
<div>On Mon, 2017-07-24 at 14:10 +0000, Jim Doherty wrote:</div>
<blockquote type="cite">
<div style="color:#000; background-color:#fff; font-family:Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:13px">
How are you identifying the high memory usage? <br>
<div id="yui_3_16_0_ym19_1_1500905216021_5198"><span></span></div>
<div class="qtdSeparateBR"><br>
<br>
</div>
<div class="yahoo_quoted" style="display: block;">
<div style="font-family: Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 13px;">
<div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 16px;">
<div dir="ltr"><font size="2" face="Arial">On Monday, July 24, 2017 9:30 AM, Peter Childs <p.childs@qmul.ac.uk> wrote:<br>
</font></div>
<br>
<br>
<div class="y_msg_container">
<div id="yiv0922426330">
<div>
<div>I've had a look at mmfsadm dump malloc and it looks to agree with the output from mmdiag --memory. and does not seam to account for the excessive memory usage.</div>
<div><br clear="none">
</div>
<div>The new machines do have idleSocketTimout set to 0 from what your saying it could be related to keeping that many connections between nodes working.</div>
<div><br clear="none">
</div>
<div>Thanks in advance</div>
<div><br clear="none">
</div>
<div>Peter.</div>
<div><br clear="none">
</div>
<div><br clear="none">
</div>
<div><br clear="none">
</div>
<div><br clear="none">
</div>
<div>[<a rel="nofollow" shape="rect" ymailto="mailto:root@dn29" target="_blank" href="mailto:root@dn29">root@dn29</a> ~]# mmdiag --memory</div>
<div><br clear="none">
</div>
<div>=== mmdiag: memory ===</div>
<div>mmfsd heap size: 2039808 bytes</div>
<div><br clear="none">
</div>
<div><br clear="none">
</div>
<div>Statistics for MemoryPool id 1 ("Shared Segment (EPHEMERAL)")</div>
<div> 128 bytes in use</div>
<div> 17500049370 hard limit on memory usage</div>
<div> 1048576 bytes committed to regions</div>
<div> 1 number of regions</div>
<div> 555 allocations</div>
<div> 555 frees</div>
<div> 0 allocation failures</div>
<div><br clear="none">
</div>
<div><br clear="none">
</div>
<div>Statistics for MemoryPool id 2 ("Shared Segment")</div>
<div> 42179592 bytes in use</div>
<div> 17500049370 hard limit on memory usage</div>
<div> 56623104 bytes committed to regions</div>
<div> 9 number of regions</div>
<div> 100027 allocations</div>
<div> 79624 frees</div>
<div> 0 allocation failures</div>
<div><br clear="none">
</div>
<div><br clear="none">
</div>
<div>Statistics for MemoryPool id 3 ("Token Manager")</div>
<div> 2099520 bytes in use</div>
<div> 17500049370 hard limit on memory usage</div>
<div> 16778240 bytes committed to regions</div>
<div> 1 number of regions</div>
<div> 4 allocations</div>
<div> 0 frees</div>
<div> 0 allocation failures</div>
<div><br clear="none">
</div>
<div><br clear="none">
</div>
<div></div>
<div class="yiv0922426330yqt1081544495" id="yiv0922426330yqt39566">
<div>On Mon, 2017-07-24 at 13:11 +0000, Jim Doherty wrote:</div>
<blockquote type="cite">
<div style="color:#000;background-color:#fff;font-family:Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:13px;">
<div id="yiv0922426330">
<div id="yiv0922426330yui_3_16_0_ym19_1_1500900502762_16662">
<div id="yiv0922426330yui_3_16_0_ym19_1_1500900502762_16661" style="color:#000;background-color:#fff;font-family:Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:13px;">
<div id="yiv0922426330yui_3_16_0_ym19_1_1500900502762_12131">There are 3 places that the GPFS mmfsd uses memory the pagepool plus 2 shared memory segments. To see the memory utilization of the shared memory segments run the command mmfsadm dump malloc
. The statistics for memory pool id 2 is where maxFilesToCache/maxStatCache objects are and the manager nodes use memory pool id 3 to track the MFTC/MSC objects.
<br clear="none">
</div>
<div id="yiv0922426330yui_3_16_0_ym19_1_1500900502762_13732"><br clear="none">
</div>
<div id="yiv0922426330yui_3_16_0_ym19_1_1500900502762_13733">You might want to upgrade to later PTF as there was a PTF to fix a memory leak that occurred in tscomm associated with network connection drops.
<br clear="none">
</div>
<div id="yiv0922426330yui_3_16_0_ym19_1_1500900502762_7195"><span></span></div>
<div class="yiv0922426330qtdSeparateBR" id="yiv0922426330yui_3_16_0_ym19_1_1500900502762_16707">
<br clear="none">
<br clear="none">
</div>
<div class="yiv0922426330yqt2983717346" id="yiv0922426330yqt44105"></div>
</div>
</div>
</div>
<div>
<div style="font-family:Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:13px;">
<div style="font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:16px;">
<div dir="ltr"><font size="2" face="Arial">On Monday, July 24, 2017 5:29 AM, Peter Childs <p.childs@qmul.ac.uk> wrote:<br clear="none">
</font></div>
<br clear="none">
<br clear="none">
<div class="yiv0922426330y_msg_container">
<div dir="ltr">We have two GPFS clusters.<br clear="none">
</div>
<div dir="ltr"><br clear="none">
</div>
<div dir="ltr">One is fairly old and running 4.2.1-2 and non CCR and the nodes run<br clear="none">
</div>
<div dir="ltr">fine using up about 1.5G of memory and is consistent (GPFS pagepool is<br clear="none">
</div>
<div dir="ltr">set to 1G, so that looks about right.)<br clear="none">
</div>
<div dir="ltr"><br clear="none">
</div>
<div dir="ltr">The other one is "newer" running 4.2.1-3 with CCR and the nodes keep<br clear="none">
</div>
<div dir="ltr">increasing in there memory usage, starting at about 1.1G and are find<br clear="none">
</div>
<div dir="ltr">for a few days however after a while they grow to 4.2G which when the<br clear="none">
</div>
<div dir="ltr">node need to run real work, means the work can't be done.<br clear="none">
</div>
<div dir="ltr"><br clear="none">
</div>
<div dir="ltr">I'm losing track of what maybe different other than CCR, and I'm trying<br clear="none">
</div>
<div dir="ltr">to find some more ideas of where to look.<br clear="none">
</div>
<div dir="ltr"><br clear="none">
</div>
<div dir="ltr">I'm checked all the standard things like pagepool and maxFilesToCache<br clear="none">
</div>
<div dir="ltr">(set to the default of 4000), workerThreads is set to 128 on the new<br clear="none">
</div>
<div dir="ltr">gpfs cluster (against default 48 on the old) <br clear="none">
</div>
<div dir="ltr"><br clear="none">
</div>
<div dir="ltr">I'm not sure what else to look at on this one hence why I'm asking the<br clear="none">
</div>
<div dir="ltr">community.<br clear="none">
</div>
<div dir="ltr"><br clear="none">
</div>
<div dir="ltr">Thanks in advance<br clear="none">
</div>
<div dir="ltr"><br clear="none">
</div>
<div dir="ltr">Peter Childs<br clear="none">
</div>
<div dir="ltr">ITS Research Storage<br clear="none">
</div>
<div dir="ltr">Queen Mary University of London.<br clear="none">
</div>
<div dir="ltr">_______________________________________________<br clear="none">
</div>
<div dir="ltr">gpfsug-discuss mailing list<br clear="none">
</div>
<div dir="ltr">gpfsug-discuss at spectrumscale.org<br clear="none">
</div>
<div dir="ltr"><a rel="nofollow" shape="rect" target="_blank" href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br clear="none">
</div>
<br clear="none">
<br clear="none">
</div>
</div>
</div>
</div>
</div>
<pre>_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
<a rel="nofollow" shape="rect" target="_blank" href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a>
</pre>
</blockquote>
</div>
<div><span></span>
<pre>-- <br clear="none"></pre>
<div style="">Peter Childs</div>
<div style="">ITS Research Storage</div>
<div style="">Queen Mary, University of London</div>
<div style=""><br clear="none">
</div>
</div>
</div>
</div>
<div class="yqt1081544495" id="yqt96229">_______________________________________________<br clear="none">
gpfsug-discuss mailing list<br clear="none">
gpfsug-discuss at spectrumscale.org<br clear="none">
<a shape="rect" href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br clear="none">
</div>
<br>
<br>
</div>
</div>
</div>
</div>
</div>
<pre>_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a>
</pre>
</blockquote>
<div><span>
<pre>-- <br></pre>
<div style="width: 71ch;">Peter Childs</div>
<div style="width: 71ch;">ITS Research Storage</div>
<div style="width: 71ch;">Queen Mary, University of London</div>
<div style="width: 71ch;"><br>
</div>
</span></div>
</body>
</html>