<div dir="ltr"><div dir="ltr"><div dir="ltr">Hi,<div><br></div><div>the best way to debug something like that is to start with top. start top then press 1 and check if any of the cores has almost 0% idle while others have plenty of CPU left. if that is the case you have one very hot thread. </div><div>to further isolate it you can press 1 again to collapse the cores, now press shirt-h which will break down each thread of a process and show them as an individual line. </div><div>now you either see one or many mmfsd's causing cpu consumption, if its many your workload is just doing a lot of work, what is more concerning is if you have just 1 thread running at the 90%+ . if thats the case write down the PID of the thread that runs so hot and run mmfsadm dump threads,kthreads >dum. you will see many entries in the file like : </div><div><br></div><div><div> MMFSADMDumpCmdThread: desc 0x7FC84C002980 handle 0x4C0F02FA parm 0x7FC9700008C0 highStackP 0x7FC783F7E530</div><div> pthread 0x83F80700 kernel thread id 49878 (slot -1) pool 21 ThPoolCommands</div><div> per-thread gbls:</div><div> 0:0x0 1:0x0 2:0x0 3:0x3 4:0xFFFFFFFFFFFFFFFF 5:0x0 6:0x0 7:0x7FC98C0067B0</div><div> 8:0x0 9:0x0 10:0x0 11:0x0 12:0x0 13:0x400000E 14:0x7FC98C004C10 15:0x0</div><div> 16:0x4 17:0x0 18:0x0</div></div><div><br></div><div>find the pid behind 'thread id' and post that section, that would be the first indication on what that thread does ...</div><div><br></div><div>sven</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Nov 20, 2018 at 11:10 PM Saula, Oluwasijibomi <<a href="mailto:oluwasijibomi.saula@ndsu.edu">oluwasijibomi.saula@ndsu.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div id="m_4193670108706930594divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif" dir="ltr">
<p style="margin-top:0;margin-bottom:0">Hello Scalers,</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">First, let me say Happy Thanksgiving to those of us in the US and to those beyond, well, it's a still happy day seeing we're still above ground!
<span>😊</span></p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">Now, what I have to discuss isn't anything extreme so don't skip the turkey for this, but lately, on a few of our compute GPFS client nodes, we've been noticing high CPU usage by the mmfsd process and are wondering why.
Here's a sample:</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0"></p>
<p style="margin:0px;font:16px Menlo;color:rgb(0,0,0);background-color:rgb(254,244,156)">
<span style="font-variant-ligatures:no-common-ligatures">[~]# top -b -n 1 | grep mmfs</span></p>
<p style="margin:0px;font:16px Menlo;color:rgb(0,0,0);background-color:rgb(254,244,156)">
<span style="font-variant-ligatures:no-common-ligatures"><span class="m_4193670108706930594Apple-converted-space"> </span>PID USER<span class="m_4193670108706930594Apple-converted-space">
</span>PR<span class="m_4193670108706930594Apple-converted-space"> </span>NI<span class="m_4193670108706930594Apple-converted-space">
</span>VIRT<span class="m_4193670108706930594Apple-converted-space"> </span>RES<span class="m_4193670108706930594Apple-converted-space">
</span>SHR S<span class="m_4193670108706930594Apple-converted-space"> </span>%CPU %MEM <span class="m_4193670108706930594Apple-converted-space">
</span>TIME+ COMMAND</span></p>
<p></p>
<p style="margin:0px;font:16px Menlo;color:rgb(0,0,0);background-color:rgb(254,244,156)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p></p>
<p style="margin-top:0;margin-bottom:0"></p>
<p style="margin:0px;font:16px Menlo;color:rgb(0,0,0);background-color:rgb(254,244,156)">
<span style="font-variant-ligatures:no-common-ligatures">231898 root <span class="m_4193670108706930594Apple-converted-space">
</span>0 -20 14.508g 4.272g<span class="m_4193670108706930594Apple-converted-space"> </span>70168 S<span class="m_4193670108706930594Apple-converted-space">
</span>93.8<span class="m_4193670108706930594Apple-converted-space"> </span>6.8<span class="m_4193670108706930594Apple-converted-space">
</span>69503:41 </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(189,49,27)"><b>mmfs</b></span><span style="font-variant-ligatures:no-common-ligatures">d</span></p>
<p style="margin:0px;font:16px Menlo;color:rgb(0,0,0);background-color:rgb(254,244,156)">
<span style="font-variant-ligatures:no-common-ligatures"><span class="m_4193670108706930594Apple-converted-space">
</span>4161 root <span class="m_4193670108706930594Apple-converted-space"> </span>0 -20<span class="m_4193670108706930594Apple-converted-space">
</span>121876 <span class="m_4193670108706930594Apple-converted-space"> </span>9412 <span class="m_4193670108706930594Apple-converted-space">
</span>1492 S <span class="m_4193670108706930594Apple-converted-space"> </span>0.0<span class="m_4193670108706930594Apple-converted-space">
</span>0.0 <span class="m_4193670108706930594Apple-converted-space"> </span>0:00.22 run</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(189,49,27)"><b>mmfs</b></span></p>
<br>
<p></p>
<p style="margin-top:0;margin-bottom:0">Obviously, this behavior was likely triggered by a not-so-convenient user job that in most cases is long finished by the time we investigate. Nevertheless, does anyone have an idea why this might be happening? Any thoughts
on preventive steps even?</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">This is GPFS v4.2.3 on Redhat 7.4, btw...</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<div id="m_4193670108706930594Signature">
<div id="m_4193670108706930594divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif,Helvetica,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<p></p>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
<font face="Calibri,Arial,Helvetica,sans-serif">Thanks,</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
<font face="Calibri,Arial,Helvetica,sans-serif"> </font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
<font face="Calibri,Arial,Helvetica,sans-serif">Siji Saula</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
<font face="Calibri,Arial,Helvetica,sans-serif">HPC System Administrator</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
<font face="Calibri,Arial,Helvetica,sans-serif">Center for Computationally Assisted Science & Technology</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
<b>NORTH DAKOTA STATE UNIVERSITY</b></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
<a href="https://www.ndsu.edu/alphaindex/buildings/Building::395" id="m_4193670108706930594LPNoLP" target="_blank"><span id="m_4193670108706930594LPNoLP"></span></a><span id="m_4193670108706930594LPNoLP" style="font-family:Calibri,Arial,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"><a href="https://www.ndsu.edu/alphaindex/buildings/Building::396" rel="noopener noreferrer" id="m_4193670108706930594LPNoLP" target="_blank"><span style="color:rgb(0,111,201)">Research
2 Building</span></a><a href="https://www.ndsu.edu/alphaindex/buildings/Building::395" rel="noopener noreferrer" id="m_4193670108706930594LPNoLP" target="_blank"></a></span><span style="font-family:Calibri,Arial,Helvetica,sans-serif,serif,EmojiFont;font-size:16px"> – </span><span class="m_4193670108706930594contextualExtensionHighlight m_4193670108706930594ms-font-color-themePrimary m_4193670108706930594ms-border-color-themePrimary m_4193670108706930594ident_1641_1697" style="font-family:Calibri,Arial,Helvetica,sans-serif,serif,EmojiFont;font-size:16px;color:rgb(0,0,0)">Room
220B</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
Dept 4100, PO Box 6050 / Fargo, ND 58108-6050</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
p:701.231.7749</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
<a id="m_4193670108706930594LPNoLP"><span id="m_4193670108706930594LPNoLP">www.ccast.ndsu.edu</span></a> | <a id="m_4193670108706930594LPNoLP"><span id="m_4193670108706930594LPNoLP">www.ndsu.edu</span></a></div>
<br>
<p></p>
</div>
</div>
</div>
</div>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
</blockquote></div>