<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif, Helvetica, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;">
<p style="margin-top:0; margin-bottom:0">Sven/Jim,</p>
<p style="margin-top:0; margin-bottom:0"><br>
</p>
<p style="margin-top:0; margin-bottom:0">Thanks for sharing your thoughts! - Currently, we have mFTC set as such:</p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">maxFilesToCache 4000</span></p>
<p></p>
<p style="margin-top:0; margin-bottom:0"><br>
</p>
<p style="margin-top:0; margin-bottom:0">However, since we have a very diverse workload, we'd have to cycle through a vast majority of our apps to find the most fitting mFTC value as this page (<a href="https://www.ibm.com/support/knowledgecenter/en/linuxonibm/liaag/wecm/l0wecm00_maxfilestocache.htm" class="OWAAutoLink" id="LPlnk842839" previewremoved="true">https://www.ibm.com/support/knowledgecenter/en/linuxonibm/liaag/wecm/l0wecm00_maxfilestocache.htm</a>)
suggests.</p>
<p style="margin-top:0; margin-bottom:0"><br>
</p>
<p style="margin-top:0; margin-bottom:0">In the meantime, I was able to gather some more info for the lone mmfsd thread (pid: 34096) running at high CPU utilization, and right away I can see the number of nonvoluntary_ctxt_switches is quite high, compared to
the other threads in the dump; however, I think I need some help interpreting all of this. Although, I should add that heavy HPC workloads (i.e. vasp, ansys...) are running on these nodes and may be somewhat related to this issue:</p>
<p style="margin-top:0; margin-bottom:0"><br>
</p>
<p style="margin-top:0; margin-bottom:0"></p>
<p style="margin-right: 0px; margin-left: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; background-color: rgb(254, 244, 156);">
Scheduling info for kernel thread <span style="color: rgb(254, 244, 156); background-color: rgb(0, 0, 0);">34096</span></p>
<p style="margin-right: 0px; margin-left: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; background-color: rgb(254, 244, 156);">
mmfsd (<span style="color: rgb(254, 244, 156); background-color: rgb(0, 0, 0);">34096</span>, #threads: 309)</p>
<p style="margin-right: 0px; margin-left: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; background-color: rgb(254, 244, 156);">
-------------------------------------------------------------------</p>
<p style="margin-right: 0px; margin-left: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; background-color: rgb(254, 244, 156);">
se.exec_start<span class="Apple-converted-space"> </span>:<span class="Apple-converted-space"> </span>8057632237.613486</p>
<p style="margin-right: 0px; margin-left: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; background-color: rgb(254, 244, 156);">
se.vruntime<span class="Apple-converted-space"> </span>:<span class="Apple-converted-space"> </span>4914854123.640008</p>
<p style="margin-right: 0px; margin-left: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; background-color: rgb(254, 244, 156);">
se.sum_exec_runtime<span class="Apple-converted-space"> </span>:<span class="Apple-converted-space"> </span>1042598557.420591</p>
<p style="margin-right: 0px; margin-left: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; background-color: rgb(254, 244, 156);">
se.nr_migrations <span class="Apple-converted-space"> </span>:<span class="Apple-converted-space"> </span>8337485</p>
<p style="margin-right: 0px; margin-left: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; background-color: rgb(254, 244, 156);">
nr_switches<span class="Apple-converted-space"> </span>: <span class="Apple-converted-space"> </span>15824325</p>
<p style="margin-right: 0px; margin-left: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; background-color: rgb(254, 244, 156);">
nr_voluntary_switches<span class="Apple-converted-space"> </span>: <span class="Apple-converted-space"> </span>4110</p>
<p style="margin-right: 0px; margin-left: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; background-color: rgb(254, 244, 156);">
nr_involuntary_switches<span class="Apple-converted-space"> </span>: <span class="Apple-converted-space"> </span>15820215</p>
<p style="margin-right: 0px; margin-left: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; background-color: rgb(254, 244, 156);">
se.load.weight <span class="Apple-converted-space"> </span>:<span class="Apple-converted-space"> </span>88761</p>
<p style="margin-right: 0px; margin-left: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; background-color: rgb(254, 244, 156);">
policy <span class="Apple-converted-space"> </span>:<span class="Apple-converted-space"> </span>0</p>
<p style="margin-right: 0px; margin-left: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; background-color: rgb(254, 244, 156);">
prio <span class="Apple-converted-space"> </span>:<span class="Apple-converted-space"> </span>100</p>
<p style="margin-right: 0px; margin-left: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; background-color: rgb(254, 244, 156);">
clock-delta<span class="Apple-converted-space"> </span>: <span class="Apple-converted-space"> </span>24</p>
<p style="margin-right: 0px; margin-left: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; background-color: rgb(254, 244, 156);">
mm->numa_scan_seq<span class="Apple-converted-space"> </span>:<span class="Apple-converted-space"> </span>88980</p>
<p style="margin-right: 0px; margin-left: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; background-color: rgb(254, 244, 156);">
numa_migrations, 5216521</p>
<p style="margin-right: 0px; margin-left: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; background-color: rgb(254, 244, 156);">
numa_faults_memory, 0, 0, 1, 1, 1</p>
<p style="margin-right: 0px; margin-left: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; background-color: rgb(254, 244, 156);">
numa_faults_memory, 1, 0, 0, 1, 1030</p>
<p style="margin-right: 0px; margin-left: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; background-color: rgb(254, 244, 156);">
numa_faults_memory, 0, 1, 0, 0, 1</p>
<p style="margin-right: 0px; margin-left: 0px; font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; background-color: rgb(254, 244, 156);">
numa_faults_memory, 1, 1, 0, 0, 1</p>
<br style="font-family: Calibri, Helvetica, sans-serif, Helvetica, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">
<p style="font-family: Calibri, Helvetica, sans-serif, Helvetica, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">
</p>
<p></p>
<p style="margin-top:0; margin-bottom:0"><br>
</p>
<p style="margin-top:0; margin-bottom:0"></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">Status for kernel thread </span><span style="color:rgb(254,244,156); background-color:rgb(0,0,0)">34096</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">Name: <span class="Apple-converted-space"> </span>mmfsd</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">Umask:<span class="Apple-converted-space"> </span>0022</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">State:<span class="Apple-converted-space"> </span>R (running)</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">Tgid: <span class="Apple-converted-space"> </span>58921</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">Ngid: <span class="Apple-converted-space"> </span>34395</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">Pid:<span class="Apple-converted-space"> </span></span><span style="color:rgb(254,244,156); background-color:rgb(0,0,0)">34096</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">PPid: <span class="Apple-converted-space"> </span>3941</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">TracerPid:<span class="Apple-converted-space"> </span>0</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">Uid:<span class="Apple-converted-space"> </span>0 <span class="Apple-converted-space">
</span>0 <span class="Apple-converted-space"> </span>0 <span class="Apple-converted-space">
</span>0</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">Gid:<span class="Apple-converted-space"> </span>0 <span class="Apple-converted-space">
</span>0 <span class="Apple-converted-space"> </span>0 <span class="Apple-converted-space">
</span>0</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">FDSize: 256</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">Groups:<span class="Apple-converted-space"> </span></span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">VmPeak: 15137612 kB</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">VmSize: 15126340 kB</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">VmLck: <span class="Apple-converted-space"> </span>4194304 kB</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">VmPin: <span class="Apple-converted-space"> </span>8388712 kB</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">VmHWM: <span class="Apple-converted-space"> </span>4424228 kB</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">VmRSS: <span class="Apple-converted-space"> </span>4420420 kB</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">RssAnon: <span class="Apple-converted-space"> </span>4350128 kB</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">RssFile: <span class="Apple-converted-space"> </span>50512 kB</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">RssShmem:<span class="Apple-converted-space"> </span>19780 kB</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">VmData: 14843812 kB</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">VmStk: <span class="Apple-converted-space"> </span>132 kB</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">VmExe: <span class="Apple-converted-space"> </span>23672 kB</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">VmLib:<span class="Apple-converted-space"> </span>121856 kB</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">VmPTE:<span class="Apple-converted-space"> </span>9652 kB</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">VmSwap:<span class="Apple-converted-space"> </span>0 kB</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">Threads:<span class="Apple-converted-space"> </span>309</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">SigQ: <span class="Apple-converted-space"> </span>5/257225</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">SigPnd: 0000000000000000</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">ShdPnd: 0000000000000000</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">SigBlk: 0000000010017a07</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">SigIgn: 0000000000000000</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">SigCgt: 0000000180015eef</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">CapInh: 0000000000000000</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">CapPrm: 0000001fffffffff</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">CapEff: 0000001fffffffff</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">CapBnd: 0000001fffffffff</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">CapAmb: 0000000000000000</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">Seccomp:<span class="Apple-converted-space"> </span>0</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">Cpus_allowed: <span class="Apple-converted-space"> </span>ffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">Cpus_allowed_list:<span class="Apple-converted-space"> </span>
0-239</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">Mems_allowed: <span class="Apple-converted-space"> </span>00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">Mems_allowed_list:<span class="Apple-converted-space"> </span>
0-1</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">voluntary_ctxt_switches:<span class="Apple-converted-space">
</span>4110</span></p>
<p style="margin:0px; font:16px Menlo; color:rgb(0,0,0); background-color:rgb(254,244,156)">
<span style="">nonvoluntary_ctxt_switches: <span class="Apple-converted-space">
</span>15820215</span></p>
<br>
<p></p>
<p style="margin-top:0; margin-bottom:0"><br>
</p>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:rgb(0,0,0); font-family:Calibri,Arial,Helvetica,sans-serif,Helvetica,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<p></p>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<font face="Calibri,Arial,Helvetica,sans-serif">Thanks,</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<font face="Calibri,Arial,Helvetica,sans-serif"> </font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<font face="Calibri,Arial,Helvetica,sans-serif">Siji Saula</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<font face="Calibri,Arial,Helvetica,sans-serif">HPC System Administrator</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<font face="Calibri,Arial,Helvetica,sans-serif">Center for Computationally Assisted Science & Technology</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<b>NORTH DAKOTA STATE UNIVERSITY</b></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<a href="https://www.ndsu.edu/alphaindex/buildings/Building::395" target="_blank" id="LPNoLP"><span id="LPNoLP"></span></a><span id="LPNoLP" style="font-family:Calibri,Arial,Helvetica,sans-serif,serif,EmojiFont; font-size:16px"><a href="https://www.ndsu.edu/alphaindex/buildings/Building::396" target="_blank" rel="noopener noreferrer" id="LPNoLP"><span style="color:rgb(0,111,201)">Research
2 Building</span></a><a href="https://www.ndsu.edu/alphaindex/buildings/Building::395" target="_blank" rel="noopener noreferrer" id="LPNoLP"></a></span><span style="font-family:Calibri,Arial,Helvetica,sans-serif,serif,EmojiFont; font-size:16px"> – </span><span tabindex="0" class="contextualExtensionHighlight ms-font-color-themePrimary ms-border-color-themePrimary ident_1641_1697" style="font-family:Calibri,Arial,Helvetica,sans-serif,serif,EmojiFont; font-size:16px; color:rgb(0,0,0)">Room
220B</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
Dept 4100, PO Box 6050 / Fargo, ND 58108-6050</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
p:701.231.7749</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<a href="file://composeviewinternalloadurl/www.ccast.ndsu.edu" target="_blank" id="LPNoLP"><span id="LPNoLP">www.ccast.ndsu.edu</span></a> | <a href="file://composeviewinternalloadurl/www.ndsu.edu" target="_blank" id="LPNoLP"><span id="LPNoLP">www.ndsu.edu</span></a></div>
<br>
<p></p>
</div>
</div>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> gpfsug-discuss-bounces@spectrumscale.org <gpfsug-discuss-bounces@spectrumscale.org> on behalf of gpfsug-discuss-request@spectrumscale.org
<gpfsug-discuss-request@spectrumscale.org><br>
<b>Sent:</b> Wednesday, November 21, 2018 9:33:10 AM<br>
<b>To:</b> gpfsug-discuss@spectrumscale.org<br>
<b>Subject:</b> gpfsug-discuss Digest, Vol 82, Issue 31</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt">
<div class="PlainText">Send gpfsug-discuss mailing list submissions to<br>
gpfsug-discuss@spectrumscale.org<br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" id="LPlnk398492" class="OWAAutoLink" previewremoved="true">
http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
or, via email, send a message with subject or body 'help' to<br>
gpfsug-discuss-request@spectrumscale.org<br>
<br>
You can reach the person managing the list at<br>
gpfsug-discuss-owner@spectrumscale.org<br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of gpfsug-discuss digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: mmfsd recording High CPU usage (Jim Doherty)<br>
2. Re: mmfsd recording High CPU usage (Sven Oehme)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Wed, 21 Nov 2018 13:01:54 +0000 (UTC)<br>
From: Jim Doherty <jjdoherty@yahoo.com><br>
To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
Subject: Re: [gpfsug-discuss] mmfsd recording High CPU usage<br>
Message-ID: <1913697205.666954.1542805314669@mail.yahoo.com><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
At a guess with no data ....?? if the application is opening more files than can fit in the maxFilesToCache (MFTC) objects? GPFS will expand the MFTC to support the open files,? but it will also scan to try and free any unused objects.??? If you can identify
the user job that is causing this? you could monitor a system more closely. <br>
<br>
Jim<br>
<br>
On Wednesday, November 21, 2018, 2:10:45 AM EST, Saula, Oluwasijibomi <oluwasijibomi.saula@ndsu.edu> wrote:
<br>
<br>
<!--#yiv4325073500 P {margin-top:0;margin-bottom:0;}--><br>
Hello Scalers,<br>
<br>
<br>
<br>
<br>
First, let me say Happy Thanksgiving to those of us in the US and to those beyond, well, it's a?still happy day seeing we're still above ground!?<br>
<br>
<br>
<br>
<br>
Now, what I have to discuss isn't anything extreme so don't skip the turkey for this, but lately, on a few of our compute GPFS client nodes, we've been noticing high CPU usage by the mmfsd process and are wondering why. Here's a sample:<br>
<br>
<br>
<br>
<br>
<br>
<br>
[~]# top -b -n 1 | grep mmfs<br>
<br>
? ?PID USER? ? ?PR? NI? ?VIRT? ? RES? ?SHR S? %CPU %MEM ? ? TIME+ COMMAND<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
231898 root ? ? ? 0 -20 14.508g 4.272g? 70168 S?93.8? 6.8?69503:41 mmfsd<br>
<br>
?4161 root ? ? ? 0 -20?121876 ? 9412 ? 1492 S ? 0.0?0.0 ? 0:00.22 runmmfs<br>
<br>
<br>
<br>
<br>
Obviously, this behavior was likely triggered by a not-so-convenient user job that in most cases is long finished by the time we investigate.?Nevertheless, does anyone have an idea why this might be happening? Any thoughts on preventive steps even?<br>
<br>
<br>
<br>
<br>
This is GPFS v4.2.3 on Redhat 7.4, btw...<br>
<br>
<br>
<br>
<br>
<br>
Thanks,?Siji SaulaHPC System AdministratorCenter for Computationally Assisted Science & TechnologyNORTH DAKOTA STATE UNIVERSITY?<br>
Research 2 Building???Room 220BDept 4100, PO Box 6050? / Fargo, ND 58108-6050p:701.231.7749www.ccast.ndsu.edu?|?www.ndsu.edu<br>
<br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at spectrumscale.org<br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" id="LPlnk979962" class="OWAAutoLink" previewremoved="true">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20181121/16de3172/attachment-0001.html" id="LPlnk867572" class="OWAAutoLink" previewremoved="true">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20181121/16de3172/attachment-0001.html</a>><br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Wed, 21 Nov 2018 07:32:55 -0800<br>
From: Sven Oehme <oehmes@gmail.com><br>
To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
Subject: Re: [gpfsug-discuss] mmfsd recording High CPU usage<br>
Message-ID:<br>
<CALssuR2PA5Q73p-i=xnNP97+bvzdqT8RO2p1NZ6duBbhRO8OCw@mail.gmail.com><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Hi,<br>
<br>
the best way to debug something like that is to start with top. start top<br>
then press 1 and check if any of the cores has almost 0% idle while others<br>
have plenty of CPU left. if that is the case you have one very hot thread.<br>
to further isolate it you can press 1 again to collapse the cores, now<br>
press shirt-h which will break down each thread of a process and show them<br>
as an individual line.<br>
now you either see one or many mmfsd's causing cpu consumption, if its many<br>
your workload is just doing a lot of work, what is more concerning is if<br>
you have just 1 thread running at the 90%+ . if thats the case write down<br>
the PID of the thread that runs so hot and run mmfsadm dump<br>
threads,kthreads >dum. you will see many entries in the file like :<br>
<br>
MMFSADMDumpCmdThread: desc 0x7FC84C002980 handle 0x4C0F02FA parm<br>
0x7FC9700008C0 highStackP 0x7FC783F7E530<br>
pthread 0x83F80700 kernel thread id 49878 (slot -1) pool 21<br>
ThPoolCommands<br>
per-thread gbls:<br>
0:0x0 1:0x0 2:0x0 3:0x3 4:0xFFFFFFFFFFFFFFFF 5:0x0 6:0x0<br>
7:0x7FC98C0067B0<br>
8:0x0 9:0x0 10:0x0 11:0x0 12:0x0 13:0x400000E 14:0x7FC98C004C10<br>
15:0x0<br>
16:0x4 17:0x0 18:0x0<br>
<br>
find the pid behind 'thread id' and post that section, that would be the<br>
first indication on what that thread does ...<br>
<br>
sven<br>
<br>
<br>
<br>
<br>
<br>
<br>
On Tue, Nov 20, 2018 at 11:10 PM Saula, Oluwasijibomi <<br>
oluwasijibomi.saula@ndsu.edu> wrote:<br>
<br>
> Hello Scalers,<br>
><br>
><br>
> First, let me say Happy Thanksgiving to those of us in the US and to those<br>
> beyond, well, it's a still happy day seeing we're still above ground! ?<br>
><br>
><br>
> Now, what I have to discuss isn't anything extreme so don't skip the<br>
> turkey for this, but lately, on a few of our compute GPFS client nodes,<br>
> we've been noticing high CPU usage by the mmfsd process and are wondering<br>
> why. Here's a sample:<br>
><br>
><br>
> [~]# top -b -n 1 | grep mmfs<br>
><br>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+<br>
> COMMAND<br>
><br>
><br>
> 231898 root 0 -20 14.508g 4.272g 70168 S 93.8 6.8 69503:41<br>
> *mmfs*d<br>
><br>
> 4161 root 0 -20 121876 9412 1492 S 0.0 0.0 0:00.22 run<br>
> *mmfs*<br>
><br>
> Obviously, this behavior was likely triggered by a not-so-convenient user<br>
> job that in most cases is long finished by the time we<br>
> investigate. Nevertheless, does anyone have an idea why this might be<br>
> happening? Any thoughts on preventive steps even?<br>
><br>
><br>
> This is GPFS v4.2.3 on Redhat 7.4, btw...<br>
><br>
><br>
> Thanks,<br>
><br>
> Siji Saula<br>
> HPC System Administrator<br>
> Center for Computationally Assisted Science & Technology<br>
> *NORTH DAKOTA STATE UNIVERSITY*<br>
><br>
><br>
> <<a href="https://www.ndsu.edu/alphaindex/buildings/Building::395" id="LPlnk663312" class="OWAAutoLink" previewremoved="true">https://www.ndsu.edu/alphaindex/buildings/Building::395</a>>Research 2<br>
> Building <<a href="https://www.ndsu.edu/alphaindex/buildings/Building::396" id="LPlnk177983" class="OWAAutoLink" previewremoved="true">https://www.ndsu.edu/alphaindex/buildings/Building::396</a>><br>
> <<a href="https://www.ndsu.edu/alphaindex/buildings/Building::395" id="LPlnk134122" class="OWAAutoLink" previewremoved="true">https://www.ndsu.edu/alphaindex/buildings/Building::395</a>> ? Room 220B<br>
> Dept 4100, PO Box 6050 / Fargo, ND 58108-6050<br>
> p:701.231.7749<br>
> <a href="http://www.ccast.ndsu.edu" id="LPlnk5344" class="OWAAutoLink" previewremoved="true">
www.ccast.ndsu.edu</a> | <a href="http://www.ndsu.edu" id="LPlnk110765" class="OWAAutoLink" previewremoved="true">
www.ndsu.edu</a><br>
><br>
> _______________________________________________<br>
> gpfsug-discuss mailing list<br>
> gpfsug-discuss at spectrumscale.org<br>
> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" id="LPlnk206377" class="OWAAutoLink" previewremoved="true">
http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
><br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20181121/e829f214/attachment.html" id="LPlnk983290" class="OWAAutoLink" previewremoved="true">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20181121/e829f214/attachment.html</a>><br>
<br>
------------------------------<br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at spectrumscale.org<br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" id="LPlnk756228" class="OWAAutoLink" previewremoved="true">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
<br>
<br>
End of gpfsug-discuss Digest, Vol 82, Issue 31<br>
**********************************************<br>
</div>
</span></font></div>
</div>
</body>
</html>