<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:12pt" ><div dir="ltr" >In addition to what Uwe mentioned</div>
<div dir="ltr" > </div>
<div dir="ltr" >I am bit concern of you needing to look into killing daemon to "fix" it.</div>
<div dir="ltr" > </div>
<div dir="ltr" >You might want to look into QoS fine-stats to get PID process information for the node</div>
<div dir="ltr" > </div>
<div dir="ltr" ><a href="http://files.gpfsug.org/presentations/2018/London/14_LuisBolinches_GPFSUG.pdf">http://files.gpfsug.org/presentations/2018/London/14_LuisBolinches_GPFSUG.pdf</a></div>
<div dir="ltr" > </div>
<div dir="ltr" >Slide 15 and KC</div>
<div dir="ltr" > </div>
<div dir="ltr" ><a href="https://www.ibm.com/support/knowledgecenter/STXKQY_5.1.0/com.ibm.spectrum.scale.v5r10.doc/bl1adm_mmqos.htm">https://www.ibm.com/support/knowledgecenter/STXKQY_5.1.0/com.ibm.spectrum.scale.v5r10.doc/bl1adm_mmqos.htm</a></div>
<div dir="ltr" > </div>
<div dir="ltr" >It consumes some memory per node. No need to set any QoS limits just enable QoS</div>
<div dir="ltr" > </div>
<div dir="ltr" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div dir="ltr" ><div style="font-size: 12pt; font-weight: bold; font-family: sans-serif; color: #7C7C5F;" ><span style="font-size:10pt;" ><font face="Arial" >--<br>Ystävällisin terveisin / Kind regards / Saludos cordiales / Salutations</font> / <font face="Arial" >Salutacions</font><br><font face="Arial" >Luis Bolinches</font></span></div>
<div style="font-size: 12pt; font-weight: bold; font-family: sans-serif; color: #7C7C5F;" ><div><span style="font-size:10pt;" ><font face="Arial" >Consultant IT Specialist</font></span></div>
<div><span style="font-size:10pt;" ><font face="Arial" >IBM Spectrum Scale development</font></span></div>
<div><span style="font-size:10pt;" ><font face="Arial" >Mobile Phone: +358503112585</font></span></div>
<div> </div>
<div><span style="font-size:10pt;" ><a href="https://www.youracclaim.com/user/luis-bolinches" rel="noopener" target="_blank"><b><font face="Arial" >https://www.youracclaim.com/user/luis-bolinches</font></b></a></span></div>
<div> </div>
<div><div><span style="font-size:10pt;" ><font face="Arial" >Ab IBM Finland Oy</font></span></div>
<div><span style="font-size:10pt;" ><font face="Arial" >Laajalahdentie 23</font></span></div>
<div><span style="font-size:10pt;" ><font face="Arial" >00330 Helsinki</font></span></div>
<div><span style="font-size:10pt;" ><font face="Arial" >Uusimaa - Finland</font></span></div><br><span style="font-size:10pt;" ><b><font face="Arial" color="#7C7C5F" >"If you always give you will always have" -- Anonymous</font></b></span></div></div>
<div style="font-size: 8pt; font-family: sans-serif; margin-top: 10px;" ><div> </div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Original message -----<br>From: "Uwe Falke" <UWEFALKE@de.ibm.com><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Cc:<br>Subject: [EXTERNAL] Re: [gpfsug-discuss] Poor client performance with high cpu usage of mmfsd process<br>Date: Thu, Nov 12, 2020 03:56<br>
<div><font size="2" face="Default Monospace,Courier New,Courier,monospace" >Hi, Kamil,<br>I suppose you'd rather not see such an issue than pursue the ugly<br>work-around to kill off processes.<br>In such situations, the first looks should be for the GPFS log (on the<br>client, on the cluster manager, and maybe on the file system manager) and<br>for the current waiters (that is the list of currently waiting threads) on<br>the hanging client.<br><br>-> /var/adm/ras/mmfs.log.latest<br>mmdiag --waiters<br><br>That might give you a first idea what is taking long and which components<br>are involved.<br><br>Also,<br>mmdiag --iohist<br>shows you the last IOs and some stats (service time, size) for them.<br><br>Either that clue is already sufficient, or you go on (if you see DIO<br>somewhere, direct IO is used which might slow down things, for example).<br>GPFS has a nice tracing which you can configure or just run the default<br>trace.<br><br>Running a dedicated (low-level) io trace can be achieved by<br>mmtracectl --start --trace=io --tracedev-write-mode=overwrite -N<br><your_critical_node><br>then, when the issue is seen, stop the trace by<br>mmtracectl --stop -N <your_critical_node><br><br>Do not wait to stop the trace once you've seen the issue, the trace file<br>cyclically overwrites its output. If the issue lasts some time you could<br>also start the trace while you see it, run the trace for say 20 secs and<br>stop again. On stopping the trace, the output gets converted into an ASCII<br>trace file named trcrpt.*(usually in /tmp/mmfs, check the command output).<br><br><br>There you should see lines with FIO which carry the inode of the related<br>file after the "tag" keyword.<br>example:<br>0.000745100 25123 TRACE_IO: FIO: read data tag 248415 43466 ioVecSize 8<br>1st buf 0x299E89BC000 disk 8D0 da 154:2083875440 nSectors 128 err 0<br>finishTime 1563473283.135212150<br><br>-> inode is 248415<br><br>there is a utility , tsfindinode, to translate that into the file path.<br>you need to build this first if not yet done:<br>cd /usr/lpp/mmfs/samples/util ; make<br>, then run<br>./tsfindinode -i <inode_num> <fs_mount_point><br><br>For the IO trace analysis there is an older tool :<br>/usr/lpp/mmfs/samples/debugtools/trsum.awk.<br><br>Then there is some new stuff I've not yet used in<br>/usr/lpp/mmfs/samples/traceanz/ (always check the README)<br><br>Hope that halps a bit.<br><br>Mit freundlichen Grüßen / Kind regards<br><br>Dr. Uwe Falke<br>IT Specialist<br>Hybrid Cloud Infrastructure / Technology Consulting & Implementation<br>Services<br>+49 175 575 2877 Mobile<br>Rathausstr. 7, 09111 Chemnitz, Germany<br>uwefalke@de.ibm.com<br><br>IBM Services<br><br>IBM Data Privacy Statement<br><br>IBM Deutschland Business & Technology Services GmbH<br>Geschäftsführung: Sven Schooss, Stefan Hierl<br>Sitz der Gesellschaft: Ehningen<br>Registergericht: Amtsgericht Stuttgart, HRB 17122<br><br><br><br>From: "Czauz, Kamil" <Kamil.Czauz@Squarepoint-Capital.com><br>To: "gpfsug-discuss@spectrumscale.org"<br><gpfsug-discuss@spectrumscale.org><br>Date: 11/11/2020 23:36<br>Subject: [EXTERNAL] [gpfsug-discuss] Poor client performance with<br>high cpu usage of mmfsd process<br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br><br><br><br>We regularly run into performance issues on our clients where the client<br>seems to hang when accessing any gpfs mount, even something simple like a<br>ls could take a few minutes to complete. This affects every gpfs mount<br>on the client, but other clients are working just fine. Also the mmfsd<br>process at this point is spinning at something like 300-500% cpu.<br> <br>The only way I have found to solve this is by killing processes that may<br>be doing heavy i/o to the gpfs mounts - but this is more of an art than a<br>science. I often end up killing many processes before finding the<br>offending one.<br> <br>My question is really about finding the offending process easier. Is<br>there something similar to iotop or a trace that I can enable that can<br>tell me what files/processes and being heavily used by the mmfsd process<br>on the client?<br> <br>-Kamil<br>Confidentiality Note: This e-mail and any attachments are confidential and<br>may be protected by legal privilege. If you are not the intended<br>recipient, be aware that any disclosure, copying, distribution or use of<br>this e-mail or any attachment is prohibited. If you have received this<br>e-mail in error, please notify us immediately by returning it to the<br>sender and delete this copy from your system. We will use any personal<br>information you give to us in accordance with our Privacy Policy which can<br>be found in the Data Protection section on our corporate website<br>www.squarepoint-capital.com. Please note that e-mails may be monitored for<br>regulatory and compliance purposes. Thank you for your cooperation.<br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> <br><br><br><br><br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> </font><br> </div></blockquote>
<div dir="ltr" > </div></div><BR>
Ellei edellä ole toisin mainittu: / Unless stated otherwise above:<BR>
Oy IBM Finland Ab<BR>
PL 265, 00101 Helsinki, Finland<BR>
Business ID, Y-tunnus: 0195876-3 <BR>
Registered in Finland<BR>
<BR>