<span style=" font-size:10pt;font-family:sans-serif">Hi Heiner,</span><br><br><span style=" font-size:10pt;font-family:sans-serif">I'm not really
able to give you insights into the decision of the events' states. Maybe
somebody else is able to answer here.</span><br><br><span style=" font-size:10pt;font-family:sans-serif">But about your
triggering debug data collection question, please have a look at this documentation
page:</span><br><a href="https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.4/com.ibm.spectrum.scale.v5r04.doc/bl1adv_createscriptforevents.htm"><span style=" font-size:10pt;color:blue;font-family:sans-serif">https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.4/com.ibm.spectrum.scale.v5r04.doc/bl1adv_createscriptforevents.htm</span></a><br><br><span style=" font-size:10pt;font-family:sans-serif">This feature is
in the product since the 5.0.x versions and should be helpful here. </span><br><span style=" font-size:10pt;font-family:sans-serif">It will trigger
your eventsCallback script when the event is raised. One of the script's
arguments is the event name. So it is possible to create a script, that</span><br><span style=" font-size:10pt;font-family:sans-serif">checks for the
event name longwaiters_found and then triggers a mmdiag --deadlock and
write it into a txt file. </span><br><br><span style=" font-size:10pt;font-family:sans-serif">The script call
has a hard time out of 60 seconds so it does not interfere too much with
the mmsysmon internals, but better would be a run time less than 1 second.</span><br><br><span style=" font-size:9pt">Mit freundlichen Grüßen / Kind regards</span><br><br><span style=" font-size:11pt"><b>Anna Greim</b></span><br><br><span style=" font-size:12pt;color:#0060a0">Software Engineer, Spectrum
Scale Development</span><br><span style=" font-size:12pt;color:#0060a0">IBM Systems</span><table width=800 style="border-collapse:collapse;"><tr height=8><td width=800 colspan=3 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><tr height=8><td width=800 colspan=3 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><tr height=8><td width=94 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><td width=344 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><td width=361 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><tr height=8><td width=94 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><td width=344 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><td width=361 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><tr height=8><td width=94 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><td width=344 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><td width=361 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><tr height=8><td width=800 colspan=3 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><tr height=8><td width=800 colspan=3 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><a href="https://www.ibm.com/privacy/us/en/"><span style=" font-size:8pt;color:blue"><u>IBM
Data Privacy Statement</u></span></a><span style=" font-size:8pt;color:#a2a2a2"></span><tr height=8><td width=800 colspan=3 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:8pt;color:#a2a2a2">IBM
Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats:
Gregor Pillen<br>Geschäftsführung: Dirk Wittkopp<br>Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart,
HRB 243294 </span></table><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:10pt;font-family:sans-serif"><br></span></p><br><br><br><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">From:
</span><span style=" font-size:9pt;font-family:sans-serif">"Billich
Heinrich Rainer (ID SD)" <heinrich.billich@id.ethz.ch></span><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">To:
</span><span style=" font-size:9pt;font-family:sans-serif">gpfsug
main discussion list <gpfsug-discuss@spectrumscale.org></span><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Date:
</span><span style=" font-size:9pt;font-family:sans-serif">16/04/2020
10:36</span><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Subject:
</span><span style=" font-size:9pt;font-family:sans-serif">[EXTERNAL]
[gpfsug-discuss] Mmhealth events longwaiters_found and
deadlock_detected</span><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Sent
by: </span><span style=" font-size:9pt;font-family:sans-serif">gpfsug-discuss-bounces@spectrumscale.org</span><br><hr noshade><br><br><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:sans-serif">Hello,</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:sans-serif"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:sans-serif">I’m
puzzled about the difference between the two mmhealth events</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:sans-serif"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:sans-serif">longwaiters_found
ERROR Detected Spectrum Scale long-waiters</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:sans-serif"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:sans-serif">and</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:sans-serif"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:sans-serif">deadlock_detected
WARNING The cluster detected a
Spectrum Scale filesystem deadlock</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:sans-serif"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:sans-serif">Especially
why the later has level WARNING only while the first has level ERROR? Longwaiters_found
is based on the output of ‘mmdiag –deadlock’ and occurs much more often
on our clusters, while the later probably is triggered by an external event
and no internal mmsysmon check? Deadlock detection is handled by mmfsd?
Whenever a deadlock is detected some debug data is collected, which
is not true for longwaiters_detected. Hm, so why is no deadlock detected
whenever mmdiag –deadlock shows waiting threads? Shouldn’t the
severity be the opposite way?</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:sans-serif"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:sans-serif">Finally:
Can we trigger some debug data collection whenever a longwaiters_found
event happens – just getting the output of ‘mmdiag –deadlock’ on the
single node could give some hints. Without I don’t see any real chance
to take any action.</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:sans-serif"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:sans-serif">Thank
you,</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:sans-serif"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:sans-serif">Heiner</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:sans-serif">--
</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;color:#104160;font-family:Arial">=======================</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;color:#104160;font-family:Arial">Heinrich
Billich</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;color:#104160;font-family:Arial">ETH
Zürich</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;color:#104160;font-family:Arial">Informatikdienste</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;color:#104160;font-family:Arial">Tel.:
+41 44 632 72 56</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;color:#104160;font-family:Arial">heinrich.billich@id.ethz.ch</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;color:#104160;font-family:Arial">========================</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:sans-serif"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:sans-serif"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:sans-serif"> </span><tt><span style=" font-size:10pt">_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></span></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><span style=" font-size:10pt">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></tt></a><tt><span style=" font-size:10pt"><br></span></tt></p><p style="margin-top:0px;margin-Bottom:0px"></p><p style="margin-top:0px;margin-Bottom:0px"></p><BR>