<div dir="ltr">if this happens you should check a couple of things : <div><br></div><div>1. are you under memory pressure or even worse started swapping . </div><div>2. is there any core running at ~ 0% idle - run top , press 1 and check the idle column. </div><div>3. is there any single thread running at ~100% - run top , press shift - h and check what the CPU % shows for the top 5 processes. </div><div><br></div><div>if you want to go the extra mile, you could run perf top -p $PID_OF_MMFSD and check what the top cpu consumers are. </div><div>confirming and providing data to any of the above being true could be the missing piece why nobody was able to find it, as this is stuff unfortunate nobody ever looks at. even a trace won't help if any of the above is true as all you see is that the system behaves correct according to the trace, its doesn't appear busy, </div><div><br></div><div>Sven</div><div><br></div><div><br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Nov 27, 2018 at 8:03 AM Oesterlin, Robert <<a href="mailto:Robert.Oesterlin@nuance.com">Robert.Oesterlin@nuance.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="m_767838434069789210WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">I have seen something like this in the past, and I have resorted to a cluster restart as well. :-( IBM and I could never really track it down, because I could not get a dump at the time of occurrence. However,
you might take a look at your NSD servers, one at a time. As I recall, we thought it was a stuck thread on one of the NSD servers, and when we restarted the “right” one it cleared the block.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The other thing I’ve done in the past to isolate problems like this (since this is related to tokens) is to look at the “token revokes” on each node, looking for ones that are sticking around for a long time.
I tossed together a quick script and ran it via mmdsh on all the node. Not pretty, but it got the job done. Run this a few times, see if any of the revokes are sticking around for a long time<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">#!/bin/sh<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">rm -f /tmp/revokelist<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">/usr/lpp/mmfs/bin/mmfsadm dump tokenmgr | grep -A 2 'revokeReq list' > /tmp/revokelist 2> /dev/null<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">if [ $? -eq 0 ]; then<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> /usr/lpp/mmfs/bin/mmfsadm dump tscomm > /tmp/tscomm.out<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> for n in `cat /tmp/revokelist | grep msgHdr | awk '{print $5}'`; do<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> grep $n /tmp/tscomm.out | tail -1<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> done<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> rm -f /tmp/tscomm.out<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">fi<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><u></u> <u></u></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><u></u> <u></u></span></p>
<p class="MsoNormal">Bob Oesterlin<u></u><u></u></p>
<p class="MsoNormal">Sr Principal Storage Engineer, Nuance<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><u></u> <u></u></span></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black"><<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a>> on behalf of Simon Thompson <<a href="mailto:S.J.Thompson@bham.ac.uk" target="_blank">S.J.Thompson@bham.ac.uk</a>><br>
<b>Reply-To: </b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a>><br>
<b>Date: </b>Tuesday, November 27, 2018 at 9:27 AM<br>
<b>To: </b>"<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a>" <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a>><br>
<b>Subject: </b>[EXTERNAL] [gpfsug-discuss] Hanging file-systems<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><u></u> <u></u></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">I have a file-system which keeps hanging over the past few weeks. Right now, its offline and taken a bunch of services out with it.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">(I have a ticket with IBM open about this as well)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We see for example:</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Waiting 305.0391 sec since 15:17:02, monitored, thread 24885 SharedHashTabFetchHandlerThread: on ThCond 0x7FE30000B408 (MsgRecordCondvar), re</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">ason 'RPC wait' for tmMsgTellAcquire1 on node 10.10.12.42 <c1n9></span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">and on that node:</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Waiting 292.4581 sec since 15:17:22, monitored, thread 20368 SharedHashTabFetchHandlerThread: on ThCond 0x7F3C2929719</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">8 (TokenCondvar), reason 'wait for SubToken to become stable'</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">On this node, if you dump tscomm, you see entries like:</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Pending messages:</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> msg_id 376617, service 13.1, msg_type 20 'tmMsgTellAcquire1', n_dest 1, n_pending 1</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> this 0x7F3CD800B930, n_xhold 1, cl 0, cbFn 0x0, age 303 sec</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> sent by 'SharedHashTabFetchHandlerThread' (0x7F3DD800A6C0)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> dest <c0n9> status pending , err 0, reply len 0 by TCP connection</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">c0n9 is itself.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">This morning when this happened, the only way to get the FS back online was to shutdown the entire cluster.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Any pointers for next place to look/how to fix?</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Simon</span><u></u><u></u></p>
</div>
</div>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
</blockquote></div>