<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
All,
<div class=""><br class="">
</div>
<div class="">mmlsnode -N waiters is great … I also appreciate the “-s” option to it. Very helpful when you know the problem started say, slightly more than half an hour ago and you therefore don’t care about sub-1800 second waiters…</div>
<div class=""><br class="">
</div>
<div class="">Kevin</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Jun 6, 2017, at 11:54 AM, Frederick Stock <<a href="mailto:stockf@us.ibm.com" class="">stockf@us.ibm.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class=""><font size="3" face="sans-serif" class="">On recent releases you can accomplish the same with the command, "mmlsnode -N waiters -L".</font><br class="">
<br class="">
<font size="3" face="sans-serif" class="">Fred<br class="">
__________________________________________________<br class="">
Fred Stock | IBM Pittsburgh Lab | 720-430-8821<br class="">
<a href="mailto:stockf@us.ibm.com" class="">stockf@us.ibm.com</a></font><br class="">
<br class="">
<br class="">
<br class="">
<font size="1" color="#5f5f5f" face="sans-serif" class="">From: </font><font size="1" face="sans-serif" class=""><a href="mailto:valdis.kletnieks@vt.edu" class="">valdis.kletnieks@vt.edu</a></font><br class="">
<font size="1" color="#5f5f5f" face="sans-serif" class="">To: </font><font size="1" face="sans-serif" class="">gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" class="">gpfsug-discuss@spectrumscale.org</a>></font><br class="">
<font size="1" color="#5f5f5f" face="sans-serif" class="">Date: </font><font size="1" face="sans-serif" class="">06/06/2017 12:46 PM</font><br class="">
<font size="1" color="#5f5f5f" face="sans-serif" class="">Subject: </font><font size="1" face="sans-serif" class="">Re: [gpfsug-discuss] gpfs waiters debugging</font><br class="">
<font size="1" color="#5f5f5f" face="sans-serif" class="">Sent by: </font><font size="1" face="sans-serif" class=""><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" class="">gpfsug-discuss-bounces@spectrumscale.org</a></font><br class="">
<hr noshade="" class="">
<br class="">
<br class="">
<br class="">
<tt class=""><font size="2" class="">On Tue, 06 Jun 2017 15:06:57 +0200, Stijn De Weirdt said:<br class="">
> oh sure, i meant waiters that last > 300 seconds or so (something that<br class="">
> could trigger deadlock). obviously we're not interested in debugging the<br class="">
> short ones, it's not that gpfs doesn't work or anything ;)<br class="">
<br class="">
At least at one time, a lot of the mm(whatever) administrative commands<br class="">
would leave one dangling waiter for the duration of the command - which<br class="">
could be a while if the command was mmdeldisk or mmrestripefs. I admit<br class="">
not having specifically checked for gpfs 4.2, but it was true for 3.2 through<br class="">
4.1....<br class="">
<br class="">
And my addition to the collective debugging knowledge: A bash one-liner to<br class="">
dump all the waiters across a cluster, sorted by wait time. Note that<br class="">
our clusters tend to be 5-8 servers, this may be painful for those of you<br class="">
who have 400+ node clusters. :)<br class="">
<br class="">
##!/bin/bash<br class="">
for i in ` mmlsnode | tail -1 | sed 's/^[ ]*[^ ]*[ ]*//'`; do ssh $i /usr/lpp/mmfs/bin/mmfsadm dump waiters | sed "s/^/$i /"; done | sort -n -r -k 3 -t' '<br class="">
<br class="">
We've found it useful - if you have 1 waiter on one node that's 1278 seconds<br class="">
old, and 3 other nodes have waiters that are 1275 seconds old, it's a good<br class="">
chance the other 3 nodes waiters are waiting on the first node's waiter to<br class="">
resolve itself....<br class="">
[attachment "attltepl.dat" deleted by Frederick Stock/Pittsburgh/IBM] _______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a><br class="">
</font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class=""><tt class=""><font size="2" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt class=""><font size="2" class=""><br class="">
</font></tt><br class="">
</div>
</blockquote>
</div>
</div>
<div class="">
<div class="">—</div>
<div class="">Kevin Buterbaugh - Senior System Administrator</div>
<div class="">Vanderbilt University - Advanced Computing Center for Research and Education</div>
<div class=""><a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" class="">Kevin.Buterbaugh@vanderbilt.edu</a> - (615)875-9633</div>
<div class=""><br class="">
</div>
<br class="Apple-interchange-newline">
</div>
<br class="">
</body>
</html>