<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>I have seen these on our cluster after the IB network goes down
(GPFS still runs over ethernet) and then comes back up. They will
retry forever it seems, even after the IB is healthy again. The
effect they seem to have is that verbs connections between some
nodes breaks and GPFS uses ethernet/ipoib instead. You may see
messages in your mmfs.log.latest about verbs being disabled "due
to too many errors". You can also see fewer verbs connections
between nodes in "mmfsadm test verbs conn" output.</p>
<p>Restarting GPFS on the nodes with waiters has fixed the issue for
me, I don't know if IBM has any other tricks to fix this without a
restart.<br>
</p>
<p>--Joey</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 9/12/19 8:16 AM, Damir Krstic wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAKV+WqfbysaKM=We3NNm5essS=--Ej=wu+yzvLfousVp4cfOYw@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div>On my cluster I have seen couple of long waiters such as
this:</div>
<div><br>
</div>
<div>gss01: Waiting 16.8543 sec since 09:07:02, ignored, thread
46230 VerbsReconnectThread: delaying for 43.145624000 more
seconds, reason: delaying for next reconnect attempt</div>
<div><br>
</div>
<div>I tried searching on gpfs wiki for this type of waiter, but
was unable to find anything of value. <br>
</div>
<div><br>
</div>
<div>Is this something to pay attention to, and what does this
waiter mean?</div>
<div><br>
</div>
<div>Thank you.</div>
<div>Damir<br>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
<a class="moz-txt-link-freetext" href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a>
</pre>
</blockquote>
</body>
</html>