<html><head><meta http-equiv="Content-Type" content="text/html charset=iso-8859-1"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">For my system I always use a dedicated admin network - as described in the gpfs manuals - for a gpfs cluster on 10/40GbE where the system will be heavily loaded.<div><br></div><div>The difference in the stability of the system is very noticeable.</div><div><br></div><div>Not sure how/if this would work on GSS - IBM ought to know :-)</div><div><br></div><div>Vic<br><div><br></div><div><br><div><div>On 21 Aug 2014, at 14:18, Salvatore Di Nardo <<a href="mailto:sdinardo@ebi.ac.uk">sdinardo@ebi.ac.uk</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">
<meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type">
<div bgcolor="#FFFFFF" text="#000000">
This is an interesting point!<br>
<br>
We use ethernet ( 10g links on the clients) but we dont have a
separate network for the admin network. <br>
<br>
Could you explain this a bit further, because the clients and the
servers we have are on different subnet so the packet are routed.. I
don't see a practical way to separate them. The clients are blades
in a chassis so even if i create 2 interfaces, they will physically
use the came "cable" to go to the first switch. even the clients (
600 clients) have different subsets.<br>
<br>
I will forward this consideration to our network admin , so see if
we can work on a dedicated network.<br>
<br>
thanks for your tip.<br>
<br>
Regards,<br>
Salvatore<br>
<br>
<br>
<br>
<br>
<div class="moz-cite-prefix">On 21/08/14 14:03, Vic Cornell wrote:<br>
</div>
<blockquote cite="mid:9B247872-CD75-4F86-A10E-33AAB6BD414A@gmail.com" type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
Hi Salvatore,
<div><br>
</div>
<div>Are you using ethernet or infiniband as the GPFS interconnect
to your clients?</div>
<div><br>
</div>
<div>If 10/40GbE - do you have a separate admin network?</div>
<div><br>
</div>
<div>I have seen behaviour similar to this where the storage
traffic causes congestion and the "admin" traffic gets lost or
delayed causing expels.</div>
<div><br>
</div>
<div>Vic</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
<div>
<div>On 21 Aug 2014, at 10:04, Salvatore Di Nardo <<a moz-do-not-send="true" href="mailto:sdinardo@ebi.ac.uk">sdinardo@ebi.ac.uk</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type">
<div bgcolor="#FFFFFF" text="#000000"> Thanks for the
feedback, but we managed to find a scenario that excludes
network problems.<br>
<br>
we have a file called <b><i>input_file</i></b> of nearly
100GB:<br>
<br>
if from <b>client A</b> we do:<br>
<br>
cat input_file >> output_file<br>
<br>
it start copying.. and we see waiter goeg a bit up,secs
but then they flushes back to 0, so we xcan say that the
copy proceed well...<br>
<br>
<br>
if now we do the same from another client ( or just
another shell on the same client) <b>client B</b> :<br>
<br>
cat input_file >> output_file<br>
<br>
<br>
( in other words we are trying to write to the same
destination) all the waiters gets up until one node get
expelled.<br>
<br>
<br>
Now, while its understandable that the destination file is
locked for one of the "cat", so have to wait ( and since
the file is BIG , have to wait for a while), its not
understandable why it stop the renewal lease. <br>
Why its doen't return just a timeout error on the copy
instead to expel the node? We can reproduce this every
time, and since our users to operations like this on files
over 100GB each you can imagine the result.<br>
<br>
<br>
<br>
As you can imagine even if its a bit silly to write at the
same time to the same destination, its also quite common
if we want to dump to a log file logs and for some reason
one of the writers, write for a lot of time keeping the
file locked.<br>
Our expels are not due to network congestion, but because
a write attempts have to wait another one. What i really
dont understand is why to take a so expreme mesure to
expell jest because a process is waiteing "to too much
time".<br>
<br>
<br>
I have ticket opened to IBM for this and the issue is
under investigation, but no luck so far..<br>
<br>
Regards,<br>
Salvatore<br>
<br>
<br>
<br>
<div class="moz-cite-prefix">On 21/08/14 09:20, Jez Tucker
(Chair) wrote:<br>
</div>
<blockquote cite="mid:53F5ABD7.80107@gpfsug.org" type="cite">
<meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type">
Hi there,<br>
<br>
I've seen the on several 'stock'? 'core'? GPFS system
(we need a better term now GSS is out) and seen ping
'working', but alongside ejections from the cluster.<br>
The GPFS internode 'ping' is somewhat more circumspect
than unix ping - and rightly so.<br>
<br>
In my experience this has _always_ been a network issue
of one sort of another. If the network is experiencing
issues, nodes will be ejected.<br>
Of course it could be unresponsive mmfsd or high
loadavg, but I've seen that only twice in 10 years over
many versions of GPFS.<br>
<br>
You need to follow the logs through from each machine in
time order to determine who could not see who and in
what order.<br>
Your best way forward is to log a SEV2 case with IBM
support, directly or via your OEM and collect and supply
a snap and traces as required by support.<br>
<br>
Without knowing your full setup, it's hard to help
further.<br>
<br>
Jez<br>
<br>
<div class="moz-cite-prefix">On 20/08/14 08:57,
Salvatore Di Nardo wrote:<br>
</div>
<blockquote cite="mid:53F454E3.40803@ebi.ac.uk" type="cite">
<meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type">
Still problems. Here some more detailed examples:<br>
<br>
<b>EXAMPLE 1:</b><br>
<blockquote>
<blockquote>
<blockquote><b><tt>EBI5-220</tt></b><b><tt> (
CLIENT)</tt></b><b><br>
</b><tt>Tue Aug 19 11:03:04.980 2014: <b>Timed
out waiting for a reply from node <GSS02B
IP> gss02b</b></tt><br>
<tt>Tue Aug 19 11:03:04.981 2014: Request sent
to <GSS02A IP> (gss02a in <a moz-do-not-send="true" href="http://gss.ebi.ac.uk/">GSS.ebi.ac.uk</a>)
to expel <GSS02B IP> (gss02b in <a moz-do-not-send="true" href="http://gss.ebi.ac.uk/">GSS.ebi.ac.uk</a>)
from cluster <a moz-do-not-send="true" href="http://gss.ebi.ac.uk/">GSS.ebi.ac.uk</a></tt><br>
<tt>Tue Aug 19 11:03:04.982 2014: This node will
be expelled from cluster <a moz-do-not-send="true" href="http://gss.ebi.ac.uk/">GSS.ebi.ac.uk</a>
due to expel msg from <EBI5-220 IP>
(ebi5-220)</tt><br>
<tt>Tue Aug 19 11:03:09.319 2014: Cluster
Manager connection broke. Probing cluster <a moz-do-not-send="true" href="http://gss.ebi.ac.uk/">GSS.ebi.ac.uk</a></tt><br>
<tt>Tue Aug 19 11:03:10.321 2014: Unable to
contact any quorum nodes during cluster probe.</tt><br>
<tt>Tue Aug 19 11:03:10.322 2014: Lost
membership in cluster <a moz-do-not-send="true" href="http://gss.ebi.ac.uk/">GSS.ebi.ac.uk</a>.
Unmounting file systems.</tt><br>
<tt>Tue Aug 19 11:03:10 BST 2014: mmcommon
preunmount invoked. File system: gpfs1
Reason: SGPanic</tt><br>
<tt>Tue Aug 19 11:03:12.066 2014: Connecting to
<GSS02A IP> gss02a <c1p687></tt><br>
<tt>Tue Aug 19 11:03:12.070 2014: Connected to
<GSS02A IP> gss02a <c1p687></tt><br>
<tt>Tue Aug 19 11:03:17.071 2014: Connecting to
<GSS02B IP> gss02b <c1p686></tt><br>
<tt>Tue Aug 19 11:03:17.072 2014: Connecting to
<GSS03B IP> gss03b <c1p685></tt><br>
<tt>Tue Aug 19 11:03:17.079 2014: Connecting to
<GSS03A IP> gss03a <c1p684></tt><br>
<tt>Tue Aug 19 11:03:17.080 2014: Connecting to
<GSS01B IP> gss01b <c1p683></tt><br>
<tt>Tue Aug 19 11:03:17.079 2014: Connecting to
<GSS01A IP> gss01a <c1p1></tt><br>
<tt>Tue Aug 19 11:04:23.105 2014: Connected to
<GSS02B IP> gss02b <c1p686></tt><br>
<tt>Tue Aug 19 11:04:23.107 2014: Connected to
<GSS03B IP> gss03b <c1p685></tt><br>
<tt>Tue Aug 19 11:04:23.112 2014: Connected to
<GSS03A IP> gss03a <c1p684></tt><br>
<tt>Tue Aug 19 11:04:23.115 2014: Connected to
<GSS01B IP> gss01b <c1p683></tt><br>
<tt>Tue Aug 19 11:04:23.121 2014: Connected to
<GSS01A IP> gss01a <c1p1></tt><br>
<tt>Tue Aug 19 11:12:28.992 2014: Node
<GSS02A IP> (gss02a in <a moz-do-not-send="true" href="http://gss.ebi.ac.uk/">GSS.ebi.ac.uk</a>)
is now the Group Leader.</tt><br>
<br>
<b><tt>GSS02B ( NSD SERVER)</tt></b><br>
<tt>...<br>
Tue Aug 19 11:03:17.070 2014: Killing
connection from <b><EBI5-220 IP></b>
because the group is not ready for it to
rejoin, err 46<br>
Tue Aug 19 11:03:25.016 2014: Killing
connection from <EBI5-102 IP> because
the group is not ready for it to rejoin, err
46<br>
Tue Aug 19 11:03:28.080 2014: Killing
connection from </tt><tt><tt><b><EBI5-220
IP></b></tt> because the group is not
ready for it to rejoin, err 46<br>
Tue Aug 19 11:03:36.019 2014: Killing
connection from <EBI5-102 IP> because
the group is not ready for it to rejoin, err
46<br>
Tue Aug 19 11:03:39.083 2014: Killing
connection from </tt><tt><tt><b><EBI5-220
IP></b></tt> because the group is not
ready for it to rejoin, err 46<br>
Tue Aug 19 11:03:47.023 2014: Killing
connection from <EBI5-102 IP> because
the group is not ready for it to rejoin, err
46<br>
Tue Aug 19 11:03:50.088 2014: Killing
connection from </tt><tt><tt><b><EBI5-220
IP></b></tt> because the group is not
ready for it to rejoin, err 46<br>
Tue Aug 19 11:03:52.218 2014: Killing
connection from </tt><tt><tt><EBI5-043
IP></tt> because the group is not ready
for it to rejoin, err 46<br>
Tue Aug 19 11:03:58.030 2014: Killing
connection from <EBI5-102 IP> because
the group is not ready for it to rejoin, err
46<br>
Tue Aug 19 11:04:01.092 2014: Killing
connection from </tt><tt><tt><b><EBI5-220
IP></b></tt> because the group is not
ready for it to rejoin, err 46<br>
Tue Aug 19 11:04:03.220 2014: Killing
connection from </tt><tt><tt><tt><EBI5-043
IP></tt></tt> because the group is not
ready for it to rejoin, err 46<br>
Tue Aug 19 11:04:09.034 2014: Killing
connection from <EBI5-102 IP> because
the group is not ready for it to rejoin, err
46<br>
Tue Aug 19 11:04:12.096 2014: Killing
connection from </tt><tt><tt><b><EBI5-220
IP></b></tt> because the group is not
ready for it to rejoin, err 46<br>
Tue Aug 19 11:04:14.224 2014: Killing
connection from </tt><tt><tt><tt><EBI5-043
IP></tt></tt> because the group is not
ready for it to rejoin, err 46<br>
Tue Aug 19 11:04:20.037 2014: Killing
connection from <EBI5-102 IP> because
the group is not ready for it to rejoin, err
46<br>
Tue Aug 19 11:04:23.103 2014: Accepted and
connected to </tt><tt><tt><b><EBI5-220
IP></b></tt> ebi5-220 <c0n618><br>
...</tt><br>
<br>
<b><tt>GSS02a ( NSD SERVER)</tt></b><br>
<tt>Tue Aug 19 11:03:04.980 2014: Expel
<GSS02B IP> (gss02b) request from
<EBI5-220 IP> (ebi5-220 in <a moz-do-not-send="true" href="http://ebi-cluster.ebi.ac.uk/">ebi-cluster.ebi.ac.uk</a>).
Expelling: <EBI5-220 IP> (ebi5-220 in <a moz-do-not-send="true" href="http://ebi-cluster.ebi.ac.uk/">ebi-cluster.ebi.ac.uk</a>)</tt><br>
<tt>Tue Aug 19 11:03:12.069 2014: Accepted and
connected to <EBI5-220 IP> ebi5-220
<c0n618></tt><br>
<br>
<br>
</blockquote>
</blockquote>
</blockquote>
===============================================<br>
<b>EXAMPLE 2</b>:<br>
<br>
<blockquote>
<blockquote>
<blockquote><b><tt>EBI5-038</tt></b><br>
<tt>Tue Aug 19 11:32:34.227 2014: <b>Disk lease
period expired in cluster <a moz-do-not-send="true" href="http://gss.ebi.ac.uk/">GSS.ebi.ac.uk</a>.
Attempting to reacquire lease.</b></tt><br>
<tt>Tue Aug 19 11:33:34.258 2014: <b>Lease is
overdue. Probing cluster <a moz-do-not-send="true" href="http://gss.ebi.ac.uk/">GSS.ebi.ac.uk</a></b></tt><br>
<tt>Tue Aug 19 11:35:24.265 2014: Close
connection to <GSS02A IP> gss02a
<c1n2> (Connection reset by peer).
Attempting reconnect.</tt><br>
<tt>Tue Aug 19 11:35:24.865 2014: Close
connection to <EBI5-014 IP> ebi5-014
<c1n457> (Connection reset by peer).
Attempting reconnect.</tt><br>
<tt>...</tt><br>
<tt>LOT MORE RESETS BY PEER</tt><br>
<tt>...</tt><br>
<tt>Tue Aug 19 11:35:25.096 2014: Close
connection to <EBI5-167 IP> ebi5-167
<c1n155> (Connection reset by peer).
Attempting reconnect.</tt><br>
<tt>Tue Aug 19 11:35:25.267 2014: Connecting to
<GSS02A IP> gss02a <c1n2></tt><br>
<tt>Tue Aug 19 11:35:25.268 2014: Close
connection to <GSS02A IP> gss02a
<c1n2> (Connection failed because
destination is still processing previous node
failure)</tt><br>
<tt>Tue Aug 19 11:35:26.267 2014: Retry
connection to <GSS02A IP> gss02a
<c1n2></tt><br>
<tt>Tue Aug 19 11:35:26.268 2014: Close
connection to <GSS02A IP> gss02a
<c1n2> (Connection failed because
destination is still processing previous node
failure)</tt><br>
<tt>Tue Aug 19 11:36:24.276 2014: Unable to
contact any quorum nodes during cluster probe.</tt><br>
<tt>Tue Aug 19 11:36:24.277 2014: <b>Lost
membership in cluster <a moz-do-not-send="true" href="http://gss.ebi.ac.uk/">GSS.ebi.ac.uk</a>.
Unmounting file systems.</b></tt><br>
<br>
<b><tt>GSS02a</tt></b><br>
<tt>Tue Aug 19 11:35:24.263 2014: Node
<EBI5-038 IP> (ebi5-038 in <a moz-do-not-send="true" href="http://ebi-cluster.ebi.ac.uk/">ebi-cluster.ebi.ac.uk</a>)
<b>is being expelled because of an expired
lease.</b> Pings sent: 60. Replies received:
60.</tt><br>
</blockquote>
</blockquote>
</blockquote>
<br>
<br>
<br>
In example 1 seems that an NSD was not repliyng to the
client, but the servers seems working fine.. how can i
trace better ( to solve) the problem? <br>
<br>
In example 2 it seems to me that for some reason the
manager are not renewing the lease in time. when this
happens , its not a single client. <br>
Loads of them fail to get the lease renewed. Why this
is happening? how can i trace to the source of the
problem?<br>
<br>
<br>
<br>
Thanks in advance for any tips.<br>
<br>
Regards,<br>
Salvatore<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at <a moz-do-not-send="true" href="http://gpfsug.org/">gpfsug.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a>
</pre>
</blockquote>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at <a moz-do-not-send="true" href="http://gpfsug.org/">gpfsug.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a>
</pre>
</blockquote>
<br>
</div>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a moz-do-not-send="true" href="http://gpfsug.org/">gpfsug.org</a><br>
<a moz-do-not-send="true" href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at <a href="http://gpfsug.org">gpfsug.org</a>
<a class="moz-txt-link-freetext" href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a>
</pre>
</blockquote>
<br>
</div>
_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://gpfsug.org">gpfsug.org</a><br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br></blockquote></div><br></div></div></body></html>