<font size=2 face="sans-serif">Hi,</font><br><br><font size=2 face="sans-serif">I was asked to elaborate a bit ( thus
also adding John and Yong Ze Chen).</font><br><br><font size=2 face="sans-serif">As written on the slide:</font><br><font size=3>One of the best ways to determine if a network layer problem
is root cause for an expel is to look at the low-level socket details dumped
in the ‘extra’ log data (mmfs dump all) saved as part of automatic data
collection on Linux GPFS nodes. </font><br><br><font size=2 face="sans-serif">So, the idea is that in expel situation,
we dump the socket state from the OS ( you can see the same using 'ss -i'
for example).</font><br><font size=2 face="sans-serif">In your example, it shows that the ca_state
is 4, there are retransmits, high rto and all the point to a network problem.</font><br><font size=2 face="sans-serif">You can find more details here: </font><a href="http://www.yonch.com/tech/linux-tcp-congestion-control-internals"><font size=2 color=blue face="sans-serif">http://www.yonch.com/tech/linux-tcp-congestion-control-internals</font></a><br><br><font size=2 face="sans-serif"><br>Regards,<br><br>Tomer Perry<br>Scalable I/O Development (Spectrum Scale)<br>email: tomp@il.ibm.com<br>1 Azrieli Center, Tel Aviv 67021, Israel<br>Global Tel: +1 720 3422758<br>Israel Tel: +972 3 9188625<br>Mobile: +972 52 2554625<br></font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">"Tomer Perry"
<TOMP@il.ibm.com></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug main discussion
list <gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">17/01/2019 13:46</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
Node expels</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><font size=2 face="sans-serif">Simon,</font><font size=3><br></font><font size=2 face="sans-serif"><br>Take a look at </font><a href="http://files.gpfsug.org/presentations/2018/USA/Scale_Network_Flow-0.8.pdf"><font size=2 color=blue face="sans-serif"><u>http://files.gpfsug.org/presentations/2018/USA/Scale_Network_Flow-0.8.pdf</u></font></a><font size=2 face="sans-serif">slide 13.</font><font size=3><br></font><font size=2 face="sans-serif"><br><br>Regards,<br><br>Tomer Perry<br>Scalable I/O Development (Spectrum Scale)<br>email: tomp@il.ibm.com<br>1 Azrieli Center, Tel Aviv 67021, Israel<br>Global Tel: +1 720 3422758<br>Israel Tel: +972 3 9188625<br>Mobile: +972 52 2554625</font><font size=3><br><br><br><br></font><font size=1 color=#5f5f5f face="sans-serif"><br>From: </font><font size=1 face="sans-serif">Simon
Thompson <S.J.Thompson@bham.ac.uk></font><font size=1 color=#5f5f5f face="sans-serif"><br>To: </font><font size=1 face="sans-serif">"gpfsug-discuss@spectrumscale.org"
<gpfsug-discuss@spectrumscale.org></font><font size=1 color=#5f5f5f face="sans-serif"><br>Date: </font><font size=1 face="sans-serif">17/01/2019
13:35</font><font size=1 color=#5f5f5f face="sans-serif"><br>Subject: </font><font size=1 face="sans-serif">[gpfsug-discuss]
Node expels</font><font size=1 color=#5f5f5f face="sans-serif"><br>Sent by: </font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><font size=3><br></font><hr noshade><font size=3><br><br></font><font size=2 face="Calibri"><br>We’ve recently been seeing quite a few node expels with messages of the
form:<br> <br>2019-01-17_11:19:30.882+0000: [W] The TCP connection to IP address 10.20.0.58
proto-pg-pf01.bear.cluster <c0n236> (socket 153) state is unexpected:
state=1 ca_state=4 snd_cwnd=1 snd_ssthresh=5 unacked=5 probes=0 backoff=7
retransmits=7 rto=26496000 rcv_ssthresh=102828 rtt=6729 rttvar=12066 sacked=0
retrans=1 reordering=3 lost=5<br>2019-01-17_11:19:30.882+0000: [I] tscCheckTcpConn: Sending debug data collection
request to node 10.20.0.58 proto-pg-pf01.bear.cluster<br>2019-01-17_11:19:30.882+0000: Sending request to collect TCP debug data
to proto-pg-pf01.bear.cluster localNode<br>2019-01-17_11:19:30.882+0000: [I] Calling user exit script gpfsSendRequestToNodes:
event sendRequestToNodes, Async command /usr/lpp/mmfs/bin/mmcommon.<br>2019-01-17_11:24:52.611+0000: [E] Timed out in 300 seconds waiting for
a commMsgCheckMessages reply from node 10.20.0.58 proto-pg-pf01.bear.cluster.
Sending expel message.<br> <br>On the client node, we see messages of the form:<br> <br>2019-01-17_11:19:31.101+0000: [N] sdrServ: Received Tcp data collection
request from 10.10.0.33<br>2019-01-17_11:19:31.102+0000: [N] GPFS will attempt to collect Tcp debug
data on this node.<br>2019-01-17_11:24:52.838+0000: [N] sdrServ: Received expel data collection
request from 10.10.0.33<br>2019-01-17_11:24:52.838+0000: [N] GPFS will attempt to collect debug data
on this node.<br>2019-01-17_11:25:02.741+0000: [N] This node will be expelled from cluster
rds.gpfs.servers due to expel msg from 10.10.12.41 (b<br>ber-les-nsd01-data.bb2.cluster in rds.gpfs.server<br>2019-01-17_11:25:03.160+0000: [N] sdrServ: Received expel data collection
request from 10.20.0.56<br> <br>They always appear to be to a specific type of hardware with the same Ethernet
controller, though the nodes are split across three data centres and we
aren’t seeing link congestion on the links between them.<br> <br>On the node I listed above, it’s not actually doing anything either as
the software on it is still being installed (i.e. it’s not doing GPFS
or any other IO other than a couple of home directories).<br> <br>Any suggestions on what “(socket 153) state is unexpected” means?<br> <br>Thanks<br> <br>Simon<br> <br> </font><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org</font></tt><font size=3 color=blue><u><br></u></font><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></tt></a><font size=3><br><br><br></font><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br></font></tt><br><br><BR>