<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi John,
<div class=""><br class="">
</div>
<div class="">Thanks for the explanation and the link to your presentation … just what I was needing.</div>
<div class=""><br class="">
</div>
<div class="">Kevin<br class="">
<div class="">
<div class="">—</div>
<div class="">Kevin Buterbaugh - Senior System Administrator</div>
<div class="">Vanderbilt University - Advanced Computing Center for Research and Education</div>
<div class=""><a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" class="">Kevin.Buterbaugh@vanderbilt.edu</a> - (615)875-9633</div>
<div class=""><br class="">
</div>
<br class="Apple-interchange-newline">
</div>
<div>
<blockquote type="cite" class="">
<div class="">On Sep 27, 2018, at 11:37 AM, John Lewars <<a href="mailto:jlewars@us.ibm.com" class="">jlewars@us.ibm.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class=""><font size="2" face="sans-serif" class="">Hi Kevin,</font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">The message below indicates that the mmfsd code had a pending message on a socket, and, when it looked at the low level socket statistics, GPFS found indications that the TCP connection was in a 'bad state'. GPFS determines
a connection to be a 'bad state' if:</font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">1) the CA_STATE for the socket is not in 0 (or open) state, which means the state must be disorder, recovery, or loss. See this paper for more details on CA_STATE:
</font><br class="">
<a href="https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.aalto.fi%2Fdownload%2Fattachments%2F69901948%2FTCP-CongestionControlFinal.pdf&data=02%7C01%7CKevin.Buterbaugh%40Vanderbilt.Edu%7C106784daf7e9407e00ef08d624978b49%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636736630629232470&sdata=gTy7ZlQruWdS9uOBQZQRqABi8AwFEXjqKQvfm%2FSnsHo%3D&reserved=0" originalsrc="https://wiki.aalto.fi/download/attachments/69901948/TCP-CongestionControlFinal.pdf" shash="xWg9d0MCh9SA3heLzQ0sGZHQpJbmgrcOS01oeBTB0ERLzHKQ66tQa/EVRwP0YfRZYgED3ZGk39OcF/4mX0zEuVA/kxbm8eNhEZBbkQbRepFnW2cdRxFcRL6Pkt1obIcMQTXXRhAE7wIcmjVv2GJV3Cq9Be8OEshqrYimSGl2JOk=" class=""><font size="2" color="blue" face="sans-serif" class="">https://wiki.aalto.fi/download/attachments/69901948/TCP-CongestionControlFinal.pdf</font></a><br class="">
<br class="">
<font size="2" face="sans-serif" class="">or</font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">2) the RTO is greater than 10 seconds and there are unacknowledged messages pending on the socket (unacked > 0).
</font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">In the example below we see that <b class="">
rto=27008000, </b>which means that the non-fast path TCP retransmission timeout is about 27 seconds, and that probably means the connection has experienced significant packet loss. If there was no expel following this message, I would suspect there was some
transient packet loss that was recovered from.</font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">There are plenty of places in which to find more details on RTO, but you might want to start with wikipedia (</font><a href="https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FTransmission_Control_Protocol&data=02%7C01%7CKevin.Buterbaugh%40Vanderbilt.Edu%7C106784daf7e9407e00ef08d624978b49%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636736630629232470&sdata=2AodSr9rUeGUtRUU37hDR29ejB%2BUmPxhGbtNGx4%2BSzY%3D&reserved=0" originalsrc="https://en.wikipedia.org/wiki/Transmission_Control_Protocol" shash="MS6X7B1f1w9ik8YboRGZUmisALwlp3TuwsG/zZJRbt86+r/Uf6sj4ao01FkuPC9wHJm86AB7CcA5TVjNNmEi+lSX+V9w0xebs4GZZrzsENAVLrWpofcy7o4sOMPw/i8oiWPG42PzlbZ58jlp7gcTBgl1umuUgOGIDqe1RWz+81k=" class=""><font size="2" color="blue" face="sans-serif" class="">https://en.wikipedia.org/wiki/Transmission_Control_Protocol</font></a><font size="2" face="sans-serif" class="">)
which states:</font><br class="">
<br class="">
<font size="3" class="">In addition, senders employ a <i class="">retransmission timeout</i>(RTO) that is based on the estimated
</font><a href="https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FRound-trip_time&data=02%7C01%7CKevin.Buterbaugh%40Vanderbilt.Edu%7C106784daf7e9407e00ef08d624978b49%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636736630629232470&sdata=i2g8cJKLq4TDNg1UNVXZ23kNsiiPNMWHgllAmMfe0tg%3D&reserved=0" originalsrc="https://en.wikipedia.org/wiki/Round-trip_time" shash="yIkgwjtFiDHnCTX/61SSmUIu83hP90Q1ZywL8zFD7XGsw4mytpIIgW+xIykHfU99LDrbJ6jVEEv6IlQI9r1M+8JgFFohU3DMATYXC6nk7t93vYGLu3YqBqpGDN1yXIjrTxDNHa3oWkqRSidjFVjkeHecloqIr/Qy+ejCFihAowY=" class=""><font size="3" color="blue" class=""><u class="">round-trip
time</u></font></a><font size="3" class=""> (or RTT) between the sender and receiver, as well as the variance in this round trip time. The behavior of this timer is specified in
</font><a href="https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc6298&data=02%7C01%7CKevin.Buterbaugh%40Vanderbilt.Edu%7C106784daf7e9407e00ef08d624978b49%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636736630629232470&sdata=%2BeYWIqeeGCSQwaqqVivAeksyaio4C%2BXMr2%2BMGEKvZVM%3D&reserved=0" originalsrc="https://tools.ietf.org/html/rfc6298" shash="Rmh0k0fI/qfMjxB4QpjN9qEwVjZDHDQmosc22Zhv0y0VnNp4sBtXApShjELLeK1Wwp1PoVhVjwPgTkhpAjhNnrJnoIniQvqG3wlaeZoIZx4dWAi6QmOK1ZxaA6Owr+jS6xkpRRqu9ch51cS3N/yqq6cn+vj0h07ogPLvlE3vFk8=" class=""><font size="3" color="blue" class=""><u class="">RFC
6298</u></font></a><font size="3" class="">. There are subtleties in the estimation of RTT. For example, senders must be careful when calculating RTT samples for retransmitted packets; typically they use
</font><a href="https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FKarn%2527s_Algorithm&data=02%7C01%7CKevin.Buterbaugh%40Vanderbilt.Edu%7C106784daf7e9407e00ef08d624978b49%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636736630629232470&sdata=TkOxj60PftecM92%2FPpIv2Pznqo6c9LFO3lAQECRUG0c%3D&reserved=0" originalsrc="https://en.wikipedia.org/wiki/Karn%27s_Algorithm" shash="Qyc1v3gYczWMQen3QGE31jO2BLUFB+FcL7zqwWqtO60un198qQvLaw67LPk7ozRywbmJLtFXwqGx4hLUoDc6x+3ilL+SadBne73vwF0lHIjJRx4hb9aL/g7D/+jB9G52vc5ZI8YmI4pRmLoynvLCZLBQOzHSDs2ZKoj0F59GX6Q=" class=""><font size="3" color="blue" class=""><u class="">Karn's
Algorithm</u></font></a><font size="3" class=""> or TCP timestamps (see </font><a href="https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc1323&data=02%7C01%7CKevin.Buterbaugh%40Vanderbilt.Edu%7C106784daf7e9407e00ef08d624978b49%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636736630629232470&sdata=UovNpxntUSX7xQkhxNnn3jZHKsnSONfmQOUDUf873OE%3D&reserved=0" originalsrc="https://tools.ietf.org/html/rfc1323" shash="pKDeLxTo+Ykq6ldHxy3Vt+96PHpUWHel6PLQJSIKy20ze52JWhF2l8dFey5Hnx5w7o8qVBOMv9z/QNo215kCB7+Hk0Yd2iRZFLPhudzuqrCDfYCdRTCydNspVnMklKqZd4/QXy6ulLk4xrFI6p6xvza0Vabit3p0Oea52GxjN08=" class=""><font size="3" color="blue" class=""><u class="">RFC
1323</u></font></a><font size="3" class="">). These individual RTT samples are then averaged over time to create a Smoothed Round Trip Time (SRTT) using
</font><a href="https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FVan_Jacobson&data=02%7C01%7CKevin.Buterbaugh%40Vanderbilt.Edu%7C106784daf7e9407e00ef08d624978b49%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636736630629232470&sdata=yxUf%2F7GxtYVQMMmd3WDrlEpOrToiTQmUcCJq7HB6pHU%3D&reserved=0" originalsrc="https://en.wikipedia.org/wiki/Van_Jacobson" shash="UwJTcKnJjDaNzMNlrOqS16VFVGpfTUNvmTHaeohuxl9YLfCN0F2Z9WnUithxJCbJkI0sB2irgmdceP4+Qekq5wJxC60LnhDCqzobIIW/RCwRDBNBsWYht/Uq4WDtTIF5tWNzCG8TRlz728EcuHgXKRxmBxe+5zGF9+wkLHendoc=" class=""><font size="3" color="blue" class=""><u class="">Jacobson</u></font></a><font size="3" class="">'s
algorithm. This SRTT value is what is finally used as the round-trip time estimate.
</font><br class="">
<font size="2" face="sans-serif" class="">[. . .]</font><br class="">
<font size="3" class="">Reliability is achieved by the sender detecting lost data and retransmitting it. TCP uses two primary techniques to identify loss. Retransmission timeout (abbreviated as RTO) and duplicate cumulative acknowledgements (DupAcks).
</font><font size="2" face="sans-serif" class=""><br class="">
</font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">Note that older versions of the Spectrum Scale code had a third criteria in checking for 'bad state', which included checking if unacked was greater than 8, but that check would sometimes call-out a socket that was
working fine, so this third check has been removed via the APAR IJ02566. All Spectrum Scale V5 code has this fix and the 4.2.X code stream picked up this fix in PTF 7 (4.2.3.7 ships APAR IJ02566).</font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">More details on debugging expels using these TCP connection messages are in the presentation you referred to, which I posted here:</font><a href="https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ibm.com%2Fdeveloperworks%2Fcommunity%2Fwikis%2Fhome%3Flang%3Den_us%23!%2Fwiki%2FGeneral%2520Parallel%2520File%2520System%2520(GPFS)%2Fpage%2FDEBUG%2520Expels&data=02%7C01%7CKevin.Buterbaugh%40Vanderbilt.Edu%7C106784daf7e9407e00ef08d624978b49%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636736630629232470&sdata=BgfLGFTA9%2FlqDjx4HMnBQMkvSj0dK0t97yNV%2FHGQpa4%3D&reserved=0" originalsrc="https://www.ibm.com/developerworks/community/wikis/home?lang=en_us#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/DEBUG%20Expels" shash="DYjom3nmXVbMNQs+Co+wF1mqJK/6ET9aEs29v7LV9RhwgfkbtVMfBjbtX+JxBnRdMMsTCcmGGDI2ejDhDnk58yS0KazhMrYPywWR48DUDd/0fOGsioU6h8x99wW0JQZEXjF658bEIT4vNAmtgMMji7SrwSRTRnfkC71hMTMGRX4=" class=""><font size="2" color="blue" face="sans-serif" class="">https://www.ibm.com/developerworks/community/wikis/home?lang=en_us#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/DEBUG%20Expels</font></a><br class="">
<br class="">
<font size="2" face="sans-serif" class="">Regards,<br class="">
John Lewars <br class="">
Technical Computing Development, IBM Poughkeepsie<br class="">
</font><br class="">
<br class="">
<font size="1" color="#800080" face="sans-serif" class="">----- Forwarded by Lyle Gayne/Poughkeepsie/IBM on 09/27/2018 11:15 AM -----</font><br class="">
<br class="">
<br class="">
<font size="3" class="">Hi All, </font><br class="">
<br class="">
<font size="3" class="">2018-09-27_09:48:50.923-0500: [E] The TCP connection to IP address 1.2.3.4 some client <c0n509> (socket 442) state is unexpected: ca_state=1 unacked=3
<b class="">rto=27008000</b></font><br class="">
<br class="">
<font size="3" class="">Seeing errors like the above and trying to track down the root cause. I know that at last weeks’ GPFS User Group meeting at ORNL this very error message was discussed, but I don’t recall the details and the slides haven’t been posted
to the website yet. IIRC, the “rto” is significant … </font><br class="">
<br class="">
<font size="3" class="">I’ve Googled, but haven’t gotten any hits, nor have I found anything in the GPFS 4.2.2 Problem Determination Guide.</font><br class="">
<br class="">
<font size="3" class="">Thanks in advance…</font><br class="">
<br class="">
<font size="3" class="">—</font><br class="">
<font size="3" class="">Kevin Buterbaugh - Senior System Administrator</font><br class="">
<font size="3" class="">Vanderbilt University - Advanced Computing Center for Research and Education</font><br class="">
<a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" class=""><font size="3" color="blue" class=""><u class="">Kevin.Buterbaugh@vanderbilt.edu</u></font></a><font size="3" class="">- (615)875-9633</font><br class="">
<br class="">
<br class="">
<tt class=""><font size="2" class="">_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a><br class="">
</font></tt><a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40Vanderbilt.Edu%7C106784daf7e9407e00ef08d624978b49%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636736630629232470&sdata=O9Ki0Wnpc4kAZBKhf9crcU%2BruP8mqgnX%2FUtPXkZDDQk%3D&reserved=0" originalsrc="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" shash="ltwTitiLKsdmbI7lFeEzCZ+5TuUkmOb5UKMMfI0LlNwT4OLl1Gv2q8MAKjoJWuPRLhow0rv1lXFm8r+tJnW/eZPYGg+S7dbx2NFO5gTxhufqgjw1RuQIv0r4EU2e+iEC0VE2bBO9Tuw0eTPQUCl9ffkSDOmvP8OCTv2ZlBe/xMU=" class=""><tt class=""><font size="2" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><br class="">
<br class="">
<br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>