<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
</head>
<body>
<!-- signature -->
<div style="color:rgb(31, 73, 125);">Damir,</div>
<div style="color:rgb(31, 73, 125);"><br>
</div>
<div style="color:rgb(31, 73, 125);">We see similar things in our environment (3.5k nodes) that seem to correlate with GPFS recovery events. I did some digging at it seemed to me that these errors more or less mean the other side of the VERBS connection hung
up on the other. The message format seems a little alarming but I think it's innocuous. I'm curious to hear what others have to say. </div>
<div style="color:rgb(31, 73, 125);"><br>
</div>
<div style="color:rgb(31, 73, 125);">-Aaron<br>
<br>
</div>
<font style="color:rgb(31, 73, 125)"></font><!-- quoted content --><br>
<br>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""> Damir Krstic<br>
<b>Sent:</b> 6/26/16, 11:23 AM<br>
<b>To:</b> gpfsug main discussion list<br>
<b>Subject:</b> [gpfsug-discuss] verbs rdma errors in logs<o:p></o:p></span></p>
</div>
</div>
<div dir="ltr">We recently enabled verbs/rdma on our IB network (previously we used IPoIB exclusively) and now are getting all sorts of errors/warning in logs:
<div><br>
</div>
<div><span style="color:rgb(33,33,33); font-family:"segoe ui","segoe wp","segoe ui wpc",tahoma,arial,sans-serif; font-size:13.3333px; line-height:normal">Jun 25 23:41:30 gssio2 mmfs: [E] VERBS RDMA rdma read error IBV_WC_RETRY_EXC_ERR to 172.41.125.27 (qnode4111-ib0.quest)
on mlx5_0 port 1 fabnum 0 vendor_err 129</span><br style="color:rgb(33,33,33); font-family:"segoe ui","segoe wp","segoe ui wpc",tahoma,arial,sans-serif; font-size:13.3333px; line-height:normal">
<span style="color:rgb(33,33,33); font-family:"segoe ui","segoe wp","segoe ui wpc",tahoma,arial,sans-serif; font-size:13.3333px; line-height:normal">Jun 25 23:41:30 gssio2 mmfs: [E] VERBS RDMA closed connection to 172.41.125.27 (qnode4111-ib0.quest) on mlx5_0
port 1 fabnum 0 due to RDMA read error IBV_WC_RETRY_EXC_ERR index 1589</span><br>
</div>
<div><span style="color:rgb(33,33,33); font-family:"segoe ui","segoe wp","segoe ui wpc",tahoma,arial,sans-serif; font-size:13.3333px; line-height:normal"><br>
</span></div>
<div><span style="color:rgb(33,33,33); font-family:"segoe ui","segoe wp","segoe ui wpc",tahoma,arial,sans-serif; font-size:13.3333px; line-height:normal">Jun 25 20:40:05 gssio2 mmfs: [N] VERBS RDMA closed connection to 172.41.124.12 (qnode4054-ib0.quest) on
mlx5_0 port 1 fabnum 0 index 1417</span><br style="color:rgb(33,33,33); font-family:"segoe ui","segoe wp","segoe ui wpc",tahoma,arial,sans-serif; font-size:13.3333px; line-height:normal">
</div>
<div><span style="color:rgb(33,33,33); font-family:"segoe ui","segoe wp","segoe ui wpc",tahoma,arial,sans-serif; font-size:13.3333px; line-height:normal"><br>
</span></div>
<div><span style="color:rgb(33,33,33); font-family:"segoe ui","segoe wp","segoe ui wpc",tahoma,arial,sans-serif; font-size:13.3333px; line-height:normal">Jun 25 qnode6131-ib0.quest.it.northwestern.edu) on mlx5_0 port 1 fabnum 0 index 195</span><br style="color:rgb(33,33,33); font-family:"segoe ui","segoe wp","segoe ui wpc",tahoma,arial,sans-serif; font-size:13.3333px; line-height:normal">
</div>
<div><span style="color:rgb(33,33,33); font-family:"segoe ui","segoe wp","segoe ui wpc",tahoma,arial,sans-serif; font-size:13.3333px; line-height:normal"><br>
</span></div>
<div><span style="color:rgb(33,33,33); font-family:"segoe ui","segoe wp","segoe ui wpc",tahoma,arial,sans-serif; font-size:13.3333px; line-height:normal">Jun 25 qnode6131-ib0.quest.it.northwestern.edu) on mlx5_0 port 1 fabnum 0 index 1044</span><br style="color:rgb(33,33,33); font-family:"segoe ui","segoe wp","segoe ui wpc",tahoma,arial,sans-serif; font-size:13.3333px; line-height:normal">
</div>
<div><span style="color:rgb(33,33,33); font-family:"segoe ui","segoe wp","segoe ui wpc",tahoma,arial,sans-serif; font-size:13.3333px; line-height:normal"><br>
</span></div>
<div><span style="color:rgb(33,33,33); font-family:"segoe ui","segoe wp","segoe ui wpc",tahoma,arial,sans-serif; font-size:13.3333px; line-height:normal">Something to note (not sure if this is important or not)</span><span style="color:rgb(33,33,33); font-family:"segoe ui","segoe wp","segoe ui wpc",tahoma,arial,sans-serif; line-height:normal"> is
that our ESS storage cluster and our login nodes are in connected mode with 64K MTU and all compute nodes are in datagram mode with 2.4K MTU.</span></div>
<div><span style="color:rgb(33,33,33); font-family:"segoe ui","segoe wp","segoe ui wpc",tahoma,arial,sans-serif; line-height:normal"><br>
</span></div>
<div><font color="#212121" face="segoe ui, segoe wp, segoe ui wpc, tahoma, arial, sans-serif"><span style="line-height:normal">Are these messages something to be concerned about? Cluster seems to be performing well and although there are some node ejections,
they do not seem higher than before we turned on verbs/rdma.</span></font></div>
<div><font color="#212121" face="segoe ui, segoe wp, segoe ui wpc, tahoma, arial, sans-serif"><span style="line-height:normal"><br>
</span></font></div>
<div><font color="#212121" face="segoe ui, segoe wp, segoe ui wpc, tahoma, arial, sans-serif"><span style="line-height:normal">Thanks,</span></font></div>
<div><font color="#212121" face="segoe ui, segoe wp, segoe ui wpc, tahoma, arial, sans-serif"><span style="line-height:normal">Damir</span></font></div>
</div>
</body>
</html>