<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
</head>
<body>
<!-- signature -->
<div style="color:rgb(31, 73, 125);">The RDMA errors I think are secondary to what's going on with either your IPoIB or Ethernet fabrics that's causing I assume IPoIB communication breakdowns and expulsions. We've had entire IB fabrics go offline and if the
nodes werent depending on it for daemon communication nobody got expelled. Do you have a subnet defined for your IPoIB network or are your nodes daemon interfaces already set to their IPoIB interface? Have you checked your SM logs?<br>
<br>
</div>
<font style="color:rgb(31, 73, 125)"></font><!-- quoted content --><br>
<br>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""> Damir Krstic<br>
<b>Sent:</b> 1/11/17, 9:39 AM<br>
<b>To:</b> gpfsug main discussion list<br>
<b>Subject:</b> [gpfsug-discuss] nodes being ejected out of the cluster<o:p></o:p></span></p>
</div>
</div>
<div dir="ltr">We are running GPFS 4.2 on our cluster (around 700 compute nodes). Our storage (ESS GL6) is also running GPFS 4.2. Compute nodes and storage are connected via Infiniband (FDR14). At the time of implementation of ESS, we were instructed to enable
RDMA in addition to IPoIB. Previously we only ran IPoIB on our GPFS3.5 cluster.
<div><br>
</div>
<div>Every since the implementation (sometime back in July of 2016) we see a lot of compute nodes being ejected. What usually precedes the ejection are following messages:</div>
<div><br>
</div>
<div>
<div style="font-family:-webkit-standard">Jan 11 02:03:15 quser13 mmfs: [E] VERBS RDMA rdma send error IBV_WC_RNR_RETRY_EXC_ERR to 172.41.2.5 (gssio2-fdr) on mlx4_0 port 1 fabnum 0 vendor_err 135 </div>
<div style="font-family:-webkit-standard">Jan 11 02:03:15 quser13 mmfs: [E] VERBS RDMA closed connection to 172.41.2.5 (gssio2-fdr) on mlx4_0 port 1 fabnum 0 due to send error IBV_WC_RNR_RETRY_EXC_ERR index 2</div>
<div style="font-family:-webkit-standard">Jan 11 02:03:26 quser13 mmfs: [E] VERBS RDMA rdma send error IBV_WC_RNR_RETRY_EXC_ERR to 172.41.2.5 (gssio2-fdr) on mlx4_0 port 1 fabnum 0 vendor_err 135 </div>
<div style="font-family:-webkit-standard">Jan 11 02:03:26 quser13 mmfs: [E] VERBS RDMA closed connection to 172.41.2.5 (gssio2-fdr) on mlx4_0 port 1 fabnum 0 due to send error IBV_WC_WR_FLUSH_ERR index 1</div>
<div style="font-family:-webkit-standard">Jan 11 02:03:26 quser13 mmfs: [E] VERBS RDMA rdma send error IBV_WC_RNR_RETRY_EXC_ERR to 172.41.2.5 (gssio2-fdr) on mlx4_0 port 1 fabnum 0 vendor_err 135 </div>
<div style="font-family:-webkit-standard">Jan 11 02:03:26 quser13 mmfs: [E] VERBS RDMA closed connection to 172.41.2.5 (gssio2-fdr) on mlx4_0 port 1 fabnum 0 due to send error IBV_WC_RNR_RETRY_EXC_ERR index 2</div>
<div style="font-family:-webkit-standard">Jan 11 02:06:38 quser11 mmfs: [E] VERBS RDMA rdma send error IBV_WC_RNR_RETRY_EXC_ERR to 172.41.2.5 (gssio2-fdr) on mlx4_0 port 1 fabnum 0 vendor_err 135 </div>
<div style="font-family:-webkit-standard">Jan 11 02:06:38 quser11 mmfs: [E] VERBS RDMA closed connection to 172.41.2.5 (gssio2-fdr) on mlx4_0 port 1 fabnum 0 due to send error IBV_WC_WR_FLUSH_ERR index 400</div>
<div><br>
</div>
</div>
<div>Even our ESS IO server sometimes ends up being ejected (case in point - yesterday morning):<br>
</div>
<div><br>
</div>
<div>
<div style="font-family:-webkit-standard">Jan 10 11:23:42 gssio2 mmfs: [E] VERBS RDMA rdma send error IBV_WC_RNR_RETRY_EXC_ERR to 172.41.2.1 (gssio1-fdr) on mlx5_1 port 1 fabnum 0 vendor_err 135</div>
<div style="font-family:-webkit-standard">Jan 10 11:23:42 gssio2 mmfs: [E] VERBS RDMA closed connection to 172.41.2.1 (gssio1-fdr) on mlx5_1 port 1 fabnum 0 due to send error IBV_WC_RNR_RETRY_EXC_ERR index 3001</div>
<div style="font-family:-webkit-standard">Jan 10 11:23:43 gssio2 mmfs: [E] VERBS RDMA rdma send error IBV_WC_RNR_RETRY_EXC_ERR to 172.41.2.1 (gssio1-fdr) on mlx5_1 port 2 fabnum 0 vendor_err 135</div>
<div style="font-family:-webkit-standard">Jan 10 11:23:43 gssio2 mmfs: [E] VERBS RDMA closed connection to 172.41.2.1 (gssio1-fdr) on mlx5_1 port 2 fabnum 0 due to send error IBV_WC_RNR_RETRY_EXC_ERR index 2671</div>
<div style="font-family:-webkit-standard">Jan 10 11:23:43 gssio2 mmfs: [E] VERBS RDMA rdma send error IBV_WC_RNR_RETRY_EXC_ERR to 172.41.2.1 (gssio1-fdr) on mlx5_0 port 2 fabnum 0 vendor_err 135</div>
<div style="font-family:-webkit-standard">Jan 10 11:23:43 gssio2 mmfs: [E] VERBS RDMA closed connection to 172.41.2.1 (gssio1-fdr) on mlx5_0 port 2 fabnum 0 due to send error IBV_WC_RNR_RETRY_EXC_ERR index 2495</div>
<div style="font-family:-webkit-standard">Jan 10 11:23:44 gssio2 mmfs: [E] VERBS RDMA rdma send error IBV_WC_RNR_RETRY_EXC_ERR to 172.41.2.1 (gssio1-fdr) on mlx5_0 port 1 fabnum 0 vendor_err 135</div>
<div style="font-family:-webkit-standard">Jan 10 11:23:44 gssio2 mmfs: [E] VERBS RDMA closed connection to 172.41.2.1 (gssio1-fdr) on mlx5_0 port 1 fabnum 0 due to send error IBV_WC_RNR_RETRY_EXC_ERR index 3077</div>
<div style="font-family:-webkit-standard">Jan 10 11:24:11 gssio2 mmfs: [N] Node 172.41.2.1 (gssio1-fdr) lease renewal is overdue. Pinging to check if it is alive</div>
</div>
<div style="font-family:-webkit-standard"><br>
</div>
<div style="font-family:-webkit-standard"><span style="font-family:sans-serif">I've had multiple PMRs open for this issue, and I am told that our ESS needs code level upgrades in order to fix this issue. Looking at the errors, I think the issue is Infiniband
related, and I am wondering if anyone on this list has seen similar issues?</span><br>
</div>
<div style="font-family:-webkit-standard"><span style="font-family:sans-serif"><br>
</span></div>
<div style="font-family:-webkit-standard"><span style="font-family:sans-serif">Thanks for your help in advance.</span></div>
<div style="font-family:-webkit-standard"><span style="font-family:sans-serif"><br>
</span></div>
<div style="font-family:-webkit-standard"><span style="font-family:sans-serif">Damir</span></div>
</div>
</body>
</html>