<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" >smells like a network problem ..</div>
<div dir="ltr" > </div>
<div dir="ltr" >IBV_WC_RETRY_EXC_ERR comes from OFED and clearly says that the data didn't get through successfully,</div>
<div dir="ltr" > </div>
<div dir="ltr" >further help .. check</div>
<div dir="ltr" >ibstat</div>
<div dir="ltr" >iblinkinfo</div>
<div dir="ltr" >ibdiagnet</div>
<div dir="ltr" >and the sminfo .. (should be the same on all members)</div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" data-history-expanded="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Ursprüngliche Nachricht -----<br>Von: "Iban Cabrillo" <cabrillo@ifca.unican.es><br>Gesendet von: gpfsug-discuss-bounces@spectrumscale.org<br>An: "gpfsug-discuss" <gpfsug-discuss@spectrumscale.org><br>CC:<br>Betreff: [EXTERNAL] [gpfsug-discuss] RDMA write error IBV_WC_RETRY_EXC_ERR<br>Datum: Fr, 9. Jul 2021 13:29<br>
<div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; color: #000000" ><div>Dear,</div>
<div><span style="font-size: 12pt;" > Since a couple of hours we are seen lots off IB error at GPFS logs, on every IB node (gpfs version is </span>5.0.4-3<span style="font-size: 12pt;" >):</span></div>
<div> </div>
<div> 2021-07-09_13:11:40.600+0200: [E] VERBS RDMA closed connection to 10.10.152.73 (node157) on mlx5_0 port 1 fabnum 0 index 251 cookie 648 RDMA write error IBV_WC_RETRY_EXC_ERR</div>
<div>2021-07-09_13:11:40.600+0200: [E] VERBS RDMA closed connection to 10.10.152.18 (node102) on mlx5_0 port 1 fabnum 0 index 227 cookie 687 RDMA write error IBV_WC_RETRY_EXC_ERR</div>
<div>2021-07-09_13:11:40.600+0200: [E] VERBS RDMA closed connection to 10.10.152.17 (node101) on mlx5_0 port 1 fabnum 0 index 298 cookie 693 RDMA write error IBV_WC_RETRY_EXC_ERR</div>
<div>2021-07-09_13:11:40.600+0200: [E] VERBS RDMA closed connection to 10.10.151.6 (node6) on mlx5_0 port 1 fabnum 0 index 18 cookie 696 RDMA write error IBV_WC_RETRY_EXC_ERR</div>
<div>2021-07-09_13:11:40.601+0200: [E] VERBS RDMA closed connection to 10.10.152.46 (node130) on mlx5_0 port 1 fabnum 0 index 254 cookie 680 RDMA write error IBV_WC_RETRY_EXC_ERR</div>
<div>2021-07-09_13:11:40.601+0200: [E] VERBS RDMA closed connection to 10.10.151.81 (node81) on mlx5_0 port 1 fabnum 0 index 289 cookie 679 RDMA read error IBV_WC_RETRY_EXC_ERR</div>
<div> </div>
<div>and ofcourse long waiters:</div>
<div> </div>
<div><div>=== mmdiag: waiters ===</div>
<div>Waiting 34.8493 sec since 13:11:35, ignored, thread 2935 VerbsReconnectThread: delaying for 25.150686000 more seconds, reason: delaying for next reconnect attempt</div>
<div>Waiting 34.6249 sec since 13:11:35, ignored, thread 10198 VerbsReconnectThread: delaying for 25.375072000 more seconds, reason: delaying for next reconnect attempt</div>
<div>Waiting 27.0957 sec since 13:11:43, ignored, thread 10052 VerbsReconnectThread: delaying for 32.904264000 more seconds, reason: delaying for next reconnect attempt</div>
<div>Waiting 14.8909 sec since 13:11:55, monitored, thread 23135 NSDThread: for RDMA write completion fast on node 10.10.151.65 <c0n258></div>
<div>Waiting 14.8891 sec since 13:11:55, monitored, thread 23109 NSDThread: for RDMA write completion fast on node 10.10.152.32 <c0n247></div>
<div>Waiting 14.8865 sec since 13:11:55, monitored, thread 23302 NSDThread: for RDMA write completion fast on node 10.10.150.1 <c0n29></div></div>
<div> </div>
<div><div>[common]</div>
<div>verbsRdma enable</div>
<div>verbsPorts mlx4_0/1/0</div>
<div>[gpfs02,gpfs04,gpfs05,gpfs06,gpfs07,gpfs08]</div>
<div>verbsPorts mlx5_0/1/0</div>
<div>[gpfs01]</div>
<div>verbsPorts mlx5_1/1/0</div>
<div>[gpfs03]</div>
<div>verbsPorts mlx5_0/1/0 mlx5_1/1/0</div></div>
<div> </div>
<div> </div>
<div><div>[common]</div>
<div>verbsRdma enable</div>
<div>verbsPorts mlx4_0/1/0</div>
<div>[gpfs02,gpfs04,gpfs05,gpfs06,gpfs07,gpfs08,wngpu001,wngpu002,wngpu003,wngpu004,wngpu005]</div>
<div>verbsPorts mlx5_0/1/0</div>
<div>[gpfs01]</div>
<div>verbsPorts mlx5_1/1/0</div>
<div>[gpfs03]</div>
<div>verbsPorts mlx5_0/1/0 mlx5_1/1/0</div></div>
<div> </div>
<div>Any advise is welcomed</div>
<div>regards, I</div>
<div> </div></div>
<div><font size="2" face="Default Monospace,Courier New,Courier,monospace" >_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> </font></div></blockquote>
<div dir="ltr" > </div></div><BR>
<BR>