<html><body><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; color: #000000"><div>Dear,</div><div><span style="font-size: 12pt;"> Since a couple of hours we are seen lots off IB error at GPFS logs, on every IB node (gpfs version is </span>5.0.4-3<span style="font-size: 12pt;">):</span></div><div><br data-mce-bogus="1"></div><div> 2021-07-09_13:11:40.600+0200: [E] VERBS RDMA closed connection to 10.10.152.73 (node157) on mlx5_0 port 1 fabnum 0 index 251 cookie 648 RDMA write error IBV_WC_RETRY_EXC_ERR</div><div>2021-07-09_13:11:40.600+0200: [E] VERBS RDMA closed connection to 10.10.152.18 (node102) on mlx5_0 port 1 fabnum 0 index 227 cookie 687 RDMA write error IBV_WC_RETRY_EXC_ERR</div><div>2021-07-09_13:11:40.600+0200: [E] VERBS RDMA closed connection to 10.10.152.17 (node101) on mlx5_0 port 1 fabnum 0 index 298 cookie 693 RDMA write error IBV_WC_RETRY_EXC_ERR</div><div>2021-07-09_13:11:40.600+0200: [E] VERBS RDMA closed connection to 10.10.151.6 (node6) on mlx5_0 port 1 fabnum 0 index 18 cookie 696 RDMA write error IBV_WC_RETRY_EXC_ERR</div><div>2021-07-09_13:11:40.601+0200: [E] VERBS RDMA closed connection to 10.10.152.46 (node130) on mlx5_0 port 1 fabnum 0 index 254 cookie 680 RDMA write error IBV_WC_RETRY_EXC_ERR</div><div>2021-07-09_13:11:40.601+0200: [E] VERBS RDMA closed connection to 10.10.151.81 (node81) on mlx5_0 port 1 fabnum 0 index 289 cookie 679 RDMA read error IBV_WC_RETRY_EXC_ERR</div><div><br></div><div>and ofcourse long waiters:</div><div><br data-mce-bogus="1"></div><div><div>=== mmdiag: waiters ===</div><div>Waiting 34.8493 sec since 13:11:35, ignored, thread 2935 VerbsReconnectThread: delaying for 25.150686000 more seconds, reason: delaying for next reconnect attempt</div><div>Waiting 34.6249 sec since 13:11:35, ignored, thread 10198 VerbsReconnectThread: delaying for 25.375072000 more seconds, reason: delaying for next reconnect attempt</div><div>Waiting 27.0957 sec since 13:11:43, ignored, thread 10052 VerbsReconnectThread: delaying for 32.904264000 more seconds, reason: delaying for next reconnect attempt</div><div>Waiting 14.8909 sec since 13:11:55, monitored, thread 23135 NSDThread: for RDMA write completion fast on node 10.10.151.65 <c0n258></div><div>Waiting 14.8891 sec since 13:11:55, monitored, thread 23109 NSDThread: for RDMA write completion fast on node 10.10.152.32 <c0n247></div><div>Waiting 14.8865 sec since 13:11:55, monitored, thread 23302 NSDThread: for RDMA write completion fast on node 10.10.150.1 <c0n29></div></div><div><br></div><div><div>[common]</div><div>verbsRdma enable</div><div>verbsPorts mlx4_0/1/0</div><div>[gpfs02,gpfs04,gpfs05,gpfs06,gpfs07,gpfs08]</div><div>verbsPorts mlx5_0/1/0</div><div>[gpfs01]</div><div>verbsPorts mlx5_1/1/0</div><div>[gpfs03]</div><div>verbsPorts mlx5_0/1/0 mlx5_1/1/0</div></div><div><br></div><div><br data-mce-bogus="1"></div><div><div>[common]</div><div>verbsRdma enable</div><div>verbsPorts mlx4_0/1/0</div><div>[gpfs02,gpfs04,gpfs05,gpfs06,gpfs07,gpfs08,wngpu001,wngpu002,wngpu003,wngpu004,wngpu005]</div><div>verbsPorts mlx5_0/1/0</div><div>[gpfs01]</div><div>verbsPorts mlx5_1/1/0</div><div>[gpfs03]</div><div>verbsPorts mlx5_0/1/0 mlx5_1/1/0</div></div><div><br></div><div>Any advise is welcomed</div><div>regards, I</div><div><br data-mce-bogus="1"></div></div>
<br><br></body></html>