<div dir="ltr">My first guess would also be rdmaSend, which the gssClientConfig.sh enables by default, but isn't scalable to large clusters. It fits with your error message:<br><div><br><a href="https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20%28GPFS%29/page/Best%20Practices%20RDMA%20Tuning">https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20%28GPFS%29/page/Best%20Practices%20RDMA%20Tuning</a><br><ul><li>"""For GPFS version 3.5.0.11 and later, IB error <span class="gmail-st">IBV_WC_RNR_RETRY_EXC_ERR
may occur if the cluster is too large when verbsRdmaSend is enabled
Idf these errors are observed in the mmfs log, disable verbsRdmaSend on
all nodes..</span> Additionally, out of memory errors may occur if
verbsRdmaSend is enabled on very large clusters. If out of memory
errors are observed, disabled verbsRdmaSend on all nodes in the cluster."""</li></ul><br></div><div>Otherwise it would be nice if you could post your mmlsconfig to see if something else sticks out..<br><br><br></div><div> -jf<br></div><div><br><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jan 11, 2017 at 4:03 PM, Olaf Weiser <span dir="ltr"><<a href="mailto:olaf.weiser@de.ibm.com" target="_blank">olaf.weiser@de.ibm.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><font face="sans-serif" size="2">most likely, there's smth wrong with your
IB fabric ... </font><br><font face="sans-serif" size="2">you say, you run ~ 700 nodes ? ...</font><br><font face="sans-serif" size="2">Are you running with </font><font size="3"><b>verbsRdmaSend</b></font><font face="sans-serif" size="2">enabled ? ,if so, please consider to disable - and discuss this within
the PMR </font><br><font face="sans-serif" size="2">another issue, you may check is -
Are you running the IPoIB in connected mode or datagram ... but as I said,
please discuss this within the PMR .. there are to much dependencies to
discuss this here .. </font><br><br><br><font face="sans-serif" size="2">cheers</font><br><br><br><div><font face="sans-serif" size="2">Mit freundlichen Grüßen / Kind regards</font><br><br><font face="sans-serif" size="2"> <br>Olaf Weiser<br> <br>EMEA Storage Competence Center Mainz, German / IBM Systems, Storage Platform,<br>------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>-------------------<br>IBM Deutschland<br>IBM Allee 1<br>71139 Ehningen<br>Phone: <a href="tel:+49%20170%205794466" value="+491705794466" target="_blank">+49-170-579-44-66</a><br>E-Mail: <a href="mailto:olaf.weiser@de.ibm.com" target="_blank">olaf.weiser@de.ibm.com</a><br>------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>-------------------<br>IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter<br>Geschäftsführung: Martina Koederitz (Vorsitzende), Susanne Peter, Norbert
Janzen, Dr. Christian Keller, Ivo Koerner, Markus Koerner<br>Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart,
HRB 14562 / WEEE-Reg.-Nr. DE 99369940 </font><br><br><br><br><font color="#5f5f5f" face="sans-serif" size="1">From:
</font><font face="sans-serif" size="1">Damir Krstic <<a href="mailto:damir.krstic@gmail.com" target="_blank">damir.krstic@gmail.com</a>></font><br><font color="#5f5f5f" face="sans-serif" size="1">To:
</font><font face="sans-serif" size="1">gpfsug main discussion
list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.<wbr>org</a>></font><br><font color="#5f5f5f" face="sans-serif" size="1">Date:
</font><font face="sans-serif" size="1">01/11/2017 03:39 PM</font><br><font color="#5f5f5f" face="sans-serif" size="1">Subject:
</font><font face="sans-serif" size="1">[gpfsug-discuss]
nodes being ejected out of the cluster</font><br><font color="#5f5f5f" face="sans-serif" size="1">Sent by:
</font><font face="sans-serif" size="1"><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@<wbr>spectrumscale.org</a></font><br><hr noshade><div><div class="h5"><br><br><br><font size="3">We are running GPFS 4.2 on our cluster (around 700 compute
nodes). Our storage (ESS GL6) is also running GPFS 4.2. Compute nodes and
storage are connected via Infiniband (FDR14). At the time of implementation
of ESS, we were instructed to enable RDMA in addition to IPoIB. Previously
we only ran IPoIB on our GPFS3.5 cluster.</font><br><br><font size="3">Every since the implementation (sometime back in July
of 2016) we see a lot of compute nodes being ejected. What usually precedes
the ejection are following messages:</font><br><br><font size="3">Jan 11 02:03:15 quser13 mmfs: [E] VERBS RDMA rdma send
error IBV_WC_RNR_RETRY_EXC_ERR to 172.41.2.5 (gssio2-fdr) on mlx4_0 port
1 fabnum 0 vendor_err 135 </font><br><font size="3">Jan 11 02:03:15 quser13 mmfs: [E] VERBS RDMA closed connection
to 172.41.2.5 (gssio2-fdr) on mlx4_0 port 1 fabnum 0 due to send error
IBV_WC_RNR_RETRY_EXC_ERR index 2</font><br><font size="3">Jan 11 02:03:26 quser13 mmfs: [E] VERBS RDMA rdma send
error IBV_WC_RNR_RETRY_EXC_ERR to 172.41.2.5 (gssio2-fdr) on mlx4_0 port
1 fabnum 0 vendor_err 135 </font><br><font size="3">Jan 11 02:03:26 quser13 mmfs: [E] VERBS RDMA closed connection
to 172.41.2.5 (gssio2-fdr) on mlx4_0 port 1 fabnum 0 due to send error
IBV_WC_WR_FLUSH_ERR index 1</font><br><font size="3">Jan 11 02:03:26 quser13 mmfs: [E] VERBS RDMA rdma send
error IBV_WC_RNR_RETRY_EXC_ERR to 172.41.2.5 (gssio2-fdr) on mlx4_0 port
1 fabnum 0 vendor_err 135 </font><br><font size="3">Jan 11 02:03:26 quser13 mmfs: [E] VERBS RDMA closed connection
to 172.41.2.5 (gssio2-fdr) on mlx4_0 port 1 fabnum 0 due to send error
IBV_WC_RNR_RETRY_EXC_ERR index 2</font><br><font size="3">Jan 11 02:06:38 quser11 mmfs: [E] VERBS RDMA rdma send
error IBV_WC_RNR_RETRY_EXC_ERR to 172.41.2.5 (gssio2-fdr) on mlx4_0 port
1 fabnum 0 vendor_err 135 </font><br><font size="3">Jan 11 02:06:38 quser11 mmfs: [E] VERBS RDMA closed connection
to 172.41.2.5 (gssio2-fdr) on mlx4_0 port 1 fabnum 0 due to send error
IBV_WC_WR_FLUSH_ERR index 400</font><br><br><font size="3">Even our ESS IO server sometimes ends up being ejected
(case in point - yesterday morning):</font><br><br><font size="3">Jan 10 11:23:42 gssio2 mmfs: [E] VERBS RDMA rdma send
error IBV_WC_RNR_RETRY_EXC_ERR to 172.41.2.1 (gssio1-fdr) on mlx5_1 port
1 fabnum 0 vendor_err 135</font><br><font size="3">Jan 10 11:23:42 gssio2 mmfs: [E] VERBS RDMA closed connection
to 172.41.2.1 (gssio1-fdr) on mlx5_1 port 1 fabnum 0 due to send error
IBV_WC_RNR_RETRY_EXC_ERR index 3001</font><br><font size="3">Jan 10 11:23:43 gssio2 mmfs: [E] VERBS RDMA rdma send
error IBV_WC_RNR_RETRY_EXC_ERR to 172.41.2.1 (gssio1-fdr) on mlx5_1 port
2 fabnum 0 vendor_err 135</font><br><font size="3">Jan 10 11:23:43 gssio2 mmfs: [E] VERBS RDMA closed connection
to 172.41.2.1 (gssio1-fdr) on mlx5_1 port 2 fabnum 0 due to send error
IBV_WC_RNR_RETRY_EXC_ERR index 2671</font><br><font size="3">Jan 10 11:23:43 gssio2 mmfs: [E] VERBS RDMA rdma send
error IBV_WC_RNR_RETRY_EXC_ERR to 172.41.2.1 (gssio1-fdr) on mlx5_0 port
2 fabnum 0 vendor_err 135</font><br><font size="3">Jan 10 11:23:43 gssio2 mmfs: [E] VERBS RDMA closed connection
to 172.41.2.1 (gssio1-fdr) on mlx5_0 port 2 fabnum 0 due to send error
IBV_WC_RNR_RETRY_EXC_ERR index 2495</font><br><font size="3">Jan 10 11:23:44 gssio2 mmfs: [E] VERBS RDMA rdma send
error IBV_WC_RNR_RETRY_EXC_ERR to 172.41.2.1 (gssio1-fdr) on mlx5_0 port
1 fabnum 0 vendor_err 135</font><br><font size="3">Jan 10 11:23:44 gssio2 mmfs: [E] VERBS RDMA closed connection
to 172.41.2.1 (gssio1-fdr) on mlx5_0 port 1 fabnum 0 due to send error
IBV_WC_RNR_RETRY_EXC_ERR index 3077</font><br><font size="3">Jan 10 11:24:11 gssio2 mmfs: [N] Node 172.41.2.1 (gssio1-fdr)
lease renewal is overdue. Pinging to check if it is alive</font><br><br><font size="3">I've had multiple PMRs open for this issue, and I am told
that our ESS needs code level upgrades in order to fix this issue. Looking
at the errors, I think the issue is Infiniband related, and I am wondering
if anyone on this list has seen similar issues?</font><br><br><font size="3">Thanks for your help in advance.</font><br><br></div></div><font size="3">Damir</font><tt><font size="2">_________________________<wbr>______________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><tt><font size="2">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</font></tt></a><tt><font size="2"><br></font></tt><br><br></div><br>
<br>______________________________<wbr>_________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</a><br>
<br></blockquote></div><br></div>