<div dir="ltr">Since I am testing out remote mounting with EDR IB routers, I'll add to the discussion.<div><br></div><div>In my lab environment I was seeing the same rdma connections being established and then disconnected shortly after. The remote filesystem would eventually mount on the clients, but it look a quite a while (~2mins). Even after mounting, accessing files or any metadata operations would take a while to execute, but eventually it happened.</div><div><br></div><div>After enabling verbsRdmaCm, everything mounted just fine and in a timely manner. Spectrum Scale was using the librdmacm.so library.</div><div><br></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">I would first double check that you have both clusters able to talk to each other on their IPoIB address, then make sure you enable verbsRdmaCm on both clusters.</span></div><div><br></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><span style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px">---------------------------------------------------------------------------------------------------------------</span><br></div><div dir="ltr"><span style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px">Zach Mance <a href="mailto:zmance@ucar.edu" target="_blank">zmance@ucar.edu</a></span><span style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px"><span> </span> (303) 497-1883 <span> </span></span><br style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px"><span style="font-size:12.8px">HPC Data Infrastructure Group</span><span style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px"> / CISL / NCAR</span><br style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px"><span style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px">--------------------------------------------------------------------------------------------------------------- </span><br></div></div></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Thu, Mar 1, 2018 at 1:41 AM, John Hearns <span dir="ltr"><<a href="mailto:john.hearns@asml.com" target="_blank">john.hearns@asml.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">In reply to Stuart,<br>
our setup is entirely Infiniband. We boot and install over IB, and rely heavily on IP over Infiniband.<br>
<br>
As for users being 'confused' due to multiple IPs, I would appreciate some more depth on that one.<br>
Sure, all batch systems are sensitive to hostnames (as I know to my cost!) but once you get that straightened out why should users care?<br>
I am not being aggressive, just keen to find out more.<br>
<span class=""><br>
<br>
<br>
-----Original Message-----<br>
From: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">gpfsug-discuss-bounces@<wbr>spectrumscale.org</a> [mailto:<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">gpfsug-discuss-<wbr>bounces@spectrumscale.org</a>] On Behalf Of Stuart Barkley<br>
Sent: Wednesday, February 28, 2018 6:50 PM<br>
To: gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.<wbr>org</a>><br>
</span><div><div class="h5">Subject: Re: [gpfsug-discuss] Problems with remote mount via routed IB<br>
<br>
The problem with CM is that it seems to require configuring IP over Infiniband.<br>
<br>
I'm rather strongly opposed to IP over IB. We did run IPoIB years ago, but pulled it out of our environment as adding unneeded complexity. It requires provisioning IP addresses across the Infiniband infrastructure and possibly adding routers to other portions of the IP infrastructure. It was also confusing some users due to multiple IPs on the compute infrastructure.<br>
<br>
We have recently been in discussions with a vendor about their support for GPFS over IB and they kept directing us to using CM (which still didn't work). CM wasn't necessary once we found out about the actual problem (we needed the undocumented verbsRdmaUseGidIndexZero configuration option among other things due to their use of SR-IOV based virtual IB interfaces).<br>
<br>
We don't use routed Infiniband and it might be that CM and IPoIB is required for IB routing, but I doubt it. It sounds like the OP is keeping IB and IP infrastructure separate.<br>
<br>
Stuart Barkley<br>
<br>
On Mon, 26 Feb 2018 at 14:16 -0000, Aaron Knister wrote:<br>
<br>
> Date: Mon, 26 Feb 2018 14:16:34<br>
> From: Aaron Knister <<a href="mailto:aaron.s.knister@nasa.gov">aaron.s.knister@nasa.gov</a>><br>
> Reply-To: gpfsug main discussion list<br>
> <<a href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.<wbr>org</a>><br>
> To: <a href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.<wbr>org</a><br>
> Subject: Re: [gpfsug-discuss] Problems with remote mount via routed IB<br>
><br>
> Hi Jan Erik,<br>
><br>
> It was my understanding that the IB hardware router required RDMA CM to work.<br>
> By default GPFS doesn't use the RDMA Connection Manager but it can be<br>
> enabled (e.g. verbsRdmaCm=enable). I think this requires a restart on<br>
> clients/servers (in both clusters) to take effect. Maybe someone else<br>
> on the list can comment in more detail-- I've been told folks have<br>
> successfully deployed IB routers with GPFS.<br>
><br>
> -Aaron<br>
><br>
> On 2/26/18 11:38 AM, Sundermann, Jan Erik (SCC) wrote:<br>
> ><br>
> > Dear all<br>
> ><br>
> > we are currently trying to remote mount a file system in a routed<br>
> > Infiniband test setup and face problems with dropped RDMA<br>
> > connections. The setup is the<br>
> > following:<br>
> ><br>
> > - Spectrum Scale Cluster 1 is setup on four servers which are<br>
> > connected to the same infiniband network. Additionally they are<br>
> > connected to a fast ethernet providing ip communication in the network <a href="http://192.168.11.0/24" rel="noreferrer" target="_blank">192.168.11.0/24</a>.<br>
> ><br>
> > - Spectrum Scale Cluster 2 is setup on four additional servers which<br>
> > are connected to a second infiniband network. These servers have IPs<br>
> > on their IB interfaces in the network <a href="http://192.168.12.0/24" rel="noreferrer" target="_blank">192.168.12.0/24</a>.<br>
> ><br>
> > - IP is routed between <a href="http://192.168.11.0/24" rel="noreferrer" target="_blank">192.168.11.0/24</a> and <a href="http://192.168.12.0/24" rel="noreferrer" target="_blank">192.168.12.0/24</a> on a<br>
> > dedicated machine.<br>
> ><br>
> > - We have a dedicated IB hardware router connected to both IB subnets.<br>
> ><br>
> ><br>
> > We tested that the routing, both IP and IB, is working between the<br>
> > two clusters without problems and that RDMA is working fine both for<br>
> > internal communication inside cluster 1 and cluster 2<br>
> ><br>
> > When trying to remote mount a file system from cluster 1 in cluster<br>
> > 2, RDMA communication is not working as expected. Instead we see<br>
> > error messages on the remote host (cluster 2)<br>
> ><br>
> ><br>
> > 2018-02-23_13:48:47.037+0100: [I] VERBS RDMA connecting to<br>
> > 192.168.11.4 (iccn004-gpfs in gpfsstorage.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 index 2<br>
> > 2018-02-23_13:48:49.890+0100: [I] VERBS RDMA connected to<br>
> > 192.168.11.4 (iccn004-gpfs in gpfsstorage.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 sl 0 index 2<br>
> > 2018-02-23_13:48:53.138+0100: [E] VERBS RDMA closed connection to<br>
> > 192.168.11.1 (iccn001-gpfs in gpfsstorage.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 error 733 index 3<br>
> > 2018-02-23_13:48:53.854+0100: [I] VERBS RDMA connecting to<br>
> > 192.168.11.1 (iccn001-gpfs in gpfsstorage.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 index 3<br>
> > 2018-02-23_13:48:54.954+0100: [E] VERBS RDMA closed connection to<br>
> > 192.168.11.3 (iccn003-gpfs in gpfsstorage.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 error 733 index 1<br>
> > 2018-02-23_13:48:55.601+0100: [I] VERBS RDMA connected to<br>
> > 192.168.11.1 (iccn001-gpfs in gpfsstorage.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 sl 0 index 3<br>
> > 2018-02-23_13:48:57.775+0100: [I] VERBS RDMA connecting to<br>
> > 192.168.11.3 (iccn003-gpfs in gpfsstorage.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 index 1<br>
> > 2018-02-23_13:48:59.557+0100: [I] VERBS RDMA connected to<br>
> > 192.168.11.3 (iccn003-gpfs in gpfsstorage.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 sl 0 index 1<br>
> > 2018-02-23_13:48:59.876+0100: [E] VERBS RDMA closed connection to<br>
> > 192.168.11.2 (iccn002-gpfs in gpfsstorage.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 error 733 index 0<br>
> > 2018-02-23_13:49:02.020+0100: [I] VERBS RDMA connecting to<br>
> > 192.168.11.2 (iccn002-gpfs in gpfsstorage.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 index 0<br>
> > 2018-02-23_13:49:03.477+0100: [I] VERBS RDMA connected to<br>
> > 192.168.11.2 (iccn002-gpfs in gpfsstorage.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 sl 0 index 0<br>
> > 2018-02-23_13:49:05.119+0100: [E] VERBS RDMA closed connection to<br>
> > 192.168.11.4 (iccn004-gpfs in gpfsstorage.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 error 733 index 2<br>
> > 2018-02-23_13:49:06.191+0100: [I] VERBS RDMA connecting to<br>
> > 192.168.11.4 (iccn004-gpfs in gpfsstorage.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 index 2<br>
> > 2018-02-23_13:49:06.548+0100: [I] VERBS RDMA connected to<br>
> > 192.168.11.4 (iccn004-gpfs in gpfsstorage.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 sl 0 index 2<br>
> > 2018-02-23_13:49:11.578+0100: [E] VERBS RDMA closed connection to<br>
> > 192.168.11.1 (iccn001-gpfs in gpfsstorage.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 error 733 index 3<br>
> > 2018-02-23_13:49:11.937+0100: [I] VERBS RDMA connecting to<br>
> > 192.168.11.1 (iccn001-gpfs in gpfsstorage.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 index 3<br>
> > 2018-02-23_13:49:11.939+0100: [I] VERBS RDMA connected to<br>
> > 192.168.11.1 (iccn001-gpfs in gpfsstorage.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 sl 0 index 3<br>
> ><br>
> ><br>
> > and in the cluster with the file system (cluster 1)<br>
> ><br>
> > 2018-02-23_13:47:36.112+0100: [E] VERBS RDMA rdma read error<br>
> > IBV_WC_RETRY_EXC_ERR to 192.168.12.5 (iccn005-ib in<br>
> > gpfsremoteclients.localdomain) on mlx4_0 port 1 fabnum 0 vendor_err<br>
> > 129<br>
> > 2018-02-23_13:47:36.112+0100: [E] VERBS RDMA closed connection to<br>
> > 192.168.12.5 (iccn005-ib in gpfsremoteclients.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 due to RDMA read error IBV_WC_RETRY_EXC_ERR index 3<br>
> > 2018-02-23_13:47:47.161+0100: [I] VERBS RDMA accepted and connected<br>
> > to<br>
> > 192.168.12.5 (iccn005-ib in gpfsremoteclients.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 sl 0 index 3<br>
> > 2018-02-23_13:48:04.317+0100: [E] VERBS RDMA rdma read error<br>
> > IBV_WC_RETRY_EXC_ERR to 192.168.12.5 (iccn005-ib in<br>
> > gpfsremoteclients.localdomain) on mlx4_0 port 1 fabnum 0 vendor_err<br>
> > 129<br>
> > 2018-02-23_13:48:04.317+0100: [E] VERBS RDMA closed connection to<br>
> > 192.168.12.5 (iccn005-ib in gpfsremoteclients.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 due to RDMA read error IBV_WC_RETRY_EXC_ERR index 3<br>
> > 2018-02-23_13:48:11.560+0100: [I] VERBS RDMA accepted and connected<br>
> > to<br>
> > 192.168.12.5 (iccn005-ib in gpfsremoteclients.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 sl 0 index 3<br>
> > 2018-02-23_13:48:32.523+0100: [E] VERBS RDMA rdma read error<br>
> > IBV_WC_RETRY_EXC_ERR to 192.168.12.5 (iccn005-ib in<br>
> > gpfsremoteclients.localdomain) on mlx4_0 port 1 fabnum 0 vendor_err<br>
> > 129<br>
> > 2018-02-23_13:48:32.523+0100: [E] VERBS RDMA closed connection to<br>
> > 192.168.12.5 (iccn005-ib in gpfsremoteclients.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 due to RDMA read error IBV_WC_RETRY_EXC_ERR index 3<br>
> > 2018-02-23_13:48:35.398+0100: [I] VERBS RDMA accepted and connected<br>
> > to<br>
> > 192.168.12.5 (iccn005-ib in gpfsremoteclients.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 sl 0 index 3<br>
> > 2018-02-23_13:48:53.135+0100: [E] VERBS RDMA rdma read error<br>
> > IBV_WC_RETRY_EXC_ERR to 192.168.12.5 (iccn005-ib in<br>
> > gpfsremoteclients.localdomain) on mlx4_0 port 1 fabnum 0 vendor_err<br>
> > 129<br>
> > 2018-02-23_13:48:53.135+0100: [E] VERBS RDMA closed connection to<br>
> > 192.168.12.5 (iccn005-ib in gpfsremoteclients.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 due to RDMA read error IBV_WC_RETRY_EXC_ERR index 3<br>
> > 2018-02-23_13:48:55.600+0100: [I] VERBS RDMA accepted and connected<br>
> > to<br>
> > 192.168.12.5 (iccn005-ib in gpfsremoteclients.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 sl 0 index 3<br>
> > 2018-02-23_13:49:11.577+0100: [E] VERBS RDMA rdma read error<br>
> > IBV_WC_RETRY_EXC_ERR to 192.168.12.5 (iccn005-ib in<br>
> > gpfsremoteclients.localdomain) on mlx4_0 port 1 fabnum 0 vendor_err<br>
> > 129<br>
> > 2018-02-23_13:49:11.577+0100: [E] VERBS RDMA closed connection to<br>
> > 192.168.12.5 (iccn005-ib in gpfsremoteclients.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 due to RDMA read error IBV_WC_RETRY_EXC_ERR index 3<br>
> > 2018-02-23_13:49:11.939+0100: [I] VERBS RDMA accepted and connected<br>
> > to<br>
> > 192.168.12.5 (iccn005-ib in gpfsremoteclients.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 sl 0 index 3<br>
> ><br>
> ><br>
> ><br>
> > Any advice on how to configure the setup in a way that would allow<br>
> > the remote mount via routed IB would be very appreciated.<br>
> ><br>
> ><br>
> > Thank you and best regards<br>
> > Jan Erik<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > ______________________________<wbr>_________________<br>
> > gpfsug-discuss mailing list<br>
> > gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
</div></div>> > <a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgp" rel="noreferrer" target="_blank">https://emea01.safelinks.<wbr>protection.outlook.com/?url=<wbr>http%3A%2F%2Fgp</a><br>
> > <a href="http://fsug.org" rel="noreferrer" target="_blank">fsug.org</a>%2Fmailman%2Flistinfo%<wbr>2Fgpfsug-discuss&data=01%7C01%<wbr>7Cjohn.h<br>
> > earns%<a href="http://40asml.com" rel="noreferrer" target="_blank">40asml.com</a>%<wbr>7Ce40045550fc3467dd62808d57ed4<wbr>d0d9%7Caf73baa8f5944e<br>
> > b2a39d93e96cad61fc%7C1&sdata=<wbr>v%2F35G6ZnlHFBm%<wbr>2BfVddvcuraFd9FRChyOSRE<br>
> > YpqcNNP8%3D&reserved=0<br>
<span class="">> ><br>
><br>
> --<br>
> Aaron Knister<br>
> NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight<br>
> Center<br>
> <a href="tel:%28301%29%20286-2776" value="+13012862776">(301) 286-2776</a><br>
> ______________________________<wbr>_________________<br>
> gpfsug-discuss mailing list<br>
> gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
</span>> <a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfs" rel="noreferrer" target="_blank">https://emea01.safelinks.<wbr>protection.outlook.com/?url=<wbr>http%3A%2F%2Fgpfs</a><br>
> <a href="http://ug.org" rel="noreferrer" target="_blank">ug.org</a>%2Fmailman%2Flistinfo%<wbr>2Fgpfsug-discuss&data=01%7C01%<wbr>7Cjohn.hearn<br>
> s%<a href="http://40asml.com" rel="noreferrer" target="_blank">40asml.com</a>%<wbr>7Ce40045550fc3467dd62808d57ed4<wbr>d0d9%7Caf73baa8f5944eb2a39d<br>
> 93e96cad61fc%7C1&sdata=v%<wbr>2F35G6ZnlHFBm%<wbr>2BfVddvcuraFd9FRChyOSREYpqcNNP<wbr>8<br>
> %3D&reserved=0<br>
<span class="">><br>
<br>
--<br>
I've never been lost; I was once bewildered for three days, but never lost!<br>
-- Daniel Boone ______________________________<wbr>_________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
</span><a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=01%7C01%7Cjohn.hearns%40asml.com%7Ce40045550fc3467dd62808d57ed4d0d9%7Caf73baa8f5944eb2a39d93e96cad61fc%7C1&sdata=v%2F35G6ZnlHFBm%2BfVddvcuraFd9FRChyOSREYpqcNNP8%3D&reserved=0" rel="noreferrer" target="_blank">https://emea01.safelinks.<wbr>protection.outlook.com/?url=<wbr>http%3A%2F%2Fgpfsug.org%<wbr>2Fmailman%2Flistinfo%2Fgpfsug-<wbr>discuss&data=01%7C01%7Cjohn.<wbr>hearns%40asml.com%<wbr>7Ce40045550fc3467dd62808d57ed4<wbr>d0d9%<wbr>7Caf73baa8f5944eb2a39d93e96cad<wbr>61fc%7C1&sdata=v%<wbr>2F35G6ZnlHFBm%<wbr>2BfVddvcuraFd9FRChyOSREYpqcNNP<wbr>8%3D&reserved=0</a><br>
<span class="im HOEnZb">-- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt.<br>
</span><div class="HOEnZb"><div class="h5">______________________________<wbr>_________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</a><br>
</div></div></blockquote></div><br></div>