<div dir="ltr">Hi Jan,<div><br></div><div>I am NOT using the pre-populated cache that mellanox refers to in it's documentation. After chatting with support, I don't believe that's necessary anymore (I didn't get a straight answer out of them).</div><div><br></div><div>For the subnet prefix, make sure to use one from the range <span style="color:rgb(61,61,61);font-family:Calibri,sans-serif;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">0xfec0000000000000-0xfec000000000001f. </span><br></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><span style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px">---------------------------------------------------------------------------------------------------------------</span><br></div><div dir="ltr"><span style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px">Zach Mance <a href="mailto:zmance@ucar.edu" target="_blank">zmance@ucar.edu</a></span><span style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px"><span> </span> (303) 497-1883 <span> </span></span><br style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px"><span style="font-size:12.8px">HPC Data Infrastructure Group</span><span style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px"> / CISL / NCAR</span><br style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px"><span style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px">--------------------------------------------------------------------------------------------------------------- </span><br></div></div></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Tue, Mar 13, 2018 at 9:24 AM, Jan Erik Sundermann <span dir="ltr"><<a href="mailto:jan.sundermann@kit.edu" target="_blank">jan.sundermann@kit.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello Zachary<br>
<br>
We are currently changing out setup to have IP over IB on all machines to be able to enable verbsRdmaCm.<br>
<br>
According to Mellanox (<a href="https://community.mellanox.com/docs/DOC-2384" rel="noreferrer" target="_blank">https://community.mellanox.co<wbr>m/docs/DOC-2384</a>) ibacm requires pre-populated caches to be distributed to all end hosts with the mapping of IP to the routable GIDs (of both IB subnets). Was this also required in your successful deployment?<br>
<br>
Best<br>
Jan Erik<span class=""><br>
<br>
<br>
<br>
On 03/12/2018 11:10 PM, Zachary Mance wrote:<br>
</span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
Since I am testing out remote mounting with EDR IB routers, I'll add to the discussion.<br>
<br>
In my lab environment I was seeing the same rdma connections being established and then disconnected shortly after. The remote filesystem would eventually mount on the clients, but it look a quite a while (~2mins). Even after mounting, accessing files or any metadata operations would take a while to execute, but eventually it happened.<br>
<br>
After enabling verbsRdmaCm, everything mounted just fine and in a timely manner. Spectrum Scale was using the librdmacm.so library.<br>
<br>
I would first double check that you have both clusters able to talk to each other on their IPoIB address, then make sure you enable verbsRdmaCm on both clusters.<br>
<br>
<br>
------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>---------------------<br></span>
Zach Mance <a href="mailto:zmance@ucar.edu" target="_blank">zmance@ucar.edu</a> <mailto:<a href="mailto:zmance@ucar.edu" target="_blank">zmance@ucar.edu</a>> <a href="tel:%28303%29%20497-1883" value="+13034971883" target="_blank">(303) 497-1883</a><span class=""><br>
HPC Data Infrastructure Group / CISL / NCAR<br>
------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>--------------------- <br>
<br></span><span class="">
On Thu, Mar 1, 2018 at 1:41 AM, John Hearns <<a href="mailto:john.hearns@asml.com" target="_blank">john.hearns@asml.com</a> <mailto:<a href="mailto:john.hearns@asml.com" target="_blank">john.hearns@asml.com</a>>> wrote:<br>
<br>
In reply to Stuart,<br>
our setup is entirely Infiniband. We boot and install over IB, and<br>
rely heavily on IP over Infiniband.<br>
<br>
As for users being 'confused' due to multiple IPs, I would<br>
appreciate some more depth on that one.<br>
Sure, all batch systems are sensitive to hostnames (as I know to my<br>
cost!) but once you get that straightened out why should users care?<br>
I am not being aggressive, just keen to find out more.<br>
<br>
<br>
<br>
-----Original Message-----<br>
From: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectru<wbr>mscale.org</a><br>
<mailto:<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces<wbr>@spectrumscale.org</a>><br></span><span class="">
[mailto:<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces<wbr>@spectrumscale.org</a><br>
<mailto:<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces<wbr>@spectrumscale.org</a>>] On Behalf Of<br>
Stuart Barkley<br>
Sent: Wednesday, February 28, 2018 6:50 PM<br>
To: gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.<wbr>org</a><br></span><div><div class="h5">
<mailto:<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectru<wbr>mscale.org</a>>><br>
Subject: Re: [gpfsug-discuss] Problems with remote mount via routed IB<br>
<br>
The problem with CM is that it seems to require configuring IP over<br>
Infiniband.<br>
<br>
I'm rather strongly opposed to IP over IB. We did run IPoIB years<br>
ago, but pulled it out of our environment as adding unneeded<br>
complexity. It requires provisioning IP addresses across the<br>
Infiniband infrastructure and possibly adding routers to other<br>
portions of the IP infrastructure. It was also confusing some users<br>
due to multiple IPs on the compute infrastructure.<br>
<br>
We have recently been in discussions with a vendor about their<br>
support for GPFS over IB and they kept directing us to using CM<br>
(which still didn't work). CM wasn't necessary once we found out<br>
about the actual problem (we needed the undocumented<br>
verbsRdmaUseGidIndexZero configuration option among other things due<br>
to their use of SR-IOV based virtual IB interfaces).<br>
<br>
We don't use routed Infiniband and it might be that CM and IPoIB is<br>
required for IB routing, but I doubt it. It sounds like the OP is<br>
keeping IB and IP infrastructure separate.<br>
<br>
Stuart Barkley<br>
<br>
On Mon, 26 Feb 2018 at 14:16 -0000, Aaron Knister wrote:<br>
<br>
> Date: Mon, 26 Feb 2018 14:16:34<br>
> From: Aaron Knister <<a href="mailto:aaron.s.knister@nasa.gov" target="_blank">aaron.s.knister@nasa.gov</a><br></div></div>
<mailto:<a href="mailto:aaron.s.knister@nasa.gov" target="_blank">aaron.s.knister@nasa.g<wbr>ov</a>>><span class=""><br>
> Reply-To: gpfsug main discussion list<br>
> <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.<wbr>org</a><br></span>
<mailto:<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectru<wbr>mscale.org</a>>><br>
> To: <a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.o<wbr>rg</a><br>
<mailto:<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectru<wbr>mscale.org</a>><span class=""><br>
> Subject: Re: [gpfsug-discuss] Problems with remote mount via<br>
routed IB<br>
><br>
> Hi Jan Erik,<br>
><br>
> It was my understanding that the IB hardware router required RDMA<br>
CM to work.<br>
> By default GPFS doesn't use the RDMA Connection Manager but it can be<br>
> enabled (e.g. verbsRdmaCm=enable). I think this requires a restart on<br>
> clients/servers (in both clusters) to take effect. Maybe someone else<br>
> on the list can comment in more detail-- I've been told folks have<br>
> successfully deployed IB routers with GPFS.<br>
><br>
> -Aaron<br>
><br>
> On 2/26/18 11:38 AM, Sundermann, Jan Erik (SCC) wrote:<br>
> ><br>
> > Dear all<br>
> ><br>
> > we are currently trying to remote mount a file system in a routed<br>
> > Infiniband test setup and face problems with dropped RDMA<br>
> > connections. The setup is the<br>
> > following:<br>
> ><br>
> > - Spectrum Scale Cluster 1 is setup on four servers which are<br>
> > connected to the same infiniband network. Additionally they are<br>
> > connected to a fast ethernet providing ip communication in the<br></span>
network <a href="http://192.168.11.0/24" rel="noreferrer" target="_blank">192.168.11.0/24</a> <<a href="http://192.168.11.0/24" rel="noreferrer" target="_blank">http://192.168.11.0/24</a>>.<span class=""><br>
> ><br>
> > - Spectrum Scale Cluster 2 is setup on four additional servers<br>
which<br>
> > are connected to a second infiniband network. These servers<br>
have IPs<br>
> > on their IB interfaces in the network <a href="http://192.168.12.0/24" rel="noreferrer" target="_blank">192.168.12.0/24</a><br></span>
<<a href="http://192.168.12.0/24" rel="noreferrer" target="_blank">http://192.168.12.0/24</a>>.<br>
> ><br>
> > - IP is routed between <a href="http://192.168.11.0/24" rel="noreferrer" target="_blank">192.168.11.0/24</a> <<a href="http://192.168.11.0/24" rel="noreferrer" target="_blank">http://192.168.11.0/24</a>><br>
and <a href="http://192.168.12.0/24" rel="noreferrer" target="_blank">192.168.12.0/24</a> <<a href="http://192.168.12.0/24" rel="noreferrer" target="_blank">http://192.168.12.0/24</a>> on a<div><div class="h5"><br>
> > dedicated machine.<br>
> ><br>
> > - We have a dedicated IB hardware router connected to both IB<br>
subnets.<br>
> ><br>
> ><br>
> > We tested that the routing, both IP and IB, is working between the<br>
> > two clusters without problems and that RDMA is working fine<br>
both for<br>
> > internal communication inside cluster 1 and cluster 2<br>
> ><br>
> > When trying to remote mount a file system from cluster 1 in cluster<br>
> > 2, RDMA communication is not working as expected. Instead we see<br>
> > error messages on the remote host (cluster 2)<br>
> ><br>
> ><br>
> > 2018-02-23_13:48:47.037+0100: [I] VERBS RDMA connecting to<br>
> > 192.168.11.4 (iccn004-gpfs in gpfsstorage.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 index 2<br>
> > 2018-02-23_13:48:49.890+0100: [I] VERBS RDMA connected to<br>
> > 192.168.11.4 (iccn004-gpfs in gpfsstorage.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 sl 0 index 2<br>
> > 2018-02-23_13:48:53.138+0100: [E] VERBS RDMA closed connection to<br>
> > 192.168.11.1 (iccn001-gpfs in gpfsstorage.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 error 733 index 3<br>
> > 2018-02-23_13:48:53.854+0100: [I] VERBS RDMA connecting to<br>
> > 192.168.11.1 (iccn001-gpfs in gpfsstorage.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 index 3<br>
> > 2018-02-23_13:48:54.954+0100: [E] VERBS RDMA closed connection to<br>
> > 192.168.11.3 (iccn003-gpfs in gpfsstorage.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 error 733 index 1<br>
> > 2018-02-23_13:48:55.601+0100: [I] VERBS RDMA connected to<br>
> > 192.168.11.1 (iccn001-gpfs in gpfsstorage.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 sl 0 index 3<br>
> > 2018-02-23_13:48:57.775+0100: [I] VERBS RDMA connecting to<br>
> > 192.168.11.3 (iccn003-gpfs in gpfsstorage.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 index 1<br>
> > 2018-02-23_13:48:59.557+0100: [I] VERBS RDMA connected to<br>
> > 192.168.11.3 (iccn003-gpfs in gpfsstorage.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 sl 0 index 1<br>
> > 2018-02-23_13:48:59.876+0100: [E] VERBS RDMA closed connection to<br>
> > 192.168.11.2 (iccn002-gpfs in gpfsstorage.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 error 733 index 0<br>
> > 2018-02-23_13:49:02.020+0100: [I] VERBS RDMA connecting to<br>
> > 192.168.11.2 (iccn002-gpfs in gpfsstorage.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 index 0<br>
> > 2018-02-23_13:49:03.477+0100: [I] VERBS RDMA connected to<br>
> > 192.168.11.2 (iccn002-gpfs in gpfsstorage.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 sl 0 index 0<br>
> > 2018-02-23_13:49:05.119+0100: [E] VERBS RDMA closed connection to<br>
> > 192.168.11.4 (iccn004-gpfs in gpfsstorage.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 error 733 index 2<br>
> > 2018-02-23_13:49:06.191+0100: [I] VERBS RDMA connecting to<br>
> > 192.168.11.4 (iccn004-gpfs in gpfsstorage.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 index 2<br>
> > 2018-02-23_13:49:06.548+0100: [I] VERBS RDMA connected to<br>
> > 192.168.11.4 (iccn004-gpfs in gpfsstorage.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 sl 0 index 2<br>
> > 2018-02-23_13:49:11.578+0100: [E] VERBS RDMA closed connection to<br>
> > 192.168.11.1 (iccn001-gpfs in gpfsstorage.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 error 733 index 3<br>
> > 2018-02-23_13:49:11.937+0100: [I] VERBS RDMA connecting to<br>
> > 192.168.11.1 (iccn001-gpfs in gpfsstorage.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 index 3<br>
> > 2018-02-23_13:49:11.939+0100: [I] VERBS RDMA connected to<br>
> > 192.168.11.1 (iccn001-gpfs in gpfsstorage.localdomain) on mlx4_0<br>
> > port 1 fabnum 0 sl 0 index 3<br>
> ><br>
> ><br>
> > and in the cluster with the file system (cluster 1)<br>
> ><br>
> > 2018-02-23_13:47:36.112+0100: [E] VERBS RDMA rdma read error<br>
> > IBV_WC_RETRY_EXC_ERR to 192.168.12.5 (iccn005-ib in<br>
> > gpfsremoteclients.localdomain) on mlx4_0 port 1 fabnum 0 vendor_err<br>
> > 129<br>
> > 2018-02-23_13:47:36.112+0100: [E] VERBS RDMA closed connection to<br>
> > 192.168.12.5 (iccn005-ib in gpfsremoteclients.localdomain) on<br>
mlx4_0<br>
> > port 1 fabnum 0 due to RDMA read error IBV_WC_RETRY_EXC_ERR index 3<br>
> > 2018-02-23_13:47:47.161+0100: [I] VERBS RDMA accepted and connected<br>
> > to<br>
> > 192.168.12.5 (iccn005-ib in gpfsremoteclients.localdomain) on<br>
mlx4_0<br>
> > port 1 fabnum 0 sl 0 index 3<br>
> > 2018-02-23_13:48:04.317+0100: [E] VERBS RDMA rdma read error<br>
> > IBV_WC_RETRY_EXC_ERR to 192.168.12.5 (iccn005-ib in<br>
> > gpfsremoteclients.localdomain) on mlx4_0 port 1 fabnum 0 vendor_err<br>
> > 129<br>
> > 2018-02-23_13:48:04.317+0100: [E] VERBS RDMA closed connection to<br>
> > 192.168.12.5 (iccn005-ib in gpfsremoteclients.localdomain) on<br>
mlx4_0<br>
> > port 1 fabnum 0 due to RDMA read error IBV_WC_RETRY_EXC_ERR index 3<br>
> > 2018-02-23_13:48:11.560+0100: [I] VERBS RDMA accepted and connected<br>
> > to<br>
> > 192.168.12.5 (iccn005-ib in gpfsremoteclients.localdomain) on<br>
mlx4_0<br>
> > port 1 fabnum 0 sl 0 index 3<br>
> > 2018-02-23_13:48:32.523+0100: [E] VERBS RDMA rdma read error<br>
> > IBV_WC_RETRY_EXC_ERR to 192.168.12.5 (iccn005-ib in<br>
> > gpfsremoteclients.localdomain) on mlx4_0 port 1 fabnum 0 vendor_err<br>
> > 129<br>
> > 2018-02-23_13:48:32.523+0100: [E] VERBS RDMA closed connection to<br>
> > 192.168.12.5 (iccn005-ib in gpfsremoteclients.localdomain) on<br>
mlx4_0<br>
> > port 1 fabnum 0 due to RDMA read error IBV_WC_RETRY_EXC_ERR index 3<br>
> > 2018-02-23_13:48:35.398+0100: [I] VERBS RDMA accepted and connected<br>
> > to<br>
> > 192.168.12.5 (iccn005-ib in gpfsremoteclients.localdomain) on<br>
mlx4_0<br>
> > port 1 fabnum 0 sl 0 index 3<br>
> > 2018-02-23_13:48:53.135+0100: [E] VERBS RDMA rdma read error<br>
> > IBV_WC_RETRY_EXC_ERR to 192.168.12.5 (iccn005-ib in<br>
> > gpfsremoteclients.localdomain) on mlx4_0 port 1 fabnum 0 vendor_err<br>
> > 129<br>
> > 2018-02-23_13:48:53.135+0100: [E] VERBS RDMA closed connection to<br>
> > 192.168.12.5 (iccn005-ib in gpfsremoteclients.localdomain) on<br>
mlx4_0<br>
> > port 1 fabnum 0 due to RDMA read error IBV_WC_RETRY_EXC_ERR index 3<br>
> > 2018-02-23_13:48:55.600+0100: [I] VERBS RDMA accepted and connected<br>
> > to<br>
> > 192.168.12.5 (iccn005-ib in gpfsremoteclients.localdomain) on<br>
mlx4_0<br>
> > port 1 fabnum 0 sl 0 index 3<br>
> > 2018-02-23_13:49:11.577+0100: [E] VERBS RDMA rdma read error<br>
> > IBV_WC_RETRY_EXC_ERR to 192.168.12.5 (iccn005-ib in<br>
> > gpfsremoteclients.localdomain) on mlx4_0 port 1 fabnum 0 vendor_err<br>
> > 129<br>
> > 2018-02-23_13:49:11.577+0100: [E] VERBS RDMA closed connection to<br>
> > 192.168.12.5 (iccn005-ib in gpfsremoteclients.localdomain) on<br>
mlx4_0<br>
> > port 1 fabnum 0 due to RDMA read error IBV_WC_RETRY_EXC_ERR index 3<br>
> > 2018-02-23_13:49:11.939+0100: [I] VERBS RDMA accepted and connected<br>
> > to<br>
> > 192.168.12.5 (iccn005-ib in gpfsremoteclients.localdomain) on<br>
mlx4_0<br>
> > port 1 fabnum 0 sl 0 index 3<br>
> ><br>
> ><br>
> ><br>
> > Any advice on how to configure the setup in a way that would allow<br>
> > the remote mount via routed IB would be very appreciated.<br>
> ><br>
> ><br>
> > Thank you and best regards<br>
> > Jan Erik<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > ______________________________<wbr>_________________<br>
> > gpfsug-discuss mailing list<br></div></div>
> > gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a> <<a href="http://spectrumscale.org" rel="noreferrer" target="_blank">http://spectrumscale.org</a>><br>
> ><br>
<a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgp" rel="noreferrer" target="_blank">https://emea01.safelinks.prote<wbr>ction.outlook.com/?url=http%<wbr>3A%2F%2Fgp</a><br>
<<a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgp" rel="noreferrer" target="_blank">https://emea01.safelinks.prot<wbr>ection.outlook.com/?url=http%<wbr>3A%2F%2Fgp</a>><br>
> > <a href="http://fsug.org" rel="noreferrer" target="_blank">fsug.org</a><br>
<<a href="http://fsug.org" rel="noreferrer" target="_blank">http://fsug.org</a>>%2Fmailman%2F<wbr>listinfo%2Fgpfsug-discuss&data<wbr>=01%7C01%7Cjohn.h<br>
> > earns%<a href="http://40asml.com" rel="noreferrer" target="_blank">40asml.com</a><br>
<<a href="http://40asml.com" rel="noreferrer" target="_blank">http://40asml.com</a>>%7Ce4004555<wbr>0fc3467dd62808d57ed4d0d9%<wbr>7Caf73baa8f5944e<span class=""><br>
> ><br>
b2a39d93e96cad61fc%7C1&sdata=v<wbr>%2F35G6ZnlHFBm%2BfVddvcuraFd9F<wbr>RChyOSRE<br>
> > YpqcNNP8%3D&reserved=0<br>
> ><br>
><br>
> --<br>
> Aaron Knister<br>
> NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight<br>
> Center<br></span>
> <a href="tel:%28301%29%20286-2776" value="+13012862776" target="_blank">(301) 286-2776</a> <tel:%28301%29%20286-2776><br>
> ______________________________<wbr>_________________<br>
> gpfsug-discuss mailing list<br>
> gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a> <<a href="http://spectrumscale.org" rel="noreferrer" target="_blank">http://spectrumscale.org</a>><br>
><br>
<a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfs" rel="noreferrer" target="_blank">https://emea01.safelinks.prote<wbr>ction.outlook.com/?url=http%<wbr>3A%2F%2Fgpfs</a><br>
<<a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfs" rel="noreferrer" target="_blank">https://emea01.safelinks.prot<wbr>ection.outlook.com/?url=http%<wbr>3A%2F%2Fgpfs</a>><br>
> <a href="http://ug.org" rel="noreferrer" target="_blank">ug.org</a><br>
<<a href="http://ug.org" rel="noreferrer" target="_blank">http://ug.org</a>>%2Fmailman%2Fli<wbr>stinfo%2Fgpfsug-discuss&data=<wbr>01%7C01%7Cjohn.hearn<br>
> s%<a href="http://40asml.com" rel="noreferrer" target="_blank">40asml.com</a><br>
<<a href="http://40asml.com" rel="noreferrer" target="_blank">http://40asml.com</a>>%7Ce4004555<wbr>0fc3467dd62808d57ed4d0d9%<wbr>7Caf73baa8f5944eb2a39d<span class=""><br>
><br>
93e96cad61fc%7C1&sdata=v%2F35G<wbr>6ZnlHFBm%2BfVddvcuraFd9FRChyOS<wbr>REYpqcNNP8<br>
> %3D&reserved=0<br>
><br>
<br>
--<br>
I've never been lost; I was once bewildered for three days, but<br>
never lost!<br>
-- Daniel Boone<br>
______________________________<wbr>_________________<br>
gpfsug-discuss mailing list<br></span>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a> <<a href="http://spectrumscale.org" rel="noreferrer" target="_blank">http://spectrumscale.org</a>><br>
<a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=01%7C01%7Cjohn.hearns%40asml.com%7Ce40045550fc3467dd62808d57ed4d0d9%7Caf73baa8f5944eb2a39d93e96cad61fc%7C1&sdata=v%2F35G6ZnlHFBm%2BfVddvcuraFd9FRChyOSREYpqcNNP8%3D&reserved=0" rel="noreferrer" target="_blank">https://emea01.safelinks.prote<wbr>ction.outlook.com/?url=http%<wbr>3A%2F%2Fgpfsug.org%2Fmailman%<wbr>2Flistinfo%2Fgpfsug-discuss&<wbr>data=01%7C01%7Cjohn.hearns%<wbr>40asml.com%7Ce40045550fc3467dd<wbr>62808d57ed4d0d9%7Caf73baa8f594<wbr>4eb2a39d93e96cad61fc%7C1&<wbr>sdata=v%2F35G6ZnlHFBm%2BfVddvc<wbr>uraFd9FRChyOSREYpqcNNP8%3D&<wbr>reserved=0</a><span class=""><br>
<<a href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=01%7C01%7Cjohn.hearns%40asml.com%7Ce40045550fc3467dd62808d57ed4d0d9%7Caf73baa8f5944eb2a39d93e96cad61fc%7C1&sdata=v%2F35G6ZnlHFBm%2BfVddvcuraFd9FRChyOSREYpqcNNP8%3D&reserved=0" rel="noreferrer" target="_blank">https://emea01.safelinks.prot<wbr>ection.outlook.com/?url=http%<wbr>3A%2F%2Fgpfsug.org%2Fmailman%<wbr>2Flistinfo%2Fgpfsug-discuss&<wbr>data=01%7C01%7Cjohn.hearns%<wbr>40asml.com%7Ce40045550fc3467dd<wbr>62808d57ed4d0d9%7Caf73baa8f594<wbr>4eb2a39d93e96cad61fc%7C1&<wbr>sdata=v%2F35G6ZnlHFBm%2BfVddvc<wbr>uraFd9FRChyOSREYpqcNNP8%3D&<wbr>reserved=0</a>><br>
-- The information contained in this communication and any<br>
attachments is confidential and may be privileged, and is for the<br>
sole use of the intended recipient(s). Any unauthorized review, use,<br>
disclosure or distribution is prohibited. Unless explicitly stated<br>
otherwise in the body of this communication or the attachment<br>
thereto (if any), the information is provided on an AS-IS basis<br>
without any express or implied warranties or liabilities. To the<br>
extent you are relying on this information, you are doing so at your<br>
own risk. If you are not the intended recipient, please notify the<br>
sender immediately by replying to this message and destroy all<br>
copies of this message and any attachments. Neither the sender nor<br>
the company/group of companies he or she represents shall be liable<br>
for the proper and complete transmission of the information<br>
contained in this communication, or for any delay in its receipt.<br>
______________________________<wbr>_________________<br>
gpfsug-discuss mailing list<br></span>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a> <<a href="http://spectrumscale.org" rel="noreferrer" target="_blank">http://spectrumscale.org</a>><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/list<wbr>info/gpfsug-discuss</a><br>
<<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/lis<wbr>tinfo/gpfsug-discuss</a>><span class=""><br>
<br>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/list<wbr>info/gpfsug-discuss</a><br>
<br>
</span></blockquote><div class="HOEnZb"><div class="h5">
<br>
-- <br>
<br>
Karlsruhe Institute of Technology (KIT)<br>
Steinbuch Centre for Computing (SCC)<br>
<br>
Jan Erik Sundermann<br>
<br>
Hermann-von-Helmholtz-Platz 1, Building 449, Room 226<br>
D-76344 Eggenstein-Leopoldshafen<br>
<br>
Tel: <a href="tel:%2B49%20721%20608%2026191" value="+4972160826191" target="_blank">+49 721 608 26191</a><br>
Email: <a href="mailto:jan.sundermann@kit.edu" target="_blank">jan.sundermann@kit.edu</a><br>
<a href="http://www.scc.kit.edu" rel="noreferrer" target="_blank">www.scc.kit.edu</a><br>
<br>
KIT – The Research University in the Helmholtz Association<br>
<br>
Since 2010, KIT has been certified as a family-friendly university.<br>
<br>
</div></div><br>______________________________<wbr>_________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</a><br>
<br></blockquote></div><br></div>