<div dir="ltr">Ed,<div><br></div><div>if you enable RDMA, GPFS will always use this as preferred data transfer. if you have subnets configured, GPFS will prefer this for communication with higher priority as the default interface. </div>
<div>so the order is RDMA , subnets, default. </div><div>if RDMA will fail for whatever reason we will use the subnets defined interface and if that fails as well we will use the default interface.</div><div><br></div><div>
the easiest way to see what is used is to run mmdiag --network (only avail on more recent versions of GPFS) it will tell you if RDMA is enabled between individual nodes as well as if a subnet connection is used or not :</div>
<div><br></div><div><div>[root@client05 ~]# mmdiag --network</div><div><br></div><div>=== mmdiag: network ===</div><div><br></div><div>Pending messages:</div><div> (none)</div><div>Inter-node communication configuration:</div>
<div> tscTcpPort 1191</div><div> my address <a href="http://192.167.13.5/16">192.167.13.5/16</a> (eth0) <c0n6></div><div> my addr list <a href="http://192.1.13.5/16">192.1.13.5/16</a> (ib1) <a href="http://192.0.13.5/16">192.0.13.5/16</a> (ib0)/<a href="http://client04.clientad.almaden.ibm.com">client04.clientad.almaden.ibm.com</a> <a href="http://192.167.13.5/16">192.167.13.5/16</a> (eth0)</div>
<div> my node number 17</div><div>TCP Connections between nodes:</div><div> Device ib0:</div><div> hostname node destination status err sock sent(MB) recvd(MB) ostype</div>
<div> client04n1 <c1n0> 192.0.4.1 connected 0 69 0 37 Linux/L</div><div> client04n2 <c1n1> 192.0.4.2 connected 0 70 0 37 Linux/L</div>
<div> client04n3 <c1n2> 192.0.4.3 connected 0 68 0 0 Linux/L</div><div> Device ib1:</div><div> hostname node destination status err sock sent(MB) recvd(MB) ostype</div>
<div> clientcl21 <c0n0> 192.1.201.21 connected 0 65 0 0 Linux/L</div><div> clientcl25 <c0n3> 192.1.201.25 connected 0 66 0 0 Linux/L</div>
<div> clientcl26 <c0n4> 192.1.201.26 connected 0 67 0 0 Linux/L</div><div> clientcl21 <c1n3> 192.1.201.21 connected 0 71 0 0 Linux/L</div>
<div> clientcl22 <c1n4> 192.1.201.22 connected 0 63 0 0 Linux/L</div><div> client10 <c1n5> 192.1.13.10 connected 0 73 0 0 Linux/L</div>
<div> client08 <c1n7> 192.1.13.8 connected 0 72 0 0 Linux/L</div><div>RDMA Connections between nodes:</div><div> Fabric 1 - Device mlx4_0 Port 1 Width 4x Speed FDR lid 13</div>
<div> hostname idx CM state VS buff RDMA_CT(ERR) RDMA_RCV_MB RDMA_SND_MB VS_CT(ERR) VS_SND_MB VS_RCV_MB WAIT_CON_SLOT WAIT_NODE_SLOT</div><div> clientcl21 0 N RTS (Y)903 0 (0 ) 0 0 192 (0 ) 0 0 0 0</div>
<div> client04n1 0 N RTS (Y)477 0 (0 ) 0 0 12367404(0 ) 107905 594 0 0</div><div> client04n1 1 N RTS (Y)477 0 (0 ) 0 0 12367404(0 ) 107901 593 0 0</div>
<div> client04n2 0 N RTS (Y)477 0 (0 ) 0 0 12371352(0 ) 107911 594 0 0</div><div> client04n2 2 N RTS (Y)477 0 (0 ) 0 0 12371352(0 ) 107902 594 0 0</div>
<div> clientcl21 0 N RTS (Y)880 0 (0 ) 0 0 11 (0 ) 0 0 0 0</div><div> client04n3 0 N RTS (Y)969 0 (0 ) 0 0 5 (0 ) 0 0 0 0</div>
<div> clientcl26 0 N RTS (Y)702 0 (0 ) 0 0 35 (0 ) 0 0 0 0</div><div> client08 0 N RTS (Y)637 0 (0 ) 0 0 16 (0 ) 0 0 0 0</div>
<div> clientcl25 0 N RTS (Y)574 0 (0 ) 0 0 14 (0 ) 0 0 0 0</div><div> clientcl22 0 N RTS (Y)507 0 (0 ) 0 0 2 (0 ) 0 0 0 0</div>
<div> client10 0 N RTS (Y)568 0 (0 ) 0 0 121 (0 ) 0 0 0 0</div><div> Fabric 2 - Device mlx4_0 Port 2 Width 4x Speed FDR lid 65</div>
<div> hostname idx CM state VS buff RDMA_CT(ERR) RDMA_RCV_MB RDMA_SND_MB VS_CT(ERR) VS_SND_MB VS_RCV_MB WAIT_CON_SLOT WAIT_NODE_SLOT</div><div> clientcl21 1 N RTS (Y)904 0 (0 ) 0 0 192 (0 ) 0 0 0 0</div>
<div> client04n1 2 N RTS (Y)477 0 (0 ) 0 0 12367404(0 ) 107897 593 0 0</div><div> client04n2 1 N RTS (Y)477 0 (0 ) 0 0 12371352(0 ) 107903 594 0 0</div>
<div> clientcl21 1 N RTS (Y)881 0 (0 ) 0 0 10 (0 ) 0 0 0 0</div><div> clientcl26 1 N RTS (Y)701 0 (0 ) 0 0 35 (0 ) 0 0 0 0</div>
<div> client08 1 N RTS (Y)637 0 (0 ) 0 0 16 (0 ) 0 0 0 0</div><div> clientcl25 1 N RTS (Y)574 0 (0 ) 0 0 14 (0 ) 0 0 0 0</div>
<div> clientcl22 1 N RTS (Y)507 0 (0 ) 0 0 2 (0 ) 0 0 0 0</div><div> client10 1 N RTS (Y)568 0 (0 ) 0 0 121 (0 ) 0 0 0 0</div>
</div><div><br></div><div>in this example you can see thet my client (client05) has multiple subnets configured as well as RDMA. </div><div>so to connected to the various TCP devices (ib0 and ib1) to different cluster nodes and also has a RDMA connection to a different set of nodes. as you can see there is basically no traffic on the TCP devices, as all the traffic uses the 2 defined RDMA fabrics. </div>
<div>there is not a single connection using the daemon interface (eth0) as all nodes are either connected via subnets or via RDMA.</div><div><br></div><div>hope this helps. Sven</div><div> </div></div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Tue, Sep 2, 2014 at 6:44 AM, Ed Wahl <span dir="ltr"><<a href="mailto:ewahl@osc.edu" target="_blank">ewahl@osc.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Seems like you are on the correct track. This is similar to my setup. subnett'ed daemon 10GbE, 1GbE with main being QDR RDMA, admin 1GbE. To my mind the most important part is Setting "privateSubnetOverride" to 1. This allows both your 1GbE and your 40GbE to be on a private subnet. Serving block over public IPs just seems wrong on SO many levels. Whether truly private/internal or not. And how many people use public IPs internally? Wait, maybe I don't want to know...<br>
<br>
Using 'verbsRdma enable' for your FDR seems to override Daemon node name for block, at least in my experience. I love the fallback to 10GbE and then 1GbE in case of disaster when using IB. Lately we seem to be generating bugs in OpenSM at a frightening rate so that has been _extremely_ helpful. Now if we could just monitor when it happens more easily than running mmfsadm test verbs conn, say by logging a failure of RDMA?<br>
<br>
<br>
Ed<br>
OSC<br>
<br>
________________________________________<br>
From: <a href="mailto:gpfsug-discuss-bounces@gpfsug.org">gpfsug-discuss-bounces@gpfsug.org</a> [<a href="mailto:gpfsug-discuss-bounces@gpfsug.org">gpfsug-discuss-bounces@gpfsug.org</a>] on behalf of Simon Thompson (Research Computing - IT Services) [<a href="mailto:S.J.Thompson@bham.ac.uk">S.J.Thompson@bham.ac.uk</a>]<br>
Sent: Monday, September 01, 2014 3:44 PM<br>
To: gpfsug main discussion list<br>
Subject: [gpfsug-discuss] GPFS admin host name vs subnets<br>
<div class="HOEnZb"><div class="h5"><br>
I was just reading through the docs at:<br>
<br>
<a href="https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General+Parallel+File+System+(GPFS)/page/GPFS+Network+Communication+Overview" target="_blank">https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General+Parallel+File+System+(GPFS)/page/GPFS+Network+Communication+Overview</a><br>
<br>
And was wondering about using admin host name bs using subnets. My reading of the page is that if say I have a 1GbE network and a 40GbE network, I could have an admin host name on the 1GbE network. But equally from the docs, it looks like I could also use subnets to achieve the same whilst allowing the admin network to be a fall back for data if necessary.<br>
<br>
For example, create the cluster using the primary name on the 1GbE network, then use the subnets property to use set the network on the 40GbE network as the first and the network on the 1GbE network as the second in the list, thus GPFS data will pass over the 40GbE network in preference and the 1GbE network will, by default only be used for admin traffic as the admin host name will just be the name of the host on the 1GbE network.<br>
<br>
Is my reading of the docs correct? Or do I really want to be creating the cluster using the 40GbE network hostnames and set the admin node name to the name of the 1GbE network interface?<br>
<br>
(there's actually also an FDR switch in there somewhere for verbs as well)<br>
<br>
Thanks<br>
<br>
Simon<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://gpfsug.org" target="_blank">gpfsug.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://gpfsug.org" target="_blank">gpfsug.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
</div></div></blockquote></div><br></div>