<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style id="owaParaStyle" type="text/css">P {margin-top:0;margin-bottom:0;}</style>
</head>
<body ocsi="0" fpstyle="1" bgcolor="#FFFFFF">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;">I don't see this in the thread but perhaps I missed it, what version are you running? I'm still on 3.5 so this is all based on that.<br>
<br>
A few notes for a little "heads up" here hoping to help with the pitfalls. I seem to recall a number of caveats when I did this a while back. Such as using the 'subnets' option being discussed, stops GPFS from failing over to other TCP networks when there
are failures. VERY important! 'mmdiag --network' will show your setup. Definitely verify this if failing downwards is in your plans. We fail from 56Gb RDMA->10GbE TCP-> 1GbE here. And having had it work during some bad power events last year it was VERY
nice that the users only noticed a slowdown when we completely lost Lustre and other resources.
<br>
<br>
Also I recall that there was a restriction on having multiple private networks, and some special switch to force this. I have a note about "privateSubnetOverride" so you might read up about this. I seem to recall this was for TCP connections and daemonnodename
being a private IP. Or maybe it was that AND mmlscluster having private IPs as well? I think the developerworks wiki had some writeup on this. I don't see it in the admin manuals.
<br>
<br>
Hopefully this may help as you plan this out.<br>
<br>
Ed Wahl<br>
OSC<br>
<br>
<br>
<div style="font-family: Times New Roman; color: #000000; font-size: 16px">
<hr tabindex="-1">
<div style="direction: ltr;" id="divRpF221127"><font face="Tahoma" size="2" color="#000000"><b>From:</b> gpfsug-discuss-bounces@gpfsug.org [gpfsug-discuss-bounces@gpfsug.org] on behalf of Salvatore Di Nardo [sdinardo@ebi.ac.uk]<br>
<b>Sent:</b> Wednesday, July 15, 2015 5:19 AM<br>
<b>To:</b> gpfsug main discussion list<br>
<b>Subject:</b> Re: [gpfsug-discuss] data interface and management infercace.<br>
</font><br>
</div>
<div></div>
<div>Thanks for the input.. this is actually very interesting! <br>
<br>
Reading here: <a class="moz-txt-link-freetext" href="https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General+Parallel+File+System+%28GPFS%29/page/GPFS+Network+Communication+Overview" target="_blank">
https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General+Parallel+File+System+%28GPFS%29/page/GPFS+Network+Communication+Overview</a> ,
<br>
specifically the " Using more than one network" part it seems to me that this way we should be able to split the lease/token/ping from the data.<br>
<br>
Supposing that I implement a GSS cluster with only NDS and a second cluster with only clients:<br>
<br>
<img alt="" src="cid:part1.03040109.00080709@ebi.ac.uk" height="440" width="776"><br>
<br>
As far i understood if on the NDS cluster add first the subnet 10.20.0.0/16 and then 10.30.0.0 is should use the internal network for all the node-to-node comunication, leaving the 10.30.0.0/30 only for data traffic witht he remote cluster ( the clients). Similarly,
in the client cluster, adding first 10.10.0.0/16 and then 10.30.0.0, will guarantee than the node-to-node comunication pass trough a different interface there the data is passing. Since the client are just "clients" the traffic trough 10.10.0.0/16 should be
minimal (only token ,lease, ping and so on ) and not affected by the rest. Should be possible at this point move aldo the "admin network" on the internal interface, so we effectively splitted all the "non data" traffic on a dedicated interface.<br>
<br>
I'm wondering if I'm missing something, and in case i didn't, what could be the real traffic in the internal (black) networks ( 1g link its fine or i still need 10g for that). Another thing I I'm wondering its the load of the "non data" traffic between the
clusters.. i suppose some "daemon traffic" goes trough the blue interface for the inter-cluster communication.
<br>
<br>
<br>
Any thoughts ?<br>
<br>
Salvatore<br>
<br>
<div class="moz-cite-prefix">On 13/07/15 18:19, Muhammad Habib wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>Did you look at "subnets" parameter used with "mmchconfig" command. I think you can use order list of subnets for daemon communication and then actual daemon interface can be used for data transfer. When the GPFS will start it will use actual daemon interface
for communication , however , once its started , it will use the IPs from the subnet list whichever coming first in the list. To further validate , you can put network sniffer before you do actual implementation or alternatively you can open a PMR with IBM.
<br>
<br>
</div>
If your cluster having expel situation , you may fine tune your cluster e.g. increase ping timeout period , having multiple NSD servers and distributing filesystems across these NSD servers. Also critical servers can have HBA cards installed for direct I/O
through fiber. <br>
<br>
</div>
Thanks<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Jul 13, 2015 at 11:22 AM, Jason Hick <span dir="ltr">
<<a href="mailto:jhick@lbl.gov" target="_blank">jhick@lbl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex; border-left:1px #ccc solid; padding-left:1ex">
<div dir="auto">
<div>Hi,</div>
<div><br>
</div>
<div>Yes having separate data and management networks has been critical for us for keeping health monitoring/communication unimpeded by data movement.</div>
<div><br>
</div>
<div>Not as important, but you can also tune the networks differently (packet sizes, buffer sizes, SAK, etc) which can help.</div>
<div><br>
</div>
<div>Jason</div>
<div><br>
On Jul 13, 2015, at 7:25 AM, Vic Cornell <<a href="mailto:viccornell@gmail.com" target="_blank">viccornell@gmail.com</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>Hi Salvatore,
<div><br>
</div>
<div>I agree that that is what the manual - and some of the wiki entries say.</div>
<div><br>
</div>
<div>However , when we have had problems (typically congestion) with ethernet networks in the past (20GbE or 40GbE) we have resolved them by setting up a separate “Admin” network.</div>
<div><br>
</div>
<div>The before and after cluster health we have seen measured in number of expels and waiters has been very marked.</div>
<div><br>
</div>
<div>Maybe someone “in the know” could comment on this split.</div>
<div><br>
</div>
<div>Regards,</div>
<div><br>
</div>
<div>Vic</div>
<div><br>
<br>
<div>
<blockquote type="cite">
<div>On 13 Jul 2015, at 14:29, Salvatore Di Nardo <<a href="mailto:sdinardo@ebi.ac.uk" target="_blank">sdinardo@ebi.ac.uk</a>> wrote:</div>
<br>
<div>
<div bgcolor="#FFFFFF"><font size="-1">Hello Vic.<br>
We are currently draining our gpfs to do all the recabling to add a management network, but looking what the admin interface does ( man mmchnode ) it says something different:<br>
<br>
</font>
<blockquote>
<blockquote><big><font size="-1"><big><tt>--admin-interface={hostname | ip_address}</tt></big></font></big><br>
<big><font size="-1"><big><tt> Specifies the name of the node to be used by GPFS administration commands when communicating between nodes. The admin node name must be specified as an IP</tt></big></font></big><br>
<big><font size="-1"><big><tt> address or a hostname that is resolved by the host command to the desired IP address. If the keyword DEFAULT is specified, the admin interface for the</tt></big></font></big><br>
<big><font size="-1"><big><tt> node is set to be equal to the daemon interface for the node.</tt></big></font></big><br>
</blockquote>
</blockquote>
<font size="-1"><br>
So, seems used only for commands propagation, hence have nothing to do with the node-to-node traffic. Infact the other interface description is:<br>
</font><big><font size="-1"><big><tt><br>
</tt></big></font></big>
<blockquote>
<blockquote><big><font size="-1"><big><tt> --daemon-interface={hostname | ip_address}</tt></big></font></big><br>
<big><font size="-1"><big><tt> Specifies the host name or IP address
</tt><tt><u><b>to be used by the GPFS daemons for node-to-node communication</b></u></tt><tt>. The host name or IP address must refer to the commu-</tt></big></font></big><br>
<big><font size="-1"><big><tt> nication adapter over which the GPFS daemons communicate. Alias interfaces are not allowed. Use the original address or a name that is resolved by the</tt></big></font></big><br>
<big><font size="-1"><big><tt> host command to that original address.</tt></big></font></big></blockquote>
</blockquote>
<font size="-1"><br>
The "expired lease" issue and file locking mechanism a( most of our expells happens when 2 clients try to write in the same file) are exactly node-to node-comunication, so im wondering what's the point to separate the "admin network". I want to be sure to
plan the right changes before we do a so massive task. We are talking about adding a new interface on 700 clients, so the recabling work its not small.
<br>
<br>
<br>
Regards,<br>
Salvatore<br>
<br>
<br>
</font><br>
<div>On 13/07/15 14:00, Vic Cornell wrote:<br>
</div>
<blockquote type="cite">Hi Salavatore,
<div><br>
</div>
<div><span style="white-space:pre-wrap"></span>Does your GSS have the facility for a 1GbE “management” network? If so I think that changing the “admin” node names of the cluster members to a set of IPs on the management network would give you the split that
you need.</div>
<div><br>
</div>
<div>What about the clients? Can they also connect to a separate admin network?</div>
<div><br>
</div>
<div>Remember that if you are using multi-cluster all of the nodes in both networks must share the same admin network.</div>
<div>
<div>
<div><br>
</div>
</div>
</div>
<div>
<div style="font-family:Helvetica; font-size:14px; font-style:normal; font-variant:normal; font-weight:normal; letter-spacing:normal; line-height:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px">
<span>Kind Regards,</span></div>
<div style="font-family:Helvetica; font-size:14px; font-style:normal; font-variant:normal; font-weight:normal; letter-spacing:normal; line-height:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px">
<span><br>
</span></div>
<div style="font-family:Helvetica; font-size:14px; font-style:normal; font-variant:normal; font-weight:normal; letter-spacing:normal; line-height:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px">
<span>Vic</span></div>
<span style="font-family:Helvetica; font-size:14px; font-style:normal; font-variant:normal; font-weight:normal; letter-spacing:normal; line-height:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px"><br>
</span><span></span></div>
<br>
<div>
<blockquote type="cite">
<div>On 13 Jul 2015, at 13:31, Salvatore Di Nardo <<a href="mailto:sdinardo@ebi.ac.uk" target="_blank">sdinardo@ebi.ac.uk</a>> wrote:</div>
<br>
<div>
<div bgcolor="#FFFFFF"><font size="-1">Anyone? </font><br>
<br>
<div>On 10/07/15 11:07, Salvatore Di Nardo wrote:<br>
</div>
<blockquote type="cite"><font size="-1">Hello guys.<br>
Quite a while ago i mentioned that we have a big expel issue on our gss ( first gen) and white a lot people suggested that the root cause could be that we use the same interface for all the traffic, and that we should split the data network from the admin
network. Finally we could plan a downtime and we are migrating the data out so, i can soon safelly play with the change, but looking what exactly i should to do i'm a bit puzzled. Our mmlscluster looks like this:<br>
<br>
</font>
<blockquote>
<blockquote>
<blockquote><tt><font size="-1">GPFS cluster information</font></tt><tt><br>
</tt><tt><font size="-1">========================</font></tt><tt><br>
</tt><tt><font size="-1"> GPFS cluster name: <a href="http://gss.ebi.ac.uk/" target="_blank">
GSS.ebi.ac.uk</a></font></tt><tt><br>
</tt><tt><font size="-1"> GPFS cluster id: 17987981184946329605</font></tt><tt><br>
</tt><tt><font size="-1"> GPFS UID domain: <a href="http://gss.ebi.ac.uk/" target="_blank">
GSS.ebi.ac.uk</a></font></tt><tt><br>
</tt><tt><font size="-1"> Remote shell command: /usr/bin/ssh</font></tt><tt><br>
</tt><tt><font size="-1"> Remote file copy command: /usr/bin/scp</font></tt><tt><br>
</tt><tt><br>
</tt><tt><font size="-1">GPFS cluster configuration servers:</font></tt><tt><br>
</tt><tt><font size="-1">-----------------------------------</font></tt><tt><br>
</tt><tt><font size="-1"> Primary server: <a href="http://gss01a.ebi.ac.uk/" target="_blank">
gss01a.ebi.ac.uk</a></font></tt><tt><br>
</tt><tt><font size="-1"> Secondary server: <a href="http://gss02b.ebi.ac.uk/" target="_blank">
gss02b.ebi.ac.uk</a></font></tt><tt><br>
</tt><tt><br>
</tt><tt><font size="-1"> Node Daemon node name IP address Admin node name Designation</font></tt><tt><br>
</tt><tt><font size="-1">-----------------------------------------------------------------------</font></tt><tt><br>
</tt><tt><font size="-1"> 1 <a href="http://gss01a.ebi.ac.uk/" target="_blank">
gss01a.ebi.ac.uk</a> 10.7.28.2 <a href="http://gss01a.ebi.ac.uk/" target="_blank">
gss01a.ebi.ac.uk</a> quorum-manager</font></tt><tt><br>
</tt><tt><font size="-1"> 2 <a href="http://gss01b.ebi.ac.uk/" target="_blank">
gss01b.ebi.ac.uk</a> 10.7.28.3 <a href="http://gss01b.ebi.ac.uk/" target="_blank">
gss01b.ebi.ac.uk</a> quorum-manager</font></tt><tt><br>
</tt><tt><font size="-1"> 3 <a href="http://gss02a.ebi.ac.uk/" target="_blank">
gss02a.ebi.ac.uk</a> 10.7.28.67 <a href="http://gss02a.ebi.ac.uk/" target="_blank">
gss02a.ebi.ac.uk</a> quorum-manager</font></tt><tt><br>
</tt><tt><font size="-1"> 4 <a href="http://gss02b.ebi.ac.uk/" target="_blank">
gss02b.ebi.ac.uk</a> 10.7.28.66 <a href="http://gss02b.ebi.ac.uk/" target="_blank">
gss02b.ebi.ac.uk</a> quorum-manager</font></tt><tt><br>
</tt><tt><font size="-1"> 5 <a href="http://gss03a.ebi.ac.uk/" target="_blank">
gss03a.ebi.ac.uk</a> 10.7.28.34 <a href="http://gss03a.ebi.ac.uk/" target="_blank">
gss03a.ebi.ac.uk</a> quorum-manager</font></tt><tt><br>
</tt><tt><font size="-1"> 6 <a href="http://gss03b.ebi.ac.uk/" target="_blank">
gss03b.ebi.ac.uk</a> 10.7.28.35 <a href="http://gss03b.ebi.ac.uk/" target="_blank">
gss03b.ebi.ac.uk</a> quorum-manager</font></tt><tt><br>
</tt></blockquote>
</blockquote>
</blockquote>
<font size="-1"><br>
It was my understanding that the "admin node" should use a different interface ( a 1g link copper should be fine), while the daemon node is where the data was passing , so should point to the bonded 10g interfaces. but when i read the mmchnode man page i start
to be quite confused. It says:<br>
<br>
</font><font size="-1"><tt> --daemon-interface={hostname | ip_address}</tt><tt><br>
</tt><tt> Specifies the host name or IP address <u><b>to be used by the GPFS daemons for node-to-node communication</b></u>. The host name or IP address must refer to the communication adapter over which the GPFS daemons communicate.
<br>
Alias interfaces are not allowed. Use the original address or a name that is resolved by the host command to that original address.</tt><tt>
</tt><tt><br>
</tt><tt> </tt><tt><br>
</tt><tt> --admin-interface={hostname | ip_address}</tt><tt><br>
</tt><tt> Specifies the name of the node to be used by GPFS administration commands when communicating between nodes. The admin node name must be specified as an IP address or a hostname that is resolved by the host command
<br>
to</tt><tt> </tt><tt>the desired IP address. If the keyword DEFAULT is specified, the admin interface for the node is set to be equal to the daemon interface for the node.</tt><tt><br>
</tt></font><font size="-1"><br>
What exactly means "node-to node-communications" ? <br>
Means DATA or also the "lease renew", and the token communication between the clients to get/steal the locks to be able to manage concurrent write to thr same file?
<br>
Since we are getting expells ( especially when several clients contends the same file ) i assumed i have to split this type of packages from the data stream, but reading the documentation it looks to me that those internal comunication between nodes use the
daemon-interface wich i suppose are used also for the data. so HOW exactly i can split them?<br>
</font><font size="-1"><br>
</font><font size="-1"><br>
Thanks in advance,<br>
Salvatore<br>
</font><font size="-1"><br>
</font><br>
<fieldset target="_blank"></fieldset> <br>
<pre>_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at <a href="http://gpfsug.org/" target="_blank">gpfsug.org</a>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a>
</pre>
</blockquote>
<br>
</div>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://gpfsug.org/" target="_blank">gpfsug.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
</div>
</blockquote>
</div>
<br>
</blockquote>
<br>
</div>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://gpfsug.org" target="_blank">gpfsug.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<blockquote type="cite">
<div><span>_______________________________________________</span><br>
<span>gpfsug-discuss mailing list</span><br>
<span>gpfsug-discuss at <a href="http://gpfsug.org" target="_blank">gpfsug.org</a></span><br>
<span><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></span><br>
</div>
</blockquote>
</div>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://gpfsug.org" rel="noreferrer" target="_blank">gpfsug.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<div class="gmail_signature">This communication contains confidential information intended only for the persons to whom it is addressed. Any other distribution, copying or disclosure is strictly prohibited. If you have received this communication in error,
please notify the sender and delete this e-mail message immediately.<br>
<br>
Le présent message contient des renseignements de nature confidentielle réservés uniquement à l'usage du destinataire. Toute diffusion, distribution, divulgation, utilisation ou reproduction de la présente communication, et de tout fichier qui y est joint,
est strictement interdite. Si vous avez reçu le présent message électronique par erreur, veuillez informer immédiatement l'expéditeur et supprimer le message de votre ordinateur et de votre serveur.</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader" target="_blank"></fieldset> <br>
<pre>_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
<a class="moz-txt-link-freetext" href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a>
</pre>
</blockquote>
<br>
</div>
</div>
</div>
</body>
</html>