<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<font size="-1">Thanks,<br>
this has already been done ( without too much success). <br>
We need to rearrange the networking and since somebody experience
was to add a copper interface for management i want to do the same,
so i'm digging a bit to aundertsand the best way yo do it.<br>
</font><small><br>
Regards,<br>
Salvatore<br>
</small><br>
<br>
<div class="moz-cite-prefix">On 14/07/15 08:31, Hagley Birgit wrote:<br>
</div>
<blockquote
cite="mid:97B2355E006F044E9B8518711889B13719CF3810@MBX114.d.ethz.ch"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<style id="owaParaStyle" type="text/css">P {margin-top:0;margin-bottom:0;}</style>
<div style="direction: ltr;font-family: Tahoma;color:
#000000;font-size: 10pt;">Hello Salvatore,<br>
<br>
as you wrote that you have about 700 clients, maybe also the
tuning recommendations for large GPFS clusters are helpful for
you. They are on the developerworks GPFS wiki:<br>
<br>
<a moz-do-not-send="true"
href="https://www.ibm.com/developerworks/community/wikis/home?lang=en#%21/wiki/General%20Parallel%20File%20System%20%28GPFS%29/page/Best%20Practices%20Network%20Tuning"
target="_blank">https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20%28GPFS%29/page/Best%20Practices%20Network%20Tuning</a><br>
<br>
<br>
To my experience especially "failureDetectionTime" and
"minMissedPingTimeout" may help in case of expelled nodes.<br>
<br>
<br>
In case you use InfiniBand, for RDMA, there also is a "Best
Practices RDMA Tuning" page:<br>
<br>
<a moz-do-not-send="true"
href="https://www.ibm.com/developerworks/community/wikis/home?lang=en#%21/wiki/General%20Parallel%20File%20System%20%28GPFS%29/page/Best%20Practices%20RDMA%20Tuning"
target="_blank">https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20%28GPFS%29/page/Best%20Practices%20RDMA%20Tuning</a><br>
<br>
<br>
<br>
Regards<br>
Birgit<br>
<br>
<div style="font-family: Times New Roman; color: #000000;
font-size: 16px">
<hr tabindex="-1">
<div style="direction: ltr;" id="divRpF534040"><font
face="Tahoma" size="2" color="#000000"><b>From:</b>
<a class="moz-txt-link-abbreviated" href="mailto:gpfsug-discuss-bounces@gpfsug.org">gpfsug-discuss-bounces@gpfsug.org</a>
[<a class="moz-txt-link-abbreviated" href="mailto:gpfsug-discuss-bounces@gpfsug.org">gpfsug-discuss-bounces@gpfsug.org</a>] on behalf of Salvatore
Di Nardo [<a class="moz-txt-link-abbreviated" href="mailto:sdinardo@ebi.ac.uk">sdinardo@ebi.ac.uk</a>]<br>
<b>Sent:</b> Monday, July 13, 2015 3:29 PM<br>
<b>To:</b> Vic Cornell<br>
<b>Cc:</b> gpfsug main discussion list<br>
<b>Subject:</b> Re: [gpfsug-discuss] data interface and
management infercace.<br>
</font><br>
</div>
<div><font size="-1">Hello Vic.<br>
We are currently draining our gpfs to do all the recabling
to add a management network, but looking what the admin
interface does ( man mmchnode ) it says something
different:<br>
<br>
</font>
<blockquote>
<blockquote><big><font size="-1"><big><tt>--admin-interface={hostname
| ip_address}</tt></big></font></big><br>
<big><font size="-1"><big><tt>
Specifies the name of the node to be used by
GPFS administration commands when communicating
between nodes. The admin node name must be
specified as an IP</tt></big></font></big><br>
<big><font size="-1"><big><tt>
address or a hostname that is resolved by the
host command to the desired IP address. If the
keyword DEFAULT is specified, the admin
interface for the</tt></big></font></big><br>
<big><font size="-1"><big><tt>
node is set to be equal to the daemon interface
for the node.</tt></big></font></big><br>
</blockquote>
</blockquote>
<font size="-1"><br>
So, seems used only for commands propagation, hence have
nothing to do with the node-to-node traffic. Infact the
other interface description is:<br>
</font><big><font size="-1"><big><tt><br>
</tt></big></font></big>
<blockquote>
<blockquote><big><font size="-1"><big><tt> --daemon-interface={hostname
| ip_address}</tt></big></font></big><br>
<big><font size="-1"><big><tt>
Specifies the host name or IP address
</tt><tt><u><b>to be used by the GPFS daemons for
node-to-node communication</b></u></tt><tt>.
The host name or IP address must refer to the
commu-</tt></big></font></big><br>
<big><font size="-1"><big><tt>
nication adapter over which the GPFS daemons
communicate. Alias interfaces are not allowed.
Use the original address or a name that is
resolved by the</tt></big></font></big><br>
<big><font size="-1"><big><tt>
host command to that original address.</tt></big></font></big></blockquote>
</blockquote>
<font size="-1"><br>
The "expired lease" issue and file locking mechanism a(
most of our expells happens when 2 clients try to write in
the same file) are exactly node-to node-comunication, so
im wondering what's the point to separate the "admin
network". I want to be sure to plan the right changes
before we do a so massive task. We are talking about
adding a new interface on 700 clients, so the recabling
work its not small.
<br>
<br>
<br>
Regards,<br>
Salvatore<br>
<br>
<br>
</font><br>
<div class="moz-cite-prefix">On 13/07/15 14:00, Vic Cornell
wrote:<br>
</div>
<blockquote type="cite">Hi Salavatore,
<div class=""><br class="">
</div>
<div class=""><span class="Apple-tab-span"
style="white-space:pre"></span>Does your GSS have the
facility for a 1GbE “management” network? If so I think
that changing the “admin” node names of the cluster
members to a set of IPs on the management network would
give you the split that you need.</div>
<div class=""><br class="">
</div>
<div class="">What about the clients? Can they also
connect to a separate admin network?</div>
<div class=""><br class="">
</div>
<div class="">Remember that if you are using multi-cluster
all of the nodes in both networks must share the same
admin network.</div>
<div class="">
<div class="">
<div class=""><br class="">
</div>
</div>
</div>
<div class="">
<div class="" style="color:rgb(0,0,0);
font-family:Helvetica; font-size:14px;
font-style:normal; font-variant:normal;
font-weight:normal; letter-spacing:normal;
line-height:normal; orphans:auto; text-align:start;
text-indent:0px; text-transform:none;
white-space:normal; widows:auto; word-spacing:0px">
<span>Kind Regards,</span></div>
<div class="" style="color:rgb(0,0,0);
font-family:Helvetica; font-size:14px;
font-style:normal; font-variant:normal;
font-weight:normal; letter-spacing:normal;
line-height:normal; orphans:auto; text-align:start;
text-indent:0px; text-transform:none;
white-space:normal; widows:auto; word-spacing:0px">
<span><br class="">
</span></div>
<div class="" style="color:rgb(0,0,0);
font-family:Helvetica; font-size:14px;
font-style:normal; font-variant:normal;
font-weight:normal; letter-spacing:normal;
line-height:normal; orphans:auto; text-align:start;
text-indent:0px; text-transform:none;
white-space:normal; widows:auto; word-spacing:0px">
<span>Vic</span></div>
<span style="color:rgb(0,0,0); font-family:Helvetica;
font-size:14px; font-style:normal;
font-variant:normal; font-weight:normal;
letter-spacing:normal; line-height:normal;
orphans:auto; text-align:start; text-indent:0px;
text-transform:none; white-space:normal; widows:auto;
word-spacing:0px"><br
class="Apple-interchange-newline">
</span><span></span></div>
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On 13 Jul 2015, at 13:31, Salvatore Di
Nardo <<a moz-do-not-send="true"
href="mailto:sdinardo@ebi.ac.uk" class=""
target="_blank">sdinardo@ebi.ac.uk</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div bgcolor="#FFFFFF" class=""><font class=""
size="-1">Anyone? </font><br class="">
<br class="">
<div class="moz-cite-prefix">On 10/07/15 11:07,
Salvatore Di Nardo wrote:<br class="">
</div>
<blockquote type="cite" class=""><font class=""
size="-1">Hello guys.<br class="">
Quite a while ago i mentioned that we have a
big expel issue on our gss ( first gen) and
white a lot people suggested that the root
cause could be that we use the same interface
for all the traffic, and that we should split
the data network from the admin network.
Finally we could plan a downtime and we are
migrating the data out so, i can soon safelly
play with the change, but looking what exactly
i should to do i'm a bit puzzled. Our
mmlscluster looks like this:<br class="">
<br class="">
</font>
<blockquote class="">
<blockquote class="">
<blockquote class=""><tt class=""><font
class="" size="-1">GPFS cluster
information</font></tt><tt class=""><br
class="">
</tt><tt class=""><font class="" size="-1">========================</font></tt><tt
class=""><br class="">
</tt><tt class=""><font class="" size="-1">
GPFS cluster name: <a
moz-do-not-send="true"
href="http://GSS.ebi.ac.uk" class=""
target="_blank">
GSS.ebi.ac.uk</a></font></tt><tt
class=""><br class="">
</tt><tt class=""><font class="" size="-1">
GPFS cluster id:
17987981184946329605</font></tt><tt
class=""><br class="">
</tt><tt class=""><font class="" size="-1">
GPFS UID domain: <a
moz-do-not-send="true"
href="http://GSS.ebi.ac.uk" class=""
target="_blank">
GSS.ebi.ac.uk</a></font></tt><tt
class=""><br class="">
</tt><tt class=""><font class="" size="-1">
Remote shell command:
/usr/bin/ssh</font></tt><tt class=""><br
class="">
</tt><tt class=""><font class="" size="-1">
Remote file copy command:
/usr/bin/scp</font></tt><tt class=""><br
class="">
</tt><tt class=""><br class="">
</tt><tt class=""><font class="" size="-1">GPFS
cluster configuration servers:</font></tt><tt
class=""><br class="">
</tt><tt class=""><font class="" size="-1">-----------------------------------</font></tt><tt
class=""><br class="">
</tt><tt class=""><font class="" size="-1">
Primary server: <a
moz-do-not-send="true"
href="http://gss01a.ebi.ac.uk"
class="" target="_blank">
gss01a.ebi.ac.uk</a></font></tt><tt
class=""><br class="">
</tt><tt class=""><font class="" size="-1">
Secondary server: <a
moz-do-not-send="true"
href="http://gss02b.ebi.ac.uk"
class="" target="_blank">
gss02b.ebi.ac.uk</a></font></tt><tt
class=""><br class="">
</tt><tt class=""><br class="">
</tt><tt class=""><font class="" size="-1"> Node
Daemon node name IP address Admin
node name Designation</font></tt><tt
class=""><br class="">
</tt><tt class=""><font class="" size="-1">-----------------------------------------------------------------------</font></tt><tt
class=""><br class="">
</tt><tt class=""><font class="" size="-1">
1 <a moz-do-not-send="true"
href="http://gss01a.ebi.ac.uk"
class="" target="_blank">
gss01a.ebi.ac.uk</a> 10.7.28.2
<a moz-do-not-send="true"
href="http://gss01a.ebi.ac.uk"
class="" target="_blank">
gss01a.ebi.ac.uk</a>
quorum-manager</font></tt><tt class=""><br
class="">
</tt><tt class=""><font class="" size="-1">
2 <a moz-do-not-send="true"
href="http://gss01b.ebi.ac.uk"
class="" target="_blank">
gss01b.ebi.ac.uk</a> 10.7.28.3
<a moz-do-not-send="true"
href="http://gss01b.ebi.ac.uk"
class="" target="_blank">
gss01b.ebi.ac.uk</a>
quorum-manager</font></tt><tt class=""><br
class="">
</tt><tt class=""><font class="" size="-1">
3 <a moz-do-not-send="true"
href="http://gss02a.ebi.ac.uk"
class="" target="_blank">
gss02a.ebi.ac.uk</a> 10.7.28.67
<a moz-do-not-send="true"
href="http://gss02a.ebi.ac.uk"
class="" target="_blank">
gss02a.ebi.ac.uk</a>
quorum-manager</font></tt><tt class=""><br
class="">
</tt><tt class=""><font class="" size="-1">
4 <a moz-do-not-send="true"
href="http://gss02b.ebi.ac.uk"
class="" target="_blank">
gss02b.ebi.ac.uk</a> 10.7.28.66
<a moz-do-not-send="true"
href="http://gss02b.ebi.ac.uk"
class="" target="_blank">
gss02b.ebi.ac.uk</a>
quorum-manager</font></tt><tt class=""><br
class="">
</tt><tt class=""><font class="" size="-1">
5 <a moz-do-not-send="true"
href="http://gss03a.ebi.ac.uk"
class="" target="_blank">
gss03a.ebi.ac.uk</a> 10.7.28.34
<a moz-do-not-send="true"
href="http://gss03a.ebi.ac.uk"
class="" target="_blank">
gss03a.ebi.ac.uk</a>
quorum-manager</font></tt><tt class=""><br
class="">
</tt><tt class=""><font class="" size="-1">
6 <a moz-do-not-send="true"
href="http://gss03b.ebi.ac.uk"
class="" target="_blank">
gss03b.ebi.ac.uk</a> 10.7.28.35
<a moz-do-not-send="true"
href="http://gss03b.ebi.ac.uk"
class="" target="_blank">
gss03b.ebi.ac.uk</a>
quorum-manager</font></tt><tt class=""><br
class="">
</tt></blockquote>
</blockquote>
</blockquote>
<font class="" size="-1"><br class="">
It was my understanding that the "admin node"
should use a different interface ( a 1g link
copper should be fine), while the daemon node
is where the data was passing , so should
point to the bonded 10g interfaces. but when
i read the mmchnode man page i start to be
quite confused. It says:<br class="">
<br class="">
</font><font class="" size="-1"><tt class="">
--daemon-interface={hostname
| ip_address}</tt><tt class=""><br class="">
</tt><tt class="">
Specifies the host name or IP address
<u class=""><b class="">to be used by the
GPFS daemons for node-to-node
communication</b></u>. The host name or
IP address must refer to the communication
adapter over which the GPFS daemons
communicate.
<br class="">
Alias interfaces
are not allowed. Use the original address or
a name that is resolved by the host command
to that original address.</tt><tt class="">
</tt><tt class=""><br class="">
</tt><tt class=""> </tt><tt
class=""><br class="">
</tt><tt class="">
--admin-interface={hostname | ip_address}</tt><tt
class=""><br class="">
</tt><tt class="">
Specifies the name of the node to be used by
GPFS administration commands when
communicating between nodes. The admin node
name must be specified as an IP address or a
hostname that is resolved by the host
command
<br class="">
to</tt><tt class="">
</tt><tt class="">the desired IP address. If
the keyword DEFAULT is specified, the admin
interface for the node is set to be equal to
the daemon interface for the node.</tt><tt
class=""><br class="">
</tt></font><font class="" size="-1"><br
class="">
What exactly means "node-to
node-communications" ? <br class="">
Means DATA or also the "lease renew", and the
token communication between the clients to
get/steal the locks to be able to manage
concurrent write to thr same file?
<br class="">
Since we are getting expells ( especially when
several clients contends the same file ) i
assumed i have to split this type of packages
from the data stream, but reading the
documentation it looks to me that those
internal comunication between nodes use the
daemon-interface wich i suppose are used also
for the data. so HOW exactly i can split them?<br
class="">
</font><font class="" size="-1"><br class="">
</font><font class="" size="-1"><br class="">
Thanks in advance,<br class="">
Salvatore<br class="">
</font><font class="" size="-1"><br class="">
</font><br class="">
<fieldset class="mimeAttachmentHeader"
target="_blank"></fieldset>
<br class="">
<pre class="">_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at <a moz-do-not-send="true" href="http://gpfsug.org" class="" target="_blank">gpfsug.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a>
</pre>
</blockquote>
<br class="">
</div>
_______________________________________________<br
class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a moz-do-not-send="true"
href="http://gpfsug.org" class="" target="_blank">gpfsug.org</a><br
class="">
<a moz-do-not-send="true"
href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"
class="" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br
class="">
</div>
</blockquote>
</div>
<br class="">
</blockquote>
<br>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
<a class="moz-txt-link-freetext" href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a>
</pre>
</blockquote>
<br>
</body>
</html>