<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I agree with this assessment. I would also recommend looking into user defined node classes so that your mmlsconfig output is more easily readable, otherwise
each node will be listed in the mmlsconfig output.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">HTH,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">-Bryan<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> gpfsug-discuss-bounces@spectrumscale.org [mailto:gpfsug-discuss-bounces@spectrumscale.org]
<b>On Behalf Of </b>David D. Johnson<br>
<b>Sent:</b> Wednesday, February 22, 2017 10:12 AM<br>
<b>To:</b> gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
<b>Subject:</b> Re: [gpfsug-discuss] Changing verbsPorts On Single Node<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I have a feeling that this is how mmchconfig is supposed to work. You’ve asked it to change the<o:p></o:p></p>
<div>
<p class="MsoNormal">configuration of one node, but the database of configuration settings needs to be propagated to<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">the entire cluster whenever a change is made. You’ll find a section in the mmlsconfig output specific<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">to the node(s) that have been changed [node155] …. At this point your configuration may be out of<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">sync on any number of nodes.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> — ddj<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Dave Johnson<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Brown University CCV/CIS<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Feb 22, 2017, at 10:57 AM, Douglas Duckworth <<a href="mailto:dod2014@med.cornell.edu">dod2014@med.cornell.edu</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">Hello!<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">I am an HPC admin at Weill Cornell Medicine in the Upper East Side of Manhattan. It's a great place with researchers working in many computationally demanding fields. I am asked to do many new things all of
the time so it's never boring. Yesterday we deployed a server that's intended to create atomic-level image of a ribosome. Pretty serious science!<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">We have two DDN GridScaler GPFS clusters with around 3PB of storage. FDR Infiniband provides the interconnect. Our compute nodes are Dell PowerEdge 12/13G servers running Centos 6 and 7 while we're using SGE for scheduling. Hopefully
soon Slurm. We also have some GPU servers from Pengiun Computing, with GTX 1080s, as well a new Ryft FPGA accelerator. I am hoping our next round of computing power will come from AMD...<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Anyway, I've been using Ansible to deploy our new GPFS nodes as well as build all other things we need at WCM. I thought that this was complete. However, apparently, the GPFS client's been trying RDMA over port mlx4_0/2 though we need
to use mlx4_0/1! Rather than running mmchconfig against the entire cluster, I have been trying it locally on the node that needs to be addressed. For example:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">sudo mmchconfig verbsPorts=mlx4_0/1 -i -N node155<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">When ran locally the desired change becomes permanent and we see RDMA active after restarting GPFS service on node. Though mmchconfig still tries to run against all nodes in the cluster! I kill it of course at the known_hosts step.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">In addition I tried:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">sudo mmchconfig verbsPorts=mlx4_0/1 -i -N node155 NodeClass=localhost<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">However the same result.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">When doing capital "i" mmchconfig does attempt ssh with all nodes. Yet the change does not persist after restarting GPFS.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">So far I consulted the following documentation:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><a href="http://ibm.co/2mcjK3P">http://ibm.co/2mcjK3P</a><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><a href="http://ibm.co/2lFSInH">http://ibm.co/2lFSInH</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Could anyone please help?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We're using GPFS client version 4.1.1-3 on Centos 6 nodes as well as 4.2.1-2 on those which are running Centos 7.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks so much!<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Best<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Doug<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><br>
Douglas Duckworth, MSc, LFCS<br>
HPC System Administrator<br>
<span style="font-size:9.5pt">Scientific Computing Unit</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Physiology and Biophysics<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Weill Cornell Medicine<o:p></o:p></p>
<div>
<p class="MsoNormal">E: <a href="mailto:doug@med.cornell.edu" target="_blank">doug@med.cornell.edu</a><br>
O: 212-746-6305<br>
F: 212-746-8690<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal">_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><o:p></o:p></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<br>
<hr>
<font face="Arial" color="Gray" size="1"><br>
Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this
email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness
or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial
product.<br>
</font>
</body>
</html>