<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;}
@font-face
{font-family:Poppins;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:12.0pt;
font-family:"Aptos",sans-serif;
mso-ligatures:standardcontextual;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#467886;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Aptos",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="#467886" vlink="#96607D" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hi JAB,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Inline with Achim, I’d be tempted to do the following (based on what’s in your email)<br>
<br>
<o:p></o:p></p>
<p class="MsoNormal">From the failed node<br>
<br>
<o:p></o:p></p>
<p class="MsoNormal">ssh <new node><o:p></o:p></p>
<p class="MsoNormal">rm -rfv /var/mmf/gen<o:p></o:p></p>
<p class="MsoNormal">confirm ssh and/or api auth etc – fix if required<o:p></o:p></p>
<p class="MsoNormal">confirm firewall ports – fix if required<br>
halt -p<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">---------------------------------<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">From a node in the cluster<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">mmdelnode <new node> -p<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">---------------------------------<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Boot new node up again<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Check pings etc, and try again.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">If that doesn’t work; let us know.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<div>
<div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="640" style="width:480.25pt;border-collapse:collapse">
<tbody>
<tr>
<td width="458" valign="top" style="width:343.45pt;padding:0cm 5.4pt 0cm 5.4pt">
<p class="MsoNormal" style="line-height:150%"><b><span style="font-size:11.0pt;line-height:150%;font-family:Poppins;color:#13357A;mso-fareast-language:EN-GB">Laurence Horrocks-Barlow | </span></b><b><span style="font-size:11.0pt;line-height:150%;font-family:Poppins;color:gray;mso-fareast-language:EN-GB">Technical
Director</span></b><span style="font-size:11.0pt;line-height:150%;font-family:"Calibri",sans-serif;mso-ligatures:none;mso-fareast-language:EN-GB"><o:p></o:p></span></p>
</td>
<td width="182" rowspan="2" valign="top" style="width:136.8pt;padding:0cm 5.4pt 0cm 5.4pt">
<p class="MsoNormal" align="right" style="text-align:right;line-height:150%"><a href="https://www.ocf.co.uk/" target="_blank"><span style="font-size:9.0pt;line-height:150%;font-family:Poppins;color:#337AB7;mso-fareast-language:EN-GB;text-decoration:none"><img border="0" width="168" height="113" style="width:1.75in;height:1.175in" id="_x0000_i1028" src="https://ocf.co.uk/media/0i1lzfjz/ocf-logo-strapline-2022-blk.png"></span></a><span style="font-size:11.0pt;line-height:150%;font-family:"Calibri",sans-serif;mso-fareast-language:EN-GB"><o:p></o:p></span></p>
</td>
</tr>
<tr>
<td width="458" valign="top" style="width:343.45pt;padding:0cm 5.4pt 0cm 5.4pt">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="444" style="width:332.65pt;border-collapse:collapse">
<tbody>
<tr>
<td width="77" style="width:58.05pt;padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-bottom:8.0pt;line-height:90%"><b><span style="font-size:8.0pt;line-height:90%;font-family:Poppins;color:#100000;mso-fareast-language:EN-GB">Phone:</span></b><span style="font-size:11.0pt;line-height:90%;font-family:"Calibri",sans-serif;mso-fareast-language:EN-GB"><o:p></o:p></span></p>
</td>
<td width="366" style="width:274.6pt;padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-bottom:8.0pt;line-height:90%"><span style="font-size:8.0pt;line-height:90%;font-family:Poppins;color:#100000;mso-fareast-language:EN-GB">0114 257 2200
</span><span style="font-size:11.0pt;line-height:90%;font-family:"Calibri",sans-serif;mso-fareast-language:EN-GB"><o:p></o:p></span></p>
</td>
</tr>
<tr>
<td width="77" valign="top" style="width:58.05pt;padding:.75pt 0cm 0cm 0cm"></td>
<td width="366" style="width:274.6pt;padding:0cm 0cm 0cm 0cm"></td>
</tr>
<tr>
<td width="77" valign="top" style="width:58.05pt;padding:.75pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-bottom:8.0pt;line-height:90%"><b><span style="font-size:8.0pt;line-height:90%;font-family:Poppins;color:#100000;mso-fareast-language:EN-GB">Address:</span></b><span style="font-size:11.0pt;line-height:90%;font-family:"Calibri",sans-serif;mso-fareast-language:EN-GB"><o:p></o:p></span></p>
</td>
<td width="366" style="width:274.6pt;padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-bottom:8.0pt;line-height:90%"><span style="font-size:8.0pt;line-height:90%;font-family:Poppins;color:#100000;mso-fareast-language:EN-GB">OCF Limited, Unit 5 Rotunda Business Centre, Thorncliffe Park, Chapeltown, Sheffield
S35 2PG</span><span style="font-size:11.0pt;line-height:90%;font-family:"Calibri",sans-serif;mso-fareast-language:EN-GB"><o:p></o:p></span></p>
</td>
</tr>
<tr>
<td width="77" valign="top" style="width:58.05pt;padding:.75pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-bottom:8.0pt;line-height:90%"><b><span style="font-size:8.0pt;line-height:90%;font-family:Poppins;color:#100000;mso-fareast-language:EN-GB">Website:</span></b><span style="font-size:11.0pt;line-height:90%;font-family:"Calibri",sans-serif;mso-fareast-language:EN-GB"><o:p></o:p></span></p>
</td>
<td width="366" style="width:274.6pt;padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-bottom:8.0pt;line-height:90%"><span style="font-size:11.0pt;line-height:90%;mso-fareast-language:EN-GB"><a href="http://www.ocf.co.uk/"><span style="font-size:8.0pt;line-height:90%;font-family:Poppins;color:blue">www.ocf.co.uk</span></a></span><span style="font-size:11.0pt;line-height:90%;font-family:"Calibri",sans-serif;mso-fareast-language:EN-GB"><o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr style="height:1.0pt">
<td width="458" valign="top" style="width:343.45pt;padding:0cm 5.4pt 0cm 5.4pt;height:1.0pt">
<p class="MsoNormal"><a href="https://www.linkedin.com/company/ocf-limited/" target="_blank"><span style="font-size:9.0pt;font-family:Poppins;color:#337AB7;mso-fareast-language:EN-GB;text-decoration:none"><img border="0" width="19" height="19" style="width:.2in;height:.2in" id="_x0000_i1027" src="https://ocf.co.uk/media/2wmpcnw2/linkedin-icons.png" alt="LinkedIn icon"></span></a><span style="font-size:9.0pt;font-family:Poppins;color:#337AB7;mso-fareast-language:EN-GB">
</span><a href="https://twitter.com/ocf_hpc?lang=en" target="_blank"><span style="font-size:9.0pt;font-family:Poppins;color:#337AB7;mso-fareast-language:EN-GB;text-decoration:none"><img border="0" width="19" height="19" style="width:.2in;height:.2in" id="_x0000_i1026" src="https://ocf.co.uk/media/ofxbnlva/twitter-icon.png" alt="Twitter icon"></span></a><span style="font-size:11.0pt;mso-fareast-language:EN-GB"><o:p></o:p></span></p>
</td>
<td width="182" valign="top" style="width:136.8pt;padding:0cm 5.4pt 0cm 5.4pt;height:1.0pt">
</td>
</tr>
<tr>
<td width="640" colspan="2" valign="top" style="width:480.25pt;padding:0cm 5.4pt 0cm 5.4pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Poppins;color:#444444;mso-fareast-language:EN-GB"><img border="0" width="602" height="10" style="width:6.2708in;height:.1041in" id="_x0000_i1025" src="https://ocf.co.uk/media/imkoyi2j/line.jpg"></span><span style="font-size:11.0pt;mso-fareast-language:EN-GB"><o:p></o:p></span></p>
</td>
</tr>
<tr>
<td width="640" colspan="2" valign="top" style="width:480.25pt;padding:0cm 5.4pt 0cm 5.4pt">
<p class="MsoNormal" style="line-height:80%;background:white"><i><span lang="EN-US" style="font-size:8.0pt;line-height:80%;font-family:Poppins;color:#A6A6A6;border:none windowtext 1.0pt;padding:0cm;mso-fareast-language:EN-GB">OCF Limited is a company registered
in England and Wales. Registered number 4132533, VAT number GB 780 6803 14. Registered office address: OCF Limited, 5 Rotunda Business Centre, Thorncliffe Park, Chapeltown, Sheffield, S35 2PG.</span></i><span style="font-size:11.0pt;line-height:80%;font-family:"Calibri",sans-serif;mso-fareast-language:EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal" style="line-height:80%;background:white"><i><span lang="EN-US" style="font-size:8.0pt;line-height:80%;font-family:Poppins;color:#A6A6A6;border:none windowtext 1.0pt;padding:0cm;mso-fareast-language:EN-GB">This message is private and confidential.
If you have received this message in error, please notify us immediately and remove it from your system.</span></i><span style="font-size:11.0pt;line-height:80%;font-family:"Calibri",sans-serif;mso-fareast-language:EN-GB"><o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-GB"> </span><span style="font-size:11.0pt;mso-fareast-language:EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-GB">-------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Poppins;color:black;mso-fareast-language:EN-GB">Scheduled Annual Leave:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Poppins;color:black;mso-fareast-language:EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Poppins;color:black;mso-fareast-language:EN-GB"><a href="https://calendar.google.com/calendar/u/0/embed?src=j5gt5c5fio5rvi4va5po200qf5v55040@import.calendar.google.com&ctz=Europe/London"><span style="color:blue">Click
to see my full annual leave calendar</span></a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Poppins;color:black;mso-fareast-language:EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:8.0pt;font-family:Poppins;color:black;mso-fareast-language:EN-GB">"It is well known that a vital ingredient of success is not knowing that what you're attempting can't be done."
</span></b><span style="font-size:8.0pt;font-family:Poppins;color:black;mso-fareast-language:EN-GB"><br>
<b><i>-- Sir Terry Pratchett</i></b><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:8.0pt;line-height:105%"><span style="font-size:11.0pt;line-height:105%;mso-fareast-language:EN-GB"><o:p> </o:p></span></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal">I think the node was not added to the cluster, but some changes on the node itself took place already (like creating /var/mmfs/gen ... and possibly others)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">There is/was a chapter/appendix to the GPFS Adminstration Guide, telling how to permanently remove GPFS from a node, that might help on how to stop the mmaddnode command to cope with a node already belonging to a cluster ...<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">--<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Mit freundlichen Grüßen / Kind regards<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Achim Rehor<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Technical Support Specialist S<span style="font-family:"Arial",sans-serif"></span>pectrum Scale and ESS (SME)<o:p></o:p></p>
<p class="MsoNormal">Advisory Product Services Professional<o:p></o:p></p>
<p class="MsoNormal">IBM Systems Storage Support - EMEA<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org">Achim.Rehor at de.ibm.com</a><mailto:<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org">Achim.Rehor at de.ibm.com</a>> +49-170-4521194<o:p></o:p></p>
<p class="MsoNormal">IBM Deutschland GmbH<o:p></o:p></p>
<p class="MsoNormal">Vorsitzender des Aufsichtsrats: Sebastian Krause<o:p></o:p></p>
<p class="MsoNormal">Geschäftsführung: Gregor Pillen (Vorsitzender), Nicole Reimer,<o:p></o:p></p>
<p class="MsoNormal">Gabriele Schwarenthorer, Christine Rupp, Frank Theisen<o:p></o:p></p>
<p class="MsoNormal">Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht<o:p></o:p></p>
<p class="MsoNormal">Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">-----Original Message-----<o:p></o:p></p>
<p class="MsoNormal">From: Jonathan Buzzard <<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org">jonathan.buzzard at strath.ac.uk</a><mailto:Jonathan%20Buzzard%20%<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org">3cjonathan.buzzard
at strath.ac.uk</a>%3e>><o:p></o:p></p>
<p class="MsoNormal">Reply-To: gpfsug main discussion list <<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org">gpfsug-discuss at gpfsug.org</a><mailto:gpfsug%20main%20discussion%20list%20%<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org">3cgpfsug-discuss
at gpfsug.org</a>%3e>><o:p></o:p></p>
<p class="MsoNormal">To: <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org">
gpfsug-discuss at gpfsug.org</a> <<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org">gpfsug-discuss at gpfsug.org</a><mailto:%<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org">22gpfsug-discuss at gpfsug.org</a>%22%20%<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org">3cgpfsug-discuss
at gpfsug.org</a>%3e>><o:p></o:p></p>
<p class="MsoNormal">Subject: [EXTERNAL] [gpfsug-discuss] Node in cluster but not in cluster<o:p></o:p></p>
<p class="MsoNormal">Date: Tue, 24 Jun 2025 16:13:48 +0100<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I was trying to add a node to the cluster but after an hour I ctrl-c it<o:p></o:p></p>
<p class="MsoNormal">as it was clearly not in a good place. It told me that it didn't make<o:p></o:p></p>
<p class="MsoNormal">changes<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">^Cmmdsh: Caught SIG INT - terminating the child processes.<o:p></o:p></p>
<p class="MsoNormal">mmaddnode: Interrupt received: No changes made.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I checked to make sure all the networking is correct, GPFS packages are<o:p></o:p></p>
<p class="MsoNormal">installed on the node etc. and it is all good so I go to add it again and<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">mmaddnode: Node <##### censored> was not added to the cluster.<o:p></o:p></p>
<p class="MsoNormal"> The node appears to already belong to a GPFS cluster.<o:p></o:p></p>
<p class="MsoNormal">mmaddnode: mmaddnode quitting. None of the specified nodes are valid.<o:p></o:p></p>
<p class="MsoNormal">mmaddnode: Command failed. Examine previous error messages to determine<o:p></o:p></p>
<p class="MsoNormal">cause.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Ok, lets try and delete it<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">mmdelnode: Incorrect node <##### censored> specified for command.<o:p></o:p></p>
<p class="MsoNormal">mmdelnode: No nodes were found that matched the input specification.<o:p></o:p></p>
<p class="MsoNormal">mmdelnode: Command failed. Examine previous error messages to determine<o:p></o:p></p>
<p class="MsoNormal">cause.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Trying to do mmsdrrestore on the node gives<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">mmsdrrestore: There is no record for this node in file<o:p></o:p></p>
<p class="MsoNormal">gpfs0:/var/mmfs/gen/mmsdrfs.<o:p></o:p></p>
<p class="MsoNormal"> Either the node is not part of the cluster, or the file is for a<o:p></o:p></p>
<p class="MsoNormal">different cluster,<o:p></o:p></p>
<p class="MsoNormal"> or not all of the node's adapter interfaces have been activated yet.<o:p></o:p></p>
<p class="MsoNormal">mmsdrrestore: Command failed. Examine previous error messages to<o:p></o:p></p>
<p class="MsoNormal">determine cause.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The node does not show up in mmlsnode or mmlscluster<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Anyone and idea what is going on?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">JAB.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>