[gpfsug-discuss] Node in cluster but not in cluster

Tue Jun 24 17:29:03 BST 2025

Hi JAB,

Inline with Achim, I’d be tempted to do the following (based on what’s in your email)

From the failed node

ssh <new node>
rm -rfv /var/mmf/gen
confirm ssh and/or api auth etc – fix if required
confirm firewall ports – fix if required
halt -p

---------------------------------

From a node in the cluster

mmdelnode <new node> -p

---------------------------------

Boot new node up again

Check pings etc, and try again.

If that doesn’t work; let us know.

Laurence Horrocks-Barlow | Technical Director
[https://ocf.co.uk/media/0i1lzfjz/ocf-logo-strapline-2022-blk.png]<https://www.ocf.co.uk/>
Phone:
0114 257 2200
Address:
OCF Limited, Unit 5 Rotunda Business Centre, Thorncliffe Park, Chapeltown, Sheffield S35 2PG
Website:
www.ocf.co.uk<http://www.ocf.co.uk/>
[LinkedIn icon]<https://www.linkedin.com/company/ocf-limited/>   [Twitter icon] <https://twitter.com/ocf_hpc?lang=en>
[https://ocf.co.uk/media/imkoyi2j/line.jpg]
OCF Limited is a company registered in England and Wales.  Registered number 4132533, VAT number GB 780 6803 14. Registered office address: OCF Limited, 5 Rotunda Business Centre, Thorncliffe Park, Chapeltown, Sheffield, S35 2PG.
This message is private and confidential. If you have received this message in error, please notify us immediately and remove it from your system.

-------------

Scheduled Annual Leave:

Click to see my full annual leave calendar<https://calendar.google.com/calendar/u/0/embed?src=j5gt5c5fio5rvi4va5po200qf5v55040@import.calendar.google.com&ctz=Europe/London>

"It is well known that a vital ingredient of success is not knowing that what you're attempting can't be done."
-- Sir Terry Pratchett

I think the node was not added to the cluster, but some changes on the node itself took place already (like creating /var/mmfs/gen ... and possibly others)

There is/was a chapter/appendix to the GPFS Adminstration Guide, telling how to permanently remove GPFS  from a node,   that might help  on how to stop the mmaddnode command  to cope with a node already belonging to a cluster ...

--

Mit freundlichen Grüßen / Kind regards

Achim Rehor

Technical Support Specialist Spectrum Scale and ESS (SME)
Advisory Product Services Professional
IBM Systems Storage Support - EMEA

Achim.Rehor at de.ibm.com<http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org><mailto:Achim.Rehor at de.ibm.com<http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org>> +49-170-4521194
IBM Deutschland GmbH
Vorsitzender des Aufsichtsrats: Sebastian Krause
Geschäftsführung: Gregor Pillen (Vorsitzender), Nicole Reimer,
Gabriele Schwarenthorer, Christine Rupp, Frank Theisen
Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht
Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940

-----Original Message-----
From: Jonathan Buzzard <jonathan.buzzard at strath.ac.uk<http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org><mailto:Jonathan%20Buzzard%20%3cjonathan.buzzard at strath.ac.uk<http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org>%3e>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org><mailto:gpfsug%20main%20discussion%20list%20%3cgpfsug-discuss at gpfsug.org<http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org>%3e>>
To: gpfsug-discuss at gpfsug.org<http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org> <gpfsug-discuss at gpfsug.org<http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org><mailto:%22gpfsug-discuss at gpfsug.org<http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org>%22%20%3cgpfsug-discuss at gpfsug.org<http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org>%3e>>
Subject: [EXTERNAL] [gpfsug-discuss] Node in cluster but not in cluster
Date: Tue, 24 Jun 2025 16:13:48 +0100

I was trying to add a node to the cluster but after an hour I ctrl-c it
as it was clearly not in a good place. It told me that it didn't make
changes

^Cmmdsh: Caught SIG INT - terminating the child processes.
mmaddnode: Interrupt received: No changes made.

I checked to make sure all the networking is correct, GPFS packages are
installed on the node etc. and it is all good so I go to add it again and

mmaddnode: Node <##### censored> was not added to the cluster.
    The node appears to already belong to a GPFS cluster.
mmaddnode: mmaddnode quitting.  None of the specified nodes are valid.
mmaddnode: Command failed. Examine previous error messages to determine
cause.

Ok, lets try and delete it

mmdelnode: Incorrect node <##### censored> specified for command.
mmdelnode: No nodes were found that matched the input specification.
mmdelnode: Command failed. Examine previous error messages to determine
cause.

Trying to do mmsdrrestore on the node gives

mmsdrrestore: There is no record for this node in file
gpfs0:/var/mmfs/gen/mmsdrfs.
   Either the node is not part of the cluster, or the file is for a
different cluster,
   or not all of the node's adapter interfaces have been activated yet.
mmsdrrestore: Command failed. Examine previous error messages to
determine cause.

The node does not show up in mmlsnode or mmlscluster

Anyone and idea what is going on?

JAB.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20250624/9670fc42/attachment-0001.htm>