[gpfsug-discuss] Problem with mmlscluster and callback scripts

[Matthias Knigge]

Hello together,

I am using the version 5.0.2.0 of GPFS and have problems with the command mmlscluster and callback-scripts. It is a small cluster of two nodes only. If I shutdown one of the nodes sometimes mmlscluster reports the following output:
[root at gpfs-tier1 gpfs5.2]# mmgetstate

Node number  Node name        GPFS state
-------------------------------------------
       1      gpfs-tier1       arbitrating
[root at gpfs-tier1 gpfs5.2]# mmlscluster
ssh: connect to host gpfs-tier2 port 22: No route to host
mmlscluster: Unable to retrieve GPFS cluster files from node gpfs-tier2
mmlscluster: Command failed. Examine previous error messages to determine cause.

Normally the output is like this:

[root at gpfs-tier1 gpfs5.2]# mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         TIERCLUSTER.gpfs-tier1
  GPFS cluster id:           12458173498278694815
  GPFS UID domain:           TIERCLUSTER.gpfs-tier1
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp
  Repository type:           server-based

GPFS cluster configuration servers:
-----------------------------------
  Primary server:    gpfs-tier2
  Secondary server:  gpfs-tier1

Node  Daemon node name  IP address      Admin node name  Designation
----------------------------------------------------------------------
   1   gpfs-tier1        192.168.178.10  gpfs-tier1       quorum-manager
   2   gpfs-tier2        192.168.178.11  gpfs-tier2       quorum-manager

[root at gpfs-tier1 gpfs5.2]# mmlscallback
NodeDownCallback
        command       = /var/mmfs/rs/nodedown.ksh
        priority      = 1
        event         = quorumNodeLeave
        parms         = %eventNode %quorumNodes

NodeUpCallback
        command       = /var/mmfs/rs/nodeup.ksh
        priority      = 1
        event         = quorumNodeJoin
        parms         = %eventNode %quorumNodes

If I shutdown the filesystem via mmshutdown the callback script works but if I shutdown the whole node the scripts does not run.
The latest log-entry in mmfs.log.latest shows only this information:

2018-09-07_13:12:36.724+0200: [I] Cluster Manager connection broke. Probing cluster TIERCLUSTER.gpfs-tier1
2018-09-07_13:12:37.226+0200: [E] Unable to contact enough other quorum nodes during cluster probe.
2018-09-07_13:12:37.226+0200: [E] Lost membership in cluster TIERCLUSTER.gpfs-tier1. Unmounting file systems.
2018-09-07_13:12:38.448+0200: [N] Connecting to 192.168.178.11 gpfs-tier2 <c0p1>

Could anybody help me in this case? I want to try to start a script if one node goes down or up to change the roles for starting the filesystem. The callback event NodeLeave and NodeJoin do not run too.
Any more information required? If yes, please let me know!

Many thanks in advance and a nice weekend!
Matthias

Best Regards

Matthias Knigge
R&D File Based Media Solutions

Rohde & Schwarz
GmbH & Co. KG
Hanomaghof 1
30449 Hannover
Telefon +49 511 67 80 7 213
Fax +49 511 37 19 74
Internet: Matthias.Knigge at rohde-schwarz.com
------------------------------------------------------------
Geschäftsführung / Executive Board: Christian Leicher (Vorsitzender / Chairman), Peter Riedel, Sitz der Gesellschaft / Company's Place of Business: München, Registereintrag / Commercial Register No.: HRA 16 270, Persönlich haftender Gesellschafter / Personally Liable Partner: RUSEG Verwaltungs-GmbH, Sitz der Gesellschaft / Company's Place of Business: München, Registereintrag / Commercial Register No.: HRB 7 534, Umsatzsteuer-Identifikationsnummer (USt-IdNr.) / VAT Identification No.: DE 130 256 683, Elektro-Altgeräte Register (EAR) / WEEE Register No.: DE 240 437 86

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20180907/765f5756/attachment-0001.htm>