<font size=2 face="sans-serif">Hi,</font><br><br><font size=2 face="sans-serif">let me start with a recommendation first
before I explain how the cluster state is build.</font><br><font size=2 face="sans-serif">Starting with 4.2.1 please use the mmhealth
command instead of using the mmces state/events command. The mmces state/event
command will be deprecated in future releases. </font><br><font size=2 face="sans-serif"><i>mmhealth node show -> show the
node state for all components (incl. CES)</i></font><br><font size=2 face="sans-serif"><i>mmhealth node show CES -> shows
the CES components only.</i></font><br><font size=2 face="sans-serif"><i>mmhealth cluster show -> show
the cluster state </i></font><br><br><font size=2 face="sans-serif">Now to your problem:</font><br><font size=2 face="sans-serif">The Spectrum Scale health monitoring
is done by a daemon which runs on each cluster node. </font><br><font size=2 face="sans-serif">This daemon is monitoring the state
of all Spectrum Scale components on the local system and based on the resulting
monitoring events it compiles a local system state (shown by mmhealth node
show).</font><br><font size=2 face="sans-serif">By having a decentralized monitoring
we reduce the monitoring overhead and increase resiliency against network
glitches.</font><br><br><font size=2 face="sans-serif">In order to show a cluster wide state
view we have to consolidate the events from all cluster nodes on a single
node. </font><br><font size=2 face="sans-serif">The health monitoring daemon running
on the cluster manager is taking the role (CSM) to receive events from
all nodes through RPC calls and to compile the cluster state (shown by
mmhealth cluster show)</font><br><font size=2 face="sans-serif">There can be cases where the (async)
event forwarding to the CSM is delayed or dropped because of network delays,
high system load, cluster manager failover or split brain cases.</font><br><font size=2 face="sans-serif">Those cases should resolve automatically
after some time when event is resend.</font><br><br><font size=2 face="sans-serif">Summary: the cluster state might be
temporary out of sync (eventually consistent), for getting a current state
you should refer to mmhealth node show. </font><br><br><font size=2 face="sans-serif">If the problem does not resolve automatically,
restarting the monitoring daemon will force a re-sync. Please open a PMR
for the 5.0 issue too if the problem persist.</font><br><font size=2 face="sans-serif"> </font><br><br><font size=2 face="sans-serif">Mit freundlichen Grüßen / Kind regards<br><br>Mathias Dietz<br><br>Spectrum Scale Development - Release Lead Architect (4.2.x)<br>Spectrum Scale RAS Architect<br>---------------------------------------------------------------------------<br>IBM Deutschland<br>Am Weiher 24<br>65451 Kelsterbach<br>Phone: +49 70342744105<br>Mobile: +49-15152801035<br>E-Mail: mdietz@de.ibm.com<br>-----------------------------------------------------------------------------<br>IBM Deutschland Research & Development GmbH<br>Vorsitzender des Aufsichtsrats: Martina Koederitz, Geschäftsführung: Dirk
WittkoppSitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht
Stuttgart, HRB 243294</font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">"Ernst Heinz
(ID SD)" <heinz.ernst@id.ethz.ch></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">"gpfsug-discuss@spectrumscale.org"
<gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">01/16/2018 06:09 PM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">[gpfsug-discuss]
GPFS GA 5.0.0.0: mmces commands with inconsistent
output</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><font size=2 face="Calibri">Hello to all peers and gurus</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">Since more or less two weeks we have gpfs
GA 5.0.0.0 running on our testenvironment</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">Today I’ve seen following behavior on
our SpectrumScale-testcluster which slighdly surprised me</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">Following:</font><br><font size=2 face="Calibri">Checking status of the cluster on different
ways</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">[root@testnas13ces01 idsd_erh_t1]# mmces
state cluster</font><br><font size=2 face="Calibri">CLUSTER
AUTH
BLOCK NETWORK
AUTH_OBJ <b>NFS</b>
OBJ
<b>SMB</b>
<b>CES</b></font><br><font size=2 face="Calibri">testnas13.ethz.ch
FAILED
DISABLED HEALTHY
DISABLED
<b>DEPEND</b>
DISABLED
<b>DEPEND</b>
<b>FAILED</b></font><br><font size=2 face="Calibri"><b> </b></font><br><font size=2 face="Calibri">[root@testnas13ces01 idsd_erh_t1]# mmces
state show -a</font><br><font size=2 face="Calibri">NODE
AUTH
BLOCK NETWORK
AUTH_OBJ
<b>NFS</b>
OBJ
<b>SMB
CES</b></font><br><font size=2 face="Calibri">testnas13ces01-i
HEALTHY DISABLED
HEALTHY
DISABLED <b>HEALTHY</b> DISABLED
<b>HEALTHY
HEALTHY</b></font><br><font size=2 face="Calibri">testnas13ces02-i
HEALTHY DISABLED
HEALTHY
DISABLED <b>HEALTHY</b> DISABLED
<b>HEALTHY
HEALTHY</b></font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">does anyone of you guys has an explanation
therefore?</font><br><font size=2 face="Calibri">Is there someone else who has seen a behavior
like this?</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">By the way we have a similar view on one
of our clusters on gpfs 4.2.3.4</font><br><font size=2 face="Calibri">(open PMR: 30218.112.848)</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">Any kind of response would be very grateful</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">Kind regards</font><br><font size=2 face="Calibri">Heinz</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">===============================================================</font><br><font size=2 face="Calibri">Heinz Ernst
ID-Systemdienste</font><br><font size=2 face="Calibri">WEC C 16
Weinbergstrasse
11</font><br><font size=2 face="Calibri">CH-8092 Zurich
</font><a href=mailto:heinz.ernst@id.ethz.ch><font size=2 color=#0082bf face="Calibri"><u>heinz.ernst@id.ethz.ch</u></font></a><br><font size=2 face="Calibri">Phone: +41 44 633 84 48
Mobile: +41 79
216 15 50</font><br><font size=2 face="Calibri">===============================================================</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri"> </font><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br></font></tt><br><br><BR>