<font size=1 color=#800080 face="sans-serif">----- Forwarded by Eric Agar/Poughkeepsie/IBM
on 09/13/2017 05:32 PM -----</font>
<br>
<br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">IBM Spectrum Scale/Poughkeepsie/IBM</font>
<br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">Michal Zacek <zacekm@img.cas.cz></font>
<br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">09/13/2017 05:29 PM</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
Wrong nodename after server restart</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">Eric Agar</font>
<br>
<hr noshade>
<br>
<br><font size=2 face="sans-serif">Hello Michal,</font>
<br>
<br><font size=2 face="sans-serif">It should not be necessary to delete
whale.img.cas.cz and rename it. But, that is an option you can take,
if you prefer it. If you decide to take that option, please see the last
paragraph of this response.</font>
<br>
<br><font size=2 face="sans-serif">The confusion starts at the moment a
node is added to the active cluster where the new node does not have the
same common domain suffix as the nodes that were already in the cluster.
The confusion increases when the GPFS daemons on some nodes, but
not all nodes, are recycled. Doing mmshutdown -a, followed by mmstartup
-a, once after the new node has been added allows all GPFS daemons on all
nodes to come up at the same time and arrive at the same answer to the
question, "what is the common domain suffix for all the nodes in the
cluster now?" In the case of your cluster, the answer will be
"the common domain suffix is the empty string" or, put another
way, "there is no common domain suffix"; that is okay, as long
as all the GPFS daemons come to the same conclusion.</font>
<br>
<br><font size=2 face="sans-serif">After you recycle the cluster, you can
check to make sure all seems well by running "tsctl shownodes up"
on every node, and make sure the answer is correct on each node.</font>
<br>
<br><font size=2 face="sans-serif">If the mmshutdown -a / mmstartup -a
recycle works, the problem should not recur with the current set of nodes
in the cluster. Even as individual GPFS daemons are recycled going
forward, they should still understand the cluster's nodes have no common
domain suffix.</font>
<br>
<br><font size=2 face="sans-serif">However, I can imagine sequences of
events that would cause the issue to occur again after nodes are deleted
or added to the cluster while the cluster is active. For example,
if whale.img.cas.cz were to be deleted from the current cluster, that action
would restore the cluster to having a common domain suffix of ".img.local",
but already running GPFS daemons would not realize it. If the delete
of whale occurred while the cluster was active, subsequent recycling of
the GPFS daemon on just a subset of the nodes would cause the recycled
daemons to understand the common domain suffix to now be ".img.local".
But, daemons that had not been recycled would still think there is
no common domain suffix. The confusion would occur again.</font>
<br>
<br><font size=2 face="sans-serif">On the other hand, adding and deleting
nodes to/from the cluster should not cause the issue to occur again as
long as the cluster continues to have the same (in this case, no) common
domain suffix.</font>
<br>
<br><font size=2 face="sans-serif">If you decide to delete whale.img.case.cz,
rename it to have the ".img.local" domain suffix, and add it
back to the cluster, it would be best to do so after all the GPFS daemons
are shut down with mmshutdown -a, but before any of the daemons are restarted
with mmstartup. This would allow all the subsequent running daemons
to come to the conclusion that ".img.local" is now the common
domain suffix.</font>
<br>
<br><font size=2 face="sans-serif">I hope this helps.</font>
<br>
<br><font size=2 face="sans-serif">Regards,</font>
<br><font size=2 face="sans-serif">Eric Agar</font>
<br>
<br><font size=2 face="sans-serif">Regards, The Spectrum Scale (GPFS) team<br>
<br>
------------------------------------------------------------------------------------------------------------------<br>
If you feel that your question can benefit other users of Spectrum
Scale (GPFS), then please post it to the public IBM developerWroks Forum
at </font><a href="https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479"><font size=2 face="sans-serif">https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479</font></a><font size=2 face="sans-serif">.
<br>
<br>
If your query concerns a potential software error in Spectrum Scale (GPFS)
and you have an IBM software maintenance contract please contact 1-800-237-5511
in the United States or your local IBM Service Center in other countries.
<br>
<br>
The forum is informally monitored as time permits and should not be used
for priority messages to the Spectrum Scale (GPFS) team.</font>
<br>
<br>
<br>
<br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Michal Zacek <zacekm@img.cas.cz></font>
<br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">IBM Spectrum Scale
<scale@us.ibm.com></font>
<br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">09/13/2017 03:42 AM</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
Wrong nodename after server restart</font>
<br>
<hr noshade>
<br>
<br>
<br><font size=3>Hello</font>
<p><font size=3>yes you are correct, Whale was added two days a go. It's
necessary to delete whale.img.cas.cz from cluster before mmshutdown/mmstartup?
If the two domains may cause problems in the future I can rename whale
(and all planed nodes) to img.local suffix.<br>
Many thanks for the prompt reply. </font>
<p><font size=3>Regards<br>
Michal</font>
<p>
<br><font size=3>Dne 12.9.2017 v 17:01 IBM Spectrum Scale napsal(a):</font>
<br><font size=2 face="sans-serif">Michal,</font><font size=3><br>
</font><font size=2 face="sans-serif"><br>
When a node is added to a cluster that has a different domain than the
rest of the nodes in the cluster, the GPFS daemons running on the various
nodes can develop an inconsistent understanding of what the common suffix
of all the domain names are. The symptoms you show with the "tsctl
shownodes up" output, and in particular the incorrect node names of
the two nodes you restarted, as seen on a node you did not restart, are
consistent with this problem. I also note your cluster appears to
have the necessary pre-condition to trip on this problem, whale.img.cas.cz
does not share a common suffix with the other nodes in the cluster. The
common suffix of the other nodes in the cluster is ".img.local".
Was whale.img.cas.cz recently added to the cluster?</font><font size=3><br>
</font><font size=2 face="sans-serif"><br>
Unfortunately, the general work-around is to recycle all the nodes at once:
mmshutdown -a, followed by mmstartup -a.</font><font size=3><br>
</font><font size=2 face="sans-serif"><br>
I hope this helps.</font><font size=3><br>
</font><font size=2 face="sans-serif"><br>
Regards, The Spectrum Scale (GPFS) team<br>
<br>
------------------------------------------------------------------------------------------------------------------<br>
If you feel that your question can benefit other users of Spectrum
Scale (GPFS), then please post it to the public IBM developerWroks Forum
at </font><a href="https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479"><font size=2 color=blue face="sans-serif"><u>https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479</u></font></a><font size=2 face="sans-serif">.
<br>
<br>
If your query concerns a potential software error in Spectrum Scale (GPFS)
and you have an IBM software maintenance contract please contact 1-800-237-5511
in the United States or your local IBM Service Center in other countries.
<br>
<br>
The forum is informally monitored as time permits and should not be used
for priority messages to the Spectrum Scale (GPFS) team.</font><font size=3><br>
<br>
<br>
</font><font size=1 color=#5f5f5f face="sans-serif"><br>
From: </font><font size=1 face="sans-serif">Michal
Zacek </font><a href=mailto:zacekm@img.cas.cz><font size=1 color=blue face="sans-serif"><u><zacekm@img.cas.cz></u></font></a><font size=1 color=#5f5f5f face="sans-serif"><br>
To: </font><a href="mailto:gpfsug-discuss@spectrumscale.org"><font size=1 color=blue face="sans-serif"><u>gpfsug-discuss@spectrumscale.org</u></font></a><font size=1 color=#5f5f5f face="sans-serif"><br>
Date: </font><font size=1 face="sans-serif">09/12/2017
05:41 AM</font><font size=1 color=#5f5f5f face="sans-serif"><br>
Subject: </font><font size=1 face="sans-serif">[gpfsug-discuss]
Wrong nodename after server restart</font><font size=1 color=#5f5f5f face="sans-serif"><br>
Sent by: </font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><font size=1 color=blue face="sans-serif"><u>gpfsug-discuss-bounces@spectrumscale.org</u></font></a><font size=3><br>
</font>
<hr noshade><font size=3><br>
<br>
</font><tt><font size=2><br>
Hi,<br>
<br>
I had to restart two of my gpfs servers (gpfs-n4 and gpfs-quorum) and <br>
after that I was unable to move CES IP address back with strange error
<br>
"mmces address move: GPFS is down on this node". After I double
checked <br>
that gpfs state is active on all nodes, I dug deeper and I think I found
<br>
problem, but I don't really know how this could happen.<br>
<br>
Look at the names of nodes:<br>
<br>
[root@gpfs-n2 ~]# mmlscluster # Looks good<br>
<br>
GPFS cluster information<br>
========================<br>
GPFS cluster name: gpfscl1.img.local<br>
GPFS cluster id: 17792677515884116443<br>
GPFS UID domain: img.local<br>
Remote shell command: /usr/bin/ssh<br>
Remote file copy command: /usr/bin/scp<br>
Repository type: CCR<br>
<br>
Node Daemon node name IP address
Admin node name <br>
Designation<br>
----------------------------------------------------------------------------------<br>
1 gpfs-n4.img.local 192.168.20.64
gpfs-n4.img.local <br>
quorum-manager<br>
2 gpfs-quorum.img.local 192.168.20.60 gpfs-quorum.img.local
quorum<br>
3 gpfs-n3.img.local 192.168.20.63
gpfs-n3.img.local <br>
quorum-manager<br>
4 tau.img.local 192.168.1.248
tau.img.local<br>
5 gpfs-n1.img.local 192.168.20.61
gpfs-n1.img.local <br>
quorum-manager<br>
6 gpfs-n2.img.local 192.168.20.62
gpfs-n2.img.local <br>
quorum-manager<br>
8 whale.img.cas.cz 147.231.150.108
whale.img.cas.cz<br>
<br>
<br>
[root@gpfs-n2 ~]# mmlsmount gpfs01 -L # not so good<br>
<br>
File system gpfs01 is mounted on 7 nodes:<br>
192.168.20.63 gpfs-n3<br>
192.168.20.61 gpfs-n1<br>
192.168.20.62 gpfs-n2<br>
192.168.1.248 tau<br>
192.168.20.64 gpfs-n4.img.local<br>
192.168.20.60 gpfs-quorum.img.local<br>
147.231.150.108 whale.img.cas.cz<br>
<br>
[root@gpfs-n2 ~]# tsctl shownodes up | tr ',' '\n' # very
wrong<br>
whale.img.cas.cz.img.local<br>
tau.img.local<br>
gpfs-quorum.img.local.img.local<br>
gpfs-n1.img.local<br>
gpfs-n2.img.local<br>
gpfs-n3.img.local<br>
gpfs-n4.img.local.img.local<br>
<br>
The "tsctl shownodes up" is the reason why I'm not able to move
CES <br>
address back to gpfs-n4 node, but the real problem are different <br>
nodenames. I think OS is configured correctly:<br>
<br>
[root@gpfs-n4 /]# hostname<br>
gpfs-n4<br>
<br>
[root@gpfs-n4 /]# hostname -f<br>
gpfs-n4.img.local<br>
<br>
[root@gpfs-n4 /]# cat /etc/resolv.conf<br>
nameserver 192.168.20.30<br>
nameserver 147.231.150.2<br>
search img.local<br>
domain img.local<br>
<br>
[root@gpfs-n4 /]# cat /etc/hosts | grep gpfs-n4<br>
192.168.20.64 gpfs-n4.img.local gpfs-n4<br>
<br>
[root@gpfs-n4 /]# host gpfs-n4<br>
gpfs-n4.img.local has address 192.168.20.64<br>
<br>
[root@gpfs-n4 /]# host 192.168.20.64<br>
64.20.168.192.in-addr.arpa domain name pointer gpfs-n4.img.local.<br>
<br>
Can someone help me with this.<br>
<br>
Thanks,<br>
Michal<br>
<br>
p.s. gpfs version: 4.2.3-2 (CentOS 7)<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at spectrumscale.org</font></tt><font size=3 color=blue><u><br>
</u></font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=l_sz-tPolX87WmSf2zBhhPpggnfQJKp7-BqV8euBp7A&s=XSPGkKRMza8PhYQg8AxeKW9cOTNeCI9uph486_6Xajo&e="><tt><font size=2 color=blue><u>https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=l_sz-tPolX87WmSf2zBhhPpggnfQJKp7-BqV8euBp7A&s=XSPGkKRMza8PhYQg8AxeKW9cOTNeCI9uph486_6Xajo&e=</u></font></tt></a><tt><font size=2><br>
</font></tt><font size=3><br>
<br>
<br>
<br>
<br>
</font>
<br><tt><font size=3>_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at spectrumscale.org<br>
</font></tt><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMDaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=IS7Hey3ILp7PJIiKIMsU1QALx2VkbMFzlLTifuKeUWM&s=m0Mg-bnQGdZIc0Jj48heHjqG4eq1nuPUhf2tsHHtt9c&e="><tt><font size=3 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></tt></a><tt><font size=3><br>
</font></tt>
<br>
<br><font size=3>-- </font>
<table width=607 style="border-collapse:collapse;">
<tr valign=top height=8>
<td width=607 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.img.cas.cz_&d=DwMDaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=IS7Hey3ILp7PJIiKIMsU1QALx2VkbMFzlLTifuKeUWM&s=_cg8ZEetucvoNkSsqozV_FB0_Bf25UZMMAJyVpjlCEk&e="><img align=bottom src=cid:_4_093BF900093DD2B400766BF88525819A alt="Institute of Molecular Genetics of the ASCR, v.
v. i." style="border:0px solid;"></a>
<tr valign=top height=8>
<td width=607 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=3 color=#0082bf face="Calibri">Michal
®áèek | Information Technologies </font>
<tr valign=top height=8>
<td width=607 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=2 color=#0082bf face="Calibri">+420
296 443 128 </font>
<tr valign=top height=8>
<td width=607 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=2 color=#0082bf face="Calibri">+420
296 443 333 </font>
<tr valign=top height=8>
<td width=607 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><a href=mailto:%0Amichal.zacek@img.cas.cz%0A><font size=2 color=#ff421e face="Calibri"><b>michal.zacek@img.cas.cz
</b></font></a>
<tr valign=top height=8>
<td width=607 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.img.cas.cz_&d=DwMDaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=IS7Hey3ILp7PJIiKIMsU1QALx2VkbMFzlLTifuKeUWM&s=_cg8ZEetucvoNkSsqozV_FB0_Bf25UZMMAJyVpjlCEk&e=" target=_blank><font size=2 color=#ff421e face="Calibri"><b>www.img.cas.cz</b></font></a><font size=2 color=#0082bf face="Calibri"><b>
</b></font>
<tr valign=top height=8>
<td width=607 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=2 color=#0082bf face="Calibri">Institute
of Molecular Genetics of the ASCR, v. v. i., Vídeòská 1083, 142 20 Prague
4, Czech Republic </font>
<tr valign=top height=8>
<td width=607 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=2 color=#0082bf face="Calibri">ID:
68378050 | VAT ID: CZ68378050 </font></table>
<br><font size=3><br>
</font>
<br>
<br>
<br>