<font size=2 face="sans-serif">Michal,</font><br><br><font size=2 face="sans-serif">When a node is added to a cluster that
has a different domain than the rest of the nodes in the cluster, the GPFS
daemons running on the various nodes can develop an inconsistent understanding
of what the common suffix of all the domain names are. The symptoms
you show with the "tsctl shownodes up" output, and in particular
the incorrect node names of the two nodes you restarted, as seen on a node
you did not restart, are consistent with this problem. I also note
your cluster appears to have the necessary pre-condition to trip on this
problem, whale.img.cas.cz does not share a common suffix with the other
nodes in the cluster. The common suffix of the other nodes in the
cluster is ".img.local". Was whale.img.cas.cz recently
added to the cluster?</font><br><br><font size=2 face="sans-serif">Unfortunately, the general work-around
is to recycle all the nodes at once: mmshutdown -a, followed by mmstartup
-a.</font><br><br><font size=2 face="sans-serif">I hope this helps.</font><br><br><font size=2 face="sans-serif">Regards, The Spectrum Scale (GPFS) team<br><br>------------------------------------------------------------------------------------------------------------------<br>If you feel that your question can benefit other users of Spectrum
Scale (GPFS), then please post it to the public IBM developerWroks Forum
at </font><a href="https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479"><font size=2 face="sans-serif">https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479</font></a><font size=2 face="sans-serif">.
<br><br>If your query concerns a potential software error in Spectrum Scale (GPFS)
and you have an IBM software maintenance contract please contact 1-800-237-5511
in the United States or your local IBM Service Center in other countries.
<br><br>The forum is informally monitored as time permits and should not be used
for priority messages to the Spectrum Scale (GPFS) team.</font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Michal Zacek <zacekm@img.cas.cz></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug-discuss@spectrumscale.org</font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">09/12/2017 05:41 AM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">[gpfsug-discuss]
Wrong nodename after server restart</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><tt><font size=2>Hi,<br><br>I had to restart two of my gpfs servers (gpfs-n4 and gpfs-quorum) and <br>after that I was unable to move CES IP address back with strange error
<br>"mmces address move: GPFS is down on this node". After I double
checked <br>that gpfs state is active on all nodes, I dug deeper and I think I found
<br>problem, but I don't really know how this could happen.<br><br>Look at the names of nodes:<br><br>[root@gpfs-n2 ~]# mmlscluster # Looks good<br><br>GPFS cluster information<br>========================<br> GPFS cluster name: gpfscl1.img.local<br> GPFS cluster id: 17792677515884116443<br> GPFS UID domain: img.local<br> Remote shell command: /usr/bin/ssh<br> Remote file copy command: /usr/bin/scp<br> Repository type: CCR<br><br> Node Daemon node name IP address
Admin node name <br>Designation<br>----------------------------------------------------------------------------------<br> 1 gpfs-n4.img.local 192.168.20.64
gpfs-n4.img.local <br>quorum-manager<br> 2 gpfs-quorum.img.local 192.168.20.60 gpfs-quorum.img.local
quorum<br> 3 gpfs-n3.img.local 192.168.20.63
gpfs-n3.img.local <br>quorum-manager<br> 4 tau.img.local 192.168.1.248
tau.img.local<br> 5 gpfs-n1.img.local 192.168.20.61
gpfs-n1.img.local <br>quorum-manager<br> 6 gpfs-n2.img.local 192.168.20.62
gpfs-n2.img.local <br>quorum-manager<br> 8 whale.img.cas.cz 147.231.150.108
whale.img.cas.cz<br><br><br>[root@gpfs-n2 ~]# mmlsmount gpfs01 -L # not so good<br><br>File system gpfs01 is mounted on 7 nodes:<br> 192.168.20.63 gpfs-n3<br> 192.168.20.61 gpfs-n1<br> 192.168.20.62 gpfs-n2<br> 192.168.1.248 tau<br> 192.168.20.64 gpfs-n4.img.local<br> 192.168.20.60 gpfs-quorum.img.local<br> 147.231.150.108 whale.img.cas.cz<br><br>[root@gpfs-n2 ~]# tsctl shownodes up | tr ',' '\n' # very
wrong<br>whale.img.cas.cz.img.local<br>tau.img.local<br>gpfs-quorum.img.local.img.local<br>gpfs-n1.img.local<br>gpfs-n2.img.local<br>gpfs-n3.img.local<br>gpfs-n4.img.local.img.local<br><br>The "tsctl shownodes up" is the reason why I'm not able to move
CES <br>address back to gpfs-n4 node, but the real problem are different <br>nodenames. I think OS is configured correctly:<br><br>[root@gpfs-n4 /]# hostname<br>gpfs-n4<br><br>[root@gpfs-n4 /]# hostname -f<br>gpfs-n4.img.local<br><br>[root@gpfs-n4 /]# cat /etc/resolv.conf<br>nameserver 192.168.20.30<br>nameserver 147.231.150.2<br>search img.local<br>domain img.local<br><br>[root@gpfs-n4 /]# cat /etc/hosts | grep gpfs-n4<br>192.168.20.64 gpfs-n4.img.local gpfs-n4<br><br>[root@gpfs-n4 /]# host gpfs-n4<br>gpfs-n4.img.local has address 192.168.20.64<br><br>[root@gpfs-n4 /]# host 192.168.20.64<br>64.20.168.192.in-addr.arpa domain name pointer gpfs-n4.img.local.<br><br>Can someone help me with this.<br><br>Thanks,<br>Michal<br><br>p.s. gpfs version: 4.2.3-2 (CentOS 7)<br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=l_sz-tPolX87WmSf2zBhhPpggnfQJKp7-BqV8euBp7A&s=XSPGkKRMza8PhYQg8AxeKW9cOTNeCI9uph486_6Xajo&e="><tt><font size=2>https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=l_sz-tPolX87WmSf2zBhhPpggnfQJKp7-BqV8euBp7A&s=XSPGkKRMza8PhYQg8AxeKW9cOTNeCI9uph486_6Xajo&e=</font></tt></a><tt><font size=2><br><br></font></tt><br><br><BR>