<span style=" font-size:10pt;font-family:sans-serif">Hi</span><br><br><span style=" font-size:10pt;font-family:sans-serif">Just check :</span><br><br><span style=" font-size:10pt;font-family:sans-serif">1) getenfore -
Selinux status</span><br><span style=" font-size:10pt;font-family:sans-serif">2) check if FW
is active - iptables -L</span><br><span style=" font-size:10pt;font-family:sans-serif">3) do u have ping
to the host report in mmlscluster ? /etc/hosts valid ? DNS is valid ?</span><br><span style=" font-size:9pt;font-family:Arial"> </span><br><span style=" font-size:10pt;font-family:Arial">Regards</span><br><span style=" font-size:9pt;font-family:Arial"> </span><br><table width=780 style="border-collapse:collapse;"><tr height=8><td width=780 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><div align=center><hr noshade></div><br><span style=" font-size:1pt;font-family:Arial"> </span><tr height=8><td width=780 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:1pt;font-family:Arial"> </span><tr height=8><td width=516 colspan=2 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:10pt;color:blue;font-family:Arial"><b>Yaron
Daniel</b></span><td width=168 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:10pt;color:#5f5f5f;font-family:Arial"> 94
Em Ha'Moshavot Rd</span><td width=96 rowspan=3 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><div align=right><img align=bottom src=cid:_1_06E97D4406E97790005E7B46C22582BD style="border:0px solid;"></div><tr height=8><td width=516 colspan=2 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:10pt;color:blue;font-family:Arial"><b>Storage
Architect – IL Lab Services (Storage)</b></span><td width=168 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:10pt;color:#5f5f5f;font-family:Arial"> Petach
Tiqva, 49527</span><tr height=8><td width=516 colspan=2 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:10pt;color:blue;font-family:Arial"><b>IBM
Global Markets, Systems HW Sales</b></span><td width=168 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:10pt;color:#5f5f5f;font-family:Arial"> Israel</span><tr height=8><td width=516 colspan=2 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:10pt;color:blue;font-family:Arial"><b> </b></span><td width=168 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:10pt;color:#5f5f5f;font-family:Arial"> </span><td width=96 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:9pt;font-family:Arial"> </span><tr height=8><td width=90 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:10pt;color:#5f5f5f;font-family:Arial">Phone:</span><td width=426 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:10pt;color:#5f5f5f;font-family:Arial">+972-3-916-5672</span><td width=168 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:10pt;color:#5f5f5f;font-family:Arial"> </span><td width=96 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:10pt"> </span><tr height=8><td width=90 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:10pt;color:#5f5f5f;font-family:Arial">Fax:</span><td width=426 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:10pt;color:#5f5f5f;font-family:Arial">+972-3-916-5672</span><td width=168 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:10pt;color:#5f5f5f;font-family:Arial">
</span><td width=96 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:10pt"> </span><tr height=8><td width=90 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:10pt;color:#5f5f5f;font-family:Arial">Mobile:</span><td width=426 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:10pt;color:#5f5f5f;font-family:Arial">+972-52-8395593</span><td width=168 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:10pt;color:#5f5f5f;font-family:Arial">
</span><td width=96 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:10pt"> </span><tr height=8><td width=90 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:10pt;color:#5f5f5f;font-family:Arial">e-mail:</span><td width=426 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:10pt;color:#5f5f5f;font-family:Arial">yard@il.ibm.com</span><td width=168 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:10pt;color:#5f5f5f;font-family:Arial">
</span><td width=96 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:10pt"> </span><tr height=8><td width=516 colspan=2 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><a href="http://www.ibm.com/il/he/"><span style=" font-size:10pt;color:blue;font-family:Arial"><u>IBM
Israel</u></span></a><td width=168 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:10pt;color:#5f5f5f;font-family:Arial">
</span><td width=96 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:10pt"> </span><tr height=8><td width=780 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:9pt;color:#5f5f5f;font-family:Arial"> </span><tr height=8><td width=780 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:9pt;color:#5f5f5f;font-family:Arial"> </span></table><p style="margin-top:0px;margin-Bottom:0px"></p><br><img src=cid:_1_06EBF47C06EBF06C005E7B46C22582BD alt="IBM Storage Strategy and Solutions v1" style="border:0px solid;"><img src=cid:_1_06EBF68406EBF06C005E7B46C22582BD alt="IBM Storage Management and Data Protection v1" style="border:0px solid;"><img src=cid:_1_06EBF88C06EBF06C005E7B46C22582BD style="border:0px solid;"><img src=cid:_1_06EBFA9406EBF06C005E7B46C22582BD style="border:0px solid;"><span style=" font-size:12pt"> </span><img src=cid:_1_06EBFCB406EBF06C005E7B46C22582BD alt="https://acclaim-production-app.s3.amazonaws.com/images/6c2c3858-6df8-45be-ac2b-f93b8da74e20/Data%2BDriven%2BMulti%2BCloud%2BStrategy%2BV1%2Bver%2B4.png" style="border:0px solid;"><span style=" font-size:12pt">
</span><img src=cid:_2_06EBFF0006EBF06C005E7B46C22582BD alt="Related image" style="border:0px solid;"><br><br><br><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">From:
</span><span style=" font-size:9pt;font-family:sans-serif">"Uwe
Falke" <UWEFALKE@de.ibm.com></span><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">To:
</span><span style=" font-size:9pt;font-family:sans-serif">renata@SLAC.STANFORD.EDU,
gpfsug main discussion list <gpfsug-discuss@spectrumscale.org></span><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Cc:
</span><span style=" font-size:9pt;font-family:sans-serif">gpfsug-discuss-bounces@spectrumscale.org</span><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Date:
</span><span style=" font-size:9pt;font-family:sans-serif">06/28/2018
10:45 AM</span><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Subject:
</span><span style=" font-size:9pt;font-family:sans-serif">Re:
[gpfsug-discuss] gpfs client cluster, lost quorum, ccr issues</span><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Sent
by: </span><span style=" font-size:9pt;font-family:sans-serif">gpfsug-discuss-bounces@spectrumscale.org</span><br><hr noshade><br><br><br><tt><span style=" font-size:10pt">Just some ideas what to try.<br>when you attempted mmdelnode, was that node still active with the IP <br>address known in the cluster? If so, shut it down and try again.<br>Mind the restrictions of mmdelnode though (can't delete NSD servers).<br><br>Try to fake one of the currently missing cluster nodes, or restore the
old <br>system backup to the reinstalled server, if available, or temporarily <br>install gpfs SW there and copy over the GPFS config stuff from a
node <br>still active (/var/mmfs/), configure the admin and daemon IFs of the faked
<br>node on that machine, then try to start the cluster and see if it comes
up <br>with quorum, if it does then go ahead and cleanly de-configure what's
<br>needed to remove that node from the cluster gracefully. Once you reach
<br>quorum with the remaining nodes you are in safe area.<br><br><br> <br>Mit freundlichen Grüßen / Kind regards<br><br> <br>Dr. Uwe Falke<br> <br>IT Specialist<br>High Performance Computing Services / Integrated Technology Services /
<br>Data Center Services<br>-------------------------------------------------------------------------------------------------------------------------------------------<br>IBM Deutschland<br>Rathausstr. 7<br>09111 Chemnitz<br>Phone: +49 371 6978 2165<br>Mobile: +49 175 575 2877<br>E-Mail: uwefalke@de.ibm.com<br>-------------------------------------------------------------------------------------------------------------------------------------------<br>IBM Deutschland Business & Technology Services GmbH / Geschäftsführung:
<br>Thomas Wolter, Sven Schooß<br>Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart,
<br>HRB 17122 <br><br><br><br><br>From: Renata Maria Dart <renata@SLAC.STANFORD.EDU><br>To: Simon Thompson <S.J.Thompson@bham.ac.uk><br>Cc: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Date: 27/06/2018 21:30<br>Subject: Re: [gpfsug-discuss] gpfs client cluster,
lost quorum, ccr <br>issues<br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br><br><br><br>Hi Simon, yes I ran<br><br>mmsdrrestore -p <working node in the cluster><br><br>and that helped to create the /var/mmfs/ccr directory which was<br>missing. But it didn't create a ccr.nodes file, so I ended up scp'ng<br>that over by hand which I hope was the right thing to do. The one<br>host that is no longer in service is still in that ccr.nodes file and<br>when I try to mmdelnode it I get:<br><br>root@ocio-gpu03 renata]# mmdelnode -N dhcp-os-129-164.slac.stanford.edu<br>mmdelnode: Unable to obtain the GPFS configuration file lock.<br>mmdelnode: GPFS was unable to obtain a lock from node <br>dhcp-os-129-164.slac.stanford.edu.<br>mmdelnode: Command failed. Examine previous error messages to determine
<br>cause.<br><br>despite the fact that it doesn't respond to ping. The mmstartup on<br>the newly reinstalled node fails as in my initial email. I should<br>mention that the two "working" nodes are running 4.2.3.4. The
person<br>who reinstalled the node that won't start up put on 4.2.3.8. I didn't<br>think that was the cause of this problem though and thought I would<br>try to get the cluster talking again before upgrading the rest of the<br>nodes or degrading the reinstalled one.<br><br>Thanks,<br>Renata<br><br><br><br><br>On Wed, 27 Jun 2018, Simon Thompson wrote:<br><br>>Have you tried running mmsdrestore in the reinstalled node to reads
to <br>the cluster and then try and startup gpfs on it?<br>><br>><br></span></tt><a href="https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1pdg_mmsdrrest.htm"><tt><span style=" font-size:10pt">https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1pdg_mmsdrrest.htm</span></tt></a><tt><span style=" font-size:10pt"><br><br>><br>>Simon<br>>________________________________________<br>>From: gpfsug-discuss-bounces@spectrumscale.org <br>[gpfsug-discuss-bounces@spectrumscale.org] on behalf of Renata Maria Dart
<br>[renata@slac.stanford.edu]<br>>Sent: 27 June 2018 19:09<br>>To: gpfsug-discuss@spectrumscale.org<br>>Subject: [gpfsug-discuss] gpfs client cluster, lost quorum, ccr issues<br>><br>>Hi, we have a client cluster of 4 nodes with 3 quorum nodes. One
of the<br>>quorum nodes is no longer in service and the other was reinstalled
with<br>>a newer OS, both without informing the gpfs admins. Gpfs is still<br>>"working" on the two remaining nodes, that is, they continue
to have <br>access<br>>to the gpfs data on the remote clusters. But, I can no longer
get<br>>any gpfs commands to work. On one of the 2 nodes that are still
serving <br>data,<br>><br>>root@ocio-gpu01 ~]# mmlscluster<br>>get file failed: Not enough CCR quorum nodes available (err 809)<br>>gpfsClusterInit: Unexpected error from ccr fget mmsdrfs. Return
code: <br>158<br>>mmlscluster: Command failed. Examine previous error messages to determine
<br>cause.<br>><br>><br>>On the reinstalled node, this fails in the same way:<br>><br>>[root@ocio-gpu02 ccr]# mmstartup<br>>get file failed: Not enough CCR quorum nodes available (err 809)<br>>gpfsClusterInit: Unexpected error from ccr fget mmsdrfs. Return
code: <br>158<br>>mmstartup: Command failed. Examine previous error messages to determine
<br>cause.<br>><br>><br>>I have looked through the users group interchanges but didn't find
<br>anything<br>>that seems to fit this scenario.<br>><br>>Is there a way to salvage this cluster? Can it be done without<br>>shutting gpfs down on the 2 nodes that continue to work?<br>><br>>Thanks for any advice,<br>><br>>Renata Dart<br>>SLAC National Accelerator Lb<br>><br>>_______________________________________________<br>>gpfsug-discuss mailing list<br>>gpfsug-discuss at spectrumscale.org<br>><br></span></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><span style=" font-size:10pt">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></tt></a><tt><span style=" font-size:10pt"><br><br>><br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></span></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><span style=" font-size:10pt">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></tt></a><tt><span style=" font-size:10pt"><br><br><br><br><br><br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></span></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><span style=" font-size:10pt">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></tt></a><tt><span style=" font-size:10pt"><br><br></span></tt><br><br><BR>