<div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif">Can you also check the time differences between nodes?</div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">We had a situation recently where the server time mismatch caused failures.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jun 28, 2018 at 2:50 AM, Kevin D Johnson <span dir="ltr"><<a href="mailto:kevindjo@us.ibm.com" target="_blank">kevindjo@us.ibm.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="m_-9111293705391702119socmaildefaultfont" dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10.5pt"><div dir="ltr">You can also try to convert to the old primary/secondary model to back it away from the default CCR configuration.</div>
<div dir="ltr"> </div>
<div dir="ltr">mmchcluster --ccr-disable -p servername</div>
<div dir="ltr"> </div>
<div dir="ltr">Then, temporarily go with only one quorum node and add more once the cluster comes back up. Once the cluster is back up and has at least two quorum nodes, do a --ccr-enable with the mmchcluster command.</div>
<div dir="ltr"><div class="m_-9111293705391702119socmaildefaultfont" dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10.5pt"><div class="m_-9111293705391702119socmaildefaultfont" dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10.5pt"><div class="m_-9111293705391702119socmaildefaultfont" dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10.5pt"><div class="m_-9111293705391702119socmaildefaultfont" dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10.5pt"><div class="m_-9111293705391702119socmaildefaultfont" dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10.5pt"><div class="m_-9111293705391702119socmaildefaultfont" dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10.5pt"><div class="m_-9111293705391702119socmaildefaultfont" dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10.5pt"><div class="m_-9111293705391702119socmaildefaultfont" dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10.5pt"><div class="m_-9111293705391702119socmaildefaultfont" dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10.5pt"><div class="m_-9111293705391702119socmaildefaultfont" dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10.5pt"><div class="m_-9111293705391702119socmaildefaultfont" dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10.5pt"><div class="m_-9111293705391702119socmaildefaultfont" dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10.5pt"><div class="m_-9111293705391702119socmaildefaultfont" dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10.5pt"><div class="m_-9111293705391702119socmaildefaultfont" dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10.5pt"><div class="m_-9111293705391702119socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt"><div class="m_-9111293705391702119socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt"><div class="m_-9111293705391702119socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt"><div class="m_-9111293705391702119socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt"><div class="m_-9111293705391702119socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt"><div class="m_-9111293705391702119socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt"><div class="m_-9111293705391702119socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt"><div class="m_-9111293705391702119socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt"><div class="m_-9111293705391702119socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt"><div class="m_-9111293705391702119socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt"><div dir="ltr"><div style="font-size:10pt;font-weight:bold;font-family:sans-serif">
<div><table style="border-collapse:collapse;border-color:rgb(105,105,105);table-layout:fixed;width:800px" border="0"> <tbody> <tr> <td> <div><span style="font-size:1.000em">Kevin D. Johnson</span><br> Spectrum Computing, Senior Managing Consultant</div>
<div><span style="color:#808080">MBA, MAcc, MS Global Technology and Development<br> IBM Certified Technical Specialist Level 2 Expert</span><br> </div> </td> <td style="overflow:hidden;width:375px;border-color:#696969"><span style="font-size:0.786em"><a href="https://www.youracclaim.com/badges/69d10078-02df-4e57-a223-bb3c9ae06306" target="_blank"><img alt="IBM Certified Technical Specialist Level 2 Expert" src="https://acclaim-production-app.s3.amazonaws.com/images/7c07ab6c-a0c1-486c-852a-e1782482cc42/Tech-Spec-Level-2.png" style="width:100px;height:100px;float:left"></a></span></td> </tr> <tr> <td style="overflow:hidden;width:375px;border-color:#696969"><span style="color:#808080"><span style="font-size:0.857em">Certified Deployment Professional - Spectrum Scale<br> Certified Solution Advisor - Spectrum Computing<br> Certified Solution Architect - Spectrum Storage Solutions</span></span></td> <td style="overflow:hidden;width:375px;border-color:#696969"> </td> </tr> <tr> <td style="overflow:hidden;width:375px;border-color:#696969"> <div><br> <span style="font-size:0.857em"><span style="color:#808080">720.349.6199 - <a href="mailto:kevindjo@us.ibm.com" target="_blank">kevindjo@us.ibm.com</a></span></span><br> </div>
<div><span style="font-size:0.857em">"To think is to achieve." - Thomas J. Watson, Sr.</span></div> </td> <td style="overflow:hidden;width:375px;border-color:#696969"> </td> </tr> </tbody></table></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div><div><div class="h5">
<div dir="ltr"> </div>
<div dir="ltr"> </div>
<blockquote dir="ltr" style="border-left:solid #aaaaaa 2px;margin-left:5px;padding-left:5px;direction:ltr;margin-right:0px">----- Original message -----<br>From: "IBM Spectrum Scale" <<a href="mailto:scale@us.ibm.com" target="_blank">scale@us.ibm.com</a>><br>Sent by: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@<wbr>spectrumscale.org</a><br>To: <a href="mailto:renata@slac.stanford.edu" target="_blank">renata@slac.stanford.edu</a>, gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.<wbr>org</a>><br>Cc:<br>Subject: Re: [gpfsug-discuss] gpfs client cluster, lost quorum, ccr issues<br>Date: Wed, Jun 27, 2018 5:15 PM<br>
<p><font size="2">Hi Renata,</font><br><br><font size="2">You may want to reduce the set of quorum nodes. If your version supports the --force option, you can run</font><br><br><font size="2">mmchnode --noquorum -N <broken-nodes> --force</font><br><br><font size="2">It is a good idea to configure tiebreaker disks in a cluster that has only 2 quorum nodes.</font><br><br><font size="2">Regards, The Spectrum Scale (GPFS) team<br><br>------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>------------------------<br>If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at <a href="https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479" target="_blank">https://www.ibm.com/<wbr>developerworks/community/<wbr>forums/html/forum?id=11111111-<wbr>0000-0000-0000-000000000479</a>.<br><br>If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries.<br><br>The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team.</font><br><br><img alt="Inactive hide details for Renata Maria Dart ---06/27/2018 02:21:52 PM---Hi, we have a client cluster of 4 nodes with 3 quorum n" src="cid:1__=0ABB082ADFE7DE038f9e8a93df938690918c0AB@" border="0" width="16" height="16"><font size="2" color="#424282">Renata Maria Dart ---06/27/2018 02:21:52 PM---Hi, we have a client cluster of 4 nodes with 3 quorum nodes. One of the quorum nodes is no longer i</font><br><br><font size="2" color="#5F5F5F">From: </font><font size="2">Renata Maria Dart <<a href="mailto:renata@slac.stanford.edu" target="_blank">renata@slac.stanford.edu</a>></font><br><font size="2" color="#5F5F5F">To: </font><font size="2"><a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.<wbr>org</a></font><br><font size="2" color="#5F5F5F">Date: </font><font size="2">06/27/2018 02:21 PM</font><br><font size="2" color="#5F5F5F">Subject: </font><font size="2">[gpfsug-discuss] gpfs client cluster, lost quorum, ccr issues</font><br><font size="2" color="#5F5F5F">Sent by: </font><font size="2"><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@<wbr>spectrumscale.org</a></font></p>
<hr style="color:#8091a5" size="2" align="left" width="100%"><br><br><br><tt><font size="3" face="">Hi, we have a client cluster of 4 nodes with 3 quorum nodes. One of the<br>quorum nodes is no longer in service and the other was reinstalled with<br>a newer OS, both without informing the gpfs admins. Gpfs is still<br>"working" on the two remaining nodes, that is, they continue to have access<br>to the gpfs data on the remote clusters. But, I can no longer get<br>any gpfs commands to work. On one of the 2 nodes that are still serving data,<br><br>root@ocio-gpu01 ~]# mmlscluster<br>get file failed: Not enough CCR quorum nodes available (err 809)<br>gpfsClusterInit: Unexpected error from ccr fget mmsdrfs. Return code: 158<br>mmlscluster: Command failed. Examine previous error messages to determine cause.<br><br><br>On the reinstalled node, this fails in the same way:<br><br>[root@ocio-gpu02 ccr]# mmstartup<br>get file failed: Not enough CCR quorum nodes available (err 809)<br>gpfsClusterInit: Unexpected error from ccr fget mmsdrfs. Return code: 158<br>mmstartup: Command failed. Examine previous error messages to determine cause.<br><br><br>I have looked through the users group interchanges but didn't find anything<br>that seems to fit this scenario.<br><br>Is there a way to salvage this cluster? Can it be done without<br>shutting gpfs down on the 2 nodes that continue to work?<br><br>Thanks for any advice,<br><br>Renata Dart<br>SLAC National Accelerator Lb<br><br>______________________________<wbr>_________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a></font></tt><br><tt><font size="3" face=""><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</a></font></tt><br><br><br><br>
<div><font size="2" face="Default Monospace,Courier New,Courier,monospace">______________________________<wbr>_________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</a></font></div></blockquote>
<div dir="ltr"> </div></div></div></div>
<br>
<br>______________________________<wbr>_________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</a><br>
<br></blockquote></div><br></div>