<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div dir="ltr" >You can also try to convert to the old primary/secondary model to back it away from the default CCR configuration.</div>
<div dir="ltr" > </div>
<div dir="ltr" >mmchcluster --ccr-disable -p servername</div>
<div dir="ltr" > </div>
<div dir="ltr" >Then, temporarily go with only one quorum node and add more once the cluster comes back up. Once the cluster is back up and has at least two quorum nodes, do a --ccr-enable with the mmchcluster command.</div>
<div dir="ltr" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div dir="ltr" ><div style="font-size: 10pt; font-weight: bold; font-family: sans-serif;" >
<div><table style="border-collapse: collapse; border-color: rgb(105, 105, 105); overflow-wrap: break-word; table-layout: fixed; width: 800px;" border="0" > <tbody> <tr> <td> <div><span style="font-size:1.000em;" >Kevin D. Johnson</span><br> Spectrum Computing, Senior Managing Consultant</div>
<div><span style="color:#808080;" >MBA, MAcc, MS Global Technology and Development<br> IBM Certified Technical Specialist Level 2 Expert</span><br> </div> </td> <td style="overflow: hidden; width: 375px; border-color : #696969;" ><span style="font-size: 0.786em;" ><a href="https://www.youracclaim.com/badges/69d10078-02df-4e57-a223-bb3c9ae06306" target="_blank"><img alt="IBM Certified Technical Specialist Level 2 Expert" src="https://acclaim-production-app.s3.amazonaws.com/images/7c07ab6c-a0c1-486c-852a-e1782482cc42/Tech-Spec-Level-2.png" style="width: 100px; height: 100px; float: left;" ></a></span></td> </tr> <tr> <td style="overflow: hidden; width: 375px; border-color : #696969;" ><span style="color:#808080;" ><span style="font-size: 0.857em;" >Certified Deployment Professional - Spectrum Scale<br> Certified Solution Advisor - Spectrum Computing<br> Certified Solution Architect - Spectrum Storage Solutions</span></span></td> <td style="overflow: hidden; width: 375px; border-color : #696969;" > </td> </tr> <tr> <td style="overflow: hidden; width: 375px; border-color : #696969;" > <div><br> <span style="font-size:0.857em;" ><span style="color:#808080;" >720.349.6199 - kevindjo@us.ibm.com</span></span><br> </div>
<div><span style="font-size:0.857em;" >"To think is to achieve." - Thomas J. Watson, Sr.</span></div> </td> <td style="overflow: hidden; width: 375px; border-color : #696969;" > </td> </tr> </tbody></table></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Original message -----<br>From: "IBM Spectrum Scale" <scale@us.ibm.com><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: renata@slac.stanford.edu, gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Cc:<br>Subject: Re: [gpfsug-discuss] gpfs client cluster, lost quorum, ccr issues<br>Date: Wed, Jun 27, 2018 5:15 PM<br>
<p><font size="2" >Hi Renata,</font><br><br><font size="2" >You may want to reduce the set of quorum nodes. If your version supports the --force option, you can run</font><br><br><font size="2" >mmchnode --noquorum -N <broken-nodes> --force</font><br><br><font size="2" >It is a good idea to configure tiebreaker disks in a cluster that has only 2 quorum nodes.</font><br><br><font size="2" >Regards, The Spectrum Scale (GPFS) team<br><br>------------------------------------------------------------------------------------------------------------------<br>If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at <a href="https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479" target="_blank">https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479</a>.<br><br>If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries.<br><br>The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team.</font><br><br><img alt="Inactive hide details for Renata Maria Dart ---06/27/2018 02:21:52 PM---Hi, we have a client cluster of 4 nodes with 3 quorum n" src="cid:1__=0ABB082ADFE7DE038f9e8a93df938690918c0AB@" border="0" width="16" height="16" ><font size="2" color="#424282" >Renata Maria Dart ---06/27/2018 02:21:52 PM---Hi, we have a client cluster of 4 nodes with 3 quorum nodes. One of the quorum nodes is no longer i</font><br><br><font size="2" color="#5F5F5F" >From: </font><font size="2" >Renata Maria Dart <renata@slac.stanford.edu></font><br><font size="2" color="#5F5F5F" >To: </font><font size="2" >gpfsug-discuss@spectrumscale.org</font><br><font size="2" color="#5F5F5F" >Date: </font><font size="2" >06/27/2018 02:21 PM</font><br><font size="2" color="#5F5F5F" >Subject: </font><font size="2" >[gpfsug-discuss] gpfs client cluster, lost quorum, ccr issues</font><br><font size="2" color="#5F5F5F" >Sent by: </font><font size="2" >gpfsug-discuss-bounces@spectrumscale.org</font></p>
<hr style="color:#8091A5; " size="2" align="left" width="100%" ><br><br><br><tt><font size="3" face="" >Hi, we have a client cluster of 4 nodes with 3 quorum nodes. One of the<br>quorum nodes is no longer in service and the other was reinstalled with<br>a newer OS, both without informing the gpfs admins. Gpfs is still<br>"working" on the two remaining nodes, that is, they continue to have access<br>to the gpfs data on the remote clusters. But, I can no longer get<br>any gpfs commands to work. On one of the 2 nodes that are still serving data,<br><br>root@ocio-gpu01 ~]# mmlscluster<br>get file failed: Not enough CCR quorum nodes available (err 809)<br>gpfsClusterInit: Unexpected error from ccr fget mmsdrfs. Return code: 158<br>mmlscluster: Command failed. Examine previous error messages to determine cause.<br><br><br>On the reinstalled node, this fails in the same way:<br><br>[root@ocio-gpu02 ccr]# mmstartup<br>get file failed: Not enough CCR quorum nodes available (err 809)<br>gpfsClusterInit: Unexpected error from ccr fget mmsdrfs. Return code: 158<br>mmstartup: Command failed. Examine previous error messages to determine cause.<br><br><br>I have looked through the users group interchanges but didn't find anything<br>that seems to fit this scenario.<br><br>Is there a way to salvage this cluster? Can it be done without<br>shutting gpfs down on the 2 nodes that continue to work?<br><br>Thanks for any advice,<br><br>Renata Dart<br>SLAC National Accelerator Lb<br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org</font></tt><br><tt><font size="3" face="" ><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></tt><br><br><br><br>
<div><font size="2" face="Default Monospace,Courier New,Courier,monospace" >_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></div></blockquote>
<div dir="ltr" > </div></div>
<BR>