<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
tt
{mso-style-priority:99;
font-family:"Courier New";}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Right, I know that I can disable CCR, and I’m asking if this seemingly broken behavior of GPFS commands when the cluster is down was the expected mode of operation
with CCR enabled. Sounds like it from the responses thus far.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">-Bryan<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> gpfsug-discuss-bounces@spectrumscale.org [mailto:gpfsug-discuss-bounces@spectrumscale.org]
<b>On Behalf Of </b>McPheeters, Gordon<br>
<b>Sent:</b> Wednesday, July 27, 2016 5:35 PM<br>
<b>To:</b> gpfsug main discussion list<br>
<b>Subject:</b> Re: [gpfsug-discuss] CCR troubles<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">mmchcluster has an option: <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Cambria Math","serif"">‐‐</span>ccr<span style="font-family:"Cambria Math","serif"">‐</span>disable<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> Reverts to the traditional primary or backup<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> configuration server semantics and destroys the CCR<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> environment. All nodes must be shut down before<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> disabling CCR.<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif";color:black">-Gordon<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Jul 27, 2016, at 5:29 PM, Bryan Banister <<a href="mailto:bbanister@jumptrading.com">bbanister@jumptrading.com</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hi Marc,</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I do understand the principal you describe. The quorum nodes are accessible over TCP/IP but GPFS happens to be down. I think that CCR should would work regardless
of whether GPFS is up or down, so that you can change the configuration on a down cluster. I could even imagine a scenario where a config parameter was set incorrectly and prevents GPFS from starting at all. If you have to have GPFS up to make config changes
because of CCR then how can you fix this issue?</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks for the response!</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">-Bryan</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span class="apple-converted-space"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> </span></span><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><span style="color:purple">gpfsug-discuss-bounces@spectrumscale.org</span></a><span class="apple-converted-space"> </span>[<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><span style="color:purple">mailto:gpfsug-discuss-bounces@spectrumscale.org</span></a>]<span class="apple-converted-space"> </span><b>On
Behalf Of<span class="apple-converted-space"> </span></b>Marc A Kaplan<br>
<b>Sent:</b><span class="apple-converted-space"> </span>Wednesday, July 27, 2016 1:03 PM<br>
<b>To:</b><span class="apple-converted-space"> </span>gpfsug main discussion list<br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re: [gpfsug-discuss] CCR troubles</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">I understand you are having problems with your cluster, but you do NOT need to have GPFS "started" to<span class="apple-converted-space"> </span></span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">display and/or change configuration paramters. You do need at least a majority of the nodes to be up and in communcation (e.g. can talk to each other by tcp/ip)</span><br>
<br>
<tt><b><span style="font-size:7.5pt">--ccr-enable</span></b></tt><br>
<span style="font-size:7.5pt;font-family:"Arial","sans-serif"">Enables the configuration server repository (CCR), which stores redundant copies of configuration data files on all quorum nodes. The advantage of CCR over the traditional primary or backup configuration
server semantics is that when using CCR, all GPFS administration commands as well as file system mounts and daemon startups work normally as long as a majority of quorum nodes are accessible.</span><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><br>
</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Think about how this must work (I have the advantage of actually NOT knowing the details, but one can reason...)</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">to maintain a consistent single configuration database, a majority of quorum nodes MUST agree on every bit of data in the configuration database.</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Even to query the database and get a correct answer, you'd have to know that a majority agree on the answer.</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">(You could ask 1 guy, but then how would you know if he was telling you what the majority opinion is? The minority need not lie to mislead you,<span class="apple-converted-space"> </span></span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">I don't think CCR guards against Byzantine failures...</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">The minority guy could just be out of touch for a while...)</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">I advise that you do some testing on a test cluster (could be virtual)... </span><br>
<br>
<br>
<br>
<span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">From: </span><span style="font-size:7.5pt;font-family:"Arial","sans-serif"">Bryan Banister <<a href="mailto:bbanister@jumptrading.com"><span style="color:purple">bbanister@jumptrading.com</span></a>></span><br>
<span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">To: </span><span style="font-size:7.5pt;font-family:"Arial","sans-serif"">"gpfsug main discussion list (<a href="mailto:gpfsug-discuss@spectrumscale.org"><span style="color:purple">gpfsug-discuss@spectrumscale.org</span></a>)"
<<a href="mailto:gpfsug-discuss@spectrumscale.org"><span style="color:purple">gpfsug-discuss@spectrumscale.org</span></a>></span><br>
<span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">Date: </span><span style="font-size:7.5pt;font-family:"Arial","sans-serif"">07/27/2016 01:37 PM</span><br>
<span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">Subject: </span><span style="font-size:7.5pt;font-family:"Arial","sans-serif"">[gpfsug-discuss] CCR troubles</span><br>
<span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">Sent by: </span><span style="font-size:7.5pt;font-family:"Arial","sans-serif""><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><span style="color:purple">gpfsug-discuss-bounces@spectrumscale.org</span></a></span><o:p></o:p></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" noshade="" style="color:#A0A0A0" align="center">
</div>
<div>
<p class="MsoNormal"><br>
<br>
<br>
<span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">When I have the GPFS cluster down, some GPFS commands no longer work like they should, or at least they did work without CCR:</span><br>
<span style="font-size:10.0pt;font-family:"Calibri","sans-serif""> </span><br>
<span style="font-size:10.0pt;font-family:"Courier New""># mmgetstate -aL # Which stalls for a really stupid amount of time and then spits out:</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">get file failed: Not enough CCR quorum nodes available (err 809)</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">gpfsClusterInit: Unexpected error from ccr fget mmsdrfs. Return code: 158</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">mmgetstate: Command failed. Examine previous error messages to determine cause.</span><br>
<span style="font-size:10.0pt;font-family:"Calibri","sans-serif""> </span><br>
<span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">And trying to change tuning parameters now also barfs when GPFS is down:</span><br>
<span style="font-size:10.0pt;font-family:"Courier New""># [root@fpia-gpfs-jcsdr01 ~]# mmlsconfig</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">get file failed: Not enough CCR quorum nodes available (err 809)</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">gpfsClusterInit: Unexpected error from ccr fget mmsdrfs. Return code: 158</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">mmlsconfig: Command failed. Examine previous error messages to determine cause.</span><br>
<span style="font-size:10.0pt;font-family:"Courier New""> </span><br>
<span style="font-size:10.0pt;font-family:"Courier New""># mmchconfig worker1Threads=128,prefetchThreads=128</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">mmchconfig: Unable to obtain the GPFS configuration file lock.</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">mmchconfig: GPFS was unable to obtain a lock from node<span class="apple-converted-space"> </span><a href="http://fpia-gpfs-jcsdr01.grid.jumptrading.com/"><span style="color:purple">fpia-gpfs-jcsdr01.grid.jumptrading.com</span></a>.</span><br>
<span style="font-size:10.0pt;font-family:"Courier New"">mmchconfig: Command failed. Examine previous error messages to determine cause.</span><br>
<span style="font-size:10.0pt;font-family:"Calibri","sans-serif""> </span><br>
<span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">Which means I will have to start GPFS, change the parameter, shut GPFS down again, and start GPFS up again just to get the new setting.</span><br>
<span style="font-size:10.0pt;font-family:"Calibri","sans-serif""> </span><br>
<span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">Is this really the new mode of operation for CCR enabled clusters?</span><br>
<span style="font-size:10.0pt;font-family:"Calibri","sans-serif""> </span><br>
<span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">I searched CCR in the Concepts, Planning, and Install Guide and also the Adv. Admin Guide, with explanation. </span><br>
<span style="font-size:10.0pt;font-family:"Calibri","sans-serif""> </span><br>
<span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">If so, then maybe I’ll go back to non CCR,</span><br>
<span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">-Bryan</span><o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" align="center">
</div>
<div>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:gray"><br>
Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this
email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness
or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial
product.</span><tt><span style="font-size:10.0pt">_______________________________________________</span></tt><span style="font-size:10.0pt;font-family:"Courier New""><br>
</span><tt><span style="font-size:10.0pt">gpfsug-discuss mailing list</span></tt><span style="font-size:10.0pt;font-family:"Courier New""><br>
</span><tt><span style="font-size:10.0pt">gpfsug-discuss at</span></tt><span class="apple-converted-space"><span style="font-size:10.0pt;font-family:"Courier New""> </span></span><tt><span style="font-size:10.0pt"><a href="http://spectrumscale.org/"><span style="color:purple">spectrumscale.org</span></a></span></tt><span style="font-size:10.0pt;font-family:"Courier New""><br>
</span><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><span style="font-size:10.0pt;color:purple">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></tt></a><o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" align="center">
</div>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:gray"><br>
Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this
email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness
or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial
product.<br>
</span><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif"">_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at<span class="apple-converted-space"> </span></span><a href="http://spectrumscale.org/"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif";color:purple">spectrumscale.org</span></a><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif""><br>
</span><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif";color:purple">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></a><o:p></o:p></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<br>
<hr>
<font face="Arial" color="Gray" size="1"><br>
Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this
email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness
or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial
product.<br>
</font>
</body>
</html>