<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Cambria;
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I now see that these
</span><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">mmccrmonitor and mmsdrserv daemons
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">are required for the CCR operations to work. This is just not clear in the error output. Even the GPFS 4.2 Problem Determination Guide doesn’t have anything explaining
the “Not enough CCR quorum nodes available” or “Unexpected error from ccr fget mmsdrfs” error messages. Thus there is no clear direction on how to fix this issue from the command output, the man pages, nor the Admin Guides.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:#1F497D">[root@fpia-gpfs-jcsdr01 ~]# man -E ascii mmccr<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:#1F497D">No manual entry for mmccr<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif";color:#1F497D">There isn’t a help for mmccr either, but at least it does print some usage info:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cambria","serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:#1F497D">[root@fpia-gpfs-jcsdr01 ~]# mmccr -h<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:#1F497D">Unknown subcommand: '-h'Usage: mmccr subcommand common-options subcommand-options...<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:#1F497D">Subcommands:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:#1F497D">Setup and Initialization:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New";color:#1F497D">[snip]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I’m still not sure how to start these
</span><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">mmccrmonitor and mmsdrserv
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">daemons without starting GPFS… could you tell me how it would be possible?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks for sharing details about how this all works Marc, I do appreciate your response!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">-Bryan<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> gpfsug-discuss-bounces@spectrumscale.org [mailto:gpfsug-discuss-bounces@spectrumscale.org]
<b>On Behalf Of </b>Marc A Kaplan<br>
<b>Sent:</b> Thursday, July 28, 2016 12:25 PM<br>
<b>To:</b> gpfsug main discussion list<br>
<b>Subject:</b> Re: [gpfsug-discuss] CCR troubles - CCR and mmXXconfig commands fine with mmshutdown<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Based on experiments on my test cluster, I can assure you that you can list and change GPFS configuration parameters with CCR enabled while GPFS
is down.</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">I understand you are having a problem with your cluster, but you are incorrectly disparaging the CCR.</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">In fact you can mmshutdown -a AND kill all GPFS related processes, including mmsdrserv and mmcrmonitor and then issue commands like:</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">mmlscluster, mmlsconfig, mmchconfig</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Those will work correctly and by-the-way re-start mmsdrserv and mmcrmonitor...
</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">(Use command like `ps auxw | grep mm` to find the relevenat processes).</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">But that will not start the main GPFS file manager process mmfsd. GPFS "proper" remains down...</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">For the following commands Linux was "up" on all nodes, but GPFS was shutdown.</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">[root@n2 gpfs-git]# mmgetstate -a</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> Node number Node name GPFS state</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">------------------------------------------</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> 1 n2 down</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> 3 n4 down</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> 4 n5 down</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> 6 n3 down</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">However if a majority of the quorum nodes can not be obtained, you WILL see a sequence of messages like this, after a noticeable "timeout":</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">(For the following test I had three quorum nodes and did a Linux shutdown on two of them...)</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">[root@n2 gpfs-git]# mmlsconfig</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">get file failed: Not enough CCR quorum nodes available (err 809)</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">gpfsClusterInit: Unexpected error from ccr fget mmsdrfs. Return code: 158</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">mmlsconfig: Command failed. Examine previous error messages to determine cause.</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">[root@n2 gpfs-git]# mmchconfig worker1Threads=1022</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">mmchconfig: Unable to obtain the GPFS configuration file lock.</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">mmchconfig: GPFS was unable to obtain a lock from node n2.frozen.</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">mmchconfig: Command failed. Examine previous error messages to determine cause.</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">[root@n2 gpfs-git]# mmgetstate -a</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">get file failed: Not enough CCR quorum nodes available (err 809)</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">gpfsClusterInit: Unexpected error from ccr fget mmsdrfs. Return code: 158</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">mmgetstate: Command failed. Examine previous error messages to determine cause.</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">HMMMM.... notice mmgetstate needs a quorum even to "know" what nodes it should check!</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Then re-starting Linux... So I have two of three quorum nodes active, but GPFS still down...</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">## From n2, login to node n3 that I just rebooted...</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">[root@n2 gpfs-git]# ssh n3</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Last login: Thu Jul 28 09:50:53 2016 from n2.frozen</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">## See if any mm processes are running? ... NOPE!</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">[root@n3 ~]# ps auxw | grep mm</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">ps auxw | grep mm</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">root 3834 0.0 0.0 112640 972 pts/0 S+ 10:12 0:00 grep --color=auto mm</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">## Check the state... notice n4 is powered off...</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">[root@n3 ~]# mmgetstate -a</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">mmgetstate -a</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> Node number Node name GPFS state</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">------------------------------------------</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> 1 n2 down</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> 3 n4 unknown</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> 4 n5 down</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> 6 n3 down</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">## Examine the cluster configuration</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">[root@n3 ~]# mmlscluster</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">mmlscluster</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">GPFS cluster information</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">========================</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> GPFS cluster name: madagascar.frozen</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> GPFS cluster id: 7399668614468035547</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> GPFS UID domain: madagascar.frozen</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> Remote shell command: /usr/bin/ssh</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> Remote file copy command: /usr/bin/scp</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> Repository type: CCR</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">GPFS cluster configuration servers:</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">-----------------------------------</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> Primary server: n2.frozen (not in use)</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> Secondary server: n4.frozen (not in use)</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> Node Daemon node name IP address Admin node name Designation</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">-------------------------------------------------------------------</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> 1 n2.frozen 172.20.0.21 n2.frozen quorum-manager-perfmon</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> 3 n4.frozen 172.20.0.23 n4.frozen quorum-manager-perfmon</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> 4 n5.frozen 172.20.0.24 n5.frozen perfmon</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> 6 n3.frozen 172.20.0.22 n3.frozen quorum-manager-perfmon</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">## notice that mmccrmonitor and mmsdrserv are running but not mmfsd</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">[root@n3 ~]# ps auxw | grep mm</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">ps auxw | grep mm</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">root 3882 0.0 0.0 114376 1720 pts/0 S 10:13 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">root 3954 0.0 0.0 491244 13040 ? Ssl 10:13 0:00 /usr/lpp/mmfs/bin/mmsdrserv 1191 10 10 /var/adm/ras/mmsdrserv.log 128 yes</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">root 4339 0.0 0.0 114376 796 pts/0 S 10:15 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">root 4345 0.0 0.0 112640 972 pts/0 S+ 10:16 0:00 grep --color=auto mm</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">## Now I can mmchconfig ... while GPFS remains down.</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">[root@n3 ~]# mmchconfig worker1Threads=1022</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">mmchconfig worker1Threads=1022</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">mmchconfig: Command successfully completed</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">mmchconfig: Propagating the cluster configuration data to all</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> affected nodes. This is an asynchronous process.</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">[root@n3 ~]# Thu Jul 28 10:18:16 PDT 2016: mmcommon pushSdr_async: mmsdrfs propagation started</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Thu Jul 28 10:18:21 PDT 2016: mmcommon pushSdr_async: mmsdrfs propagation completed; mmdsh rc=0</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">[root@n3 ~]# mmgetstate -a</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">mmgetstate -a</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> Node number Node name GPFS state</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">------------------------------------------</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> 1 n2 down</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> 3 n4 unknown</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> 4 n5 down</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif""> 6 n3 down</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">## Quorum node n4 remains unreachable... But n2 and n3 are running Linux.</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">[root@n3 ~]# ping -c 1 n4</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">ping -c 1 n4</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">PING n4.frozen (172.20.0.23) 56(84) bytes of data.</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">From n3.frozen (172.20.0.22) icmp_seq=1 Destination Host Unreachable</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">--- n4.frozen ping statistics ---</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">[root@n3 ~]# exit</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">exit</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">logout</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Connection to n3 closed.</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">[root@n2 gpfs-git]# ps auwx | grep mm</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">root 3264 0.0 0.0 114376 812 pts/1 S 10:21 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">root 3271 0.0 0.0 112640 980 pts/1 S+ 10:21 0:00 grep --color=auto mm</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">root 31820 0.0 0.0 114376 1728 pts/1 S 09:42 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">root 32058 0.0 0.0 493264 12000 ? Ssl 09:42 0:00 /usr/lpp/mmfs/bin/mmsdrserv 1191 10 10 /var/adm/ras/mmsdrserv.log 1</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">root 32263 0.0 0.0 1700732 17600 ? Sl 09:42 0:00 python /usr/lpp/mmfs/bin/mmsysmon.py</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">[root@n2 gpfs-git]#</span><br>
<br>
<o:p></o:p></p>
</div>
<br>
<hr>
<font face="Arial" color="Gray" size="1"><br>
Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this
email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness
or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial
product.<br>
</font>
</body>
</html>