<font size=2 face="sans-serif">I think the idea is that you should not
need to know the details of how ccr and sdrserv are implemented nor how
they work.</font><br><font size=2 face="sans-serif">At this moment, I don't!</font><br><br><font size=2 face="sans-serif">Literally, I just installed GPFS and
defined my system with mmcrcluster and so forth and "it just works".
</font><br><font size=2 face="sans-serif">As I wrote, just running mmlscluster
or mmlsconfig or similar configuration create, list, change, delete commands
should start up</font><br><font size=2 face="sans-serif">ccr and sdrserv under the covers.</font><br><br><font size=2 face="sans-serif">Okay, now "I hear you" --
it ain't working for you today. Presumably it did a while ago?</font><br><font size=2 face="sans-serif">Let's think about that... Troubleshooting
0,1,2 in order of suspicion...</font><br><br><font size=2 face="sans-serif">0. Check that you can ping and ssh from
each quorum node to every other quorum node. Q*(Q-1) tests</font><br><br><font size=2 face="sans-serif">1. Check that you have plenty of free
space in /var on each quorum node. Hmmm... we're not talking huge,
but</font><br><font size=2 face="sans-serif">see if /var/mmfs/tmp is filled with
junk.... Before and After clearing most of that out I had and have:</font><br><br><font size=2 face="sans-serif">[root@bog-wifi ~]# du -shk /var/mmfs</font><br><font size=2 face="sans-serif">84532 /var/mmfs</font><br><br><font size=2 face="sans-serif">## clean all big and old files out of
/var/mmfs/tmp</font><br><br><font size=2 face="sans-serif">[root@bog-wifi ~]# du -shk /var/mmfs</font><br><font size=2 face="sans-serif">9004 /var/mmfs</font><br><br><font size=2 face="sans-serif">Because we know that /var/mmfs is where
GPFS store configuration "stuff" -</font><br><font size=2 face="sans-serif"><br>2. Check that we have GPFS software correctly installed on each quorum
node:</font><br><br><font size=2 face="sans-serif"> rpm -qa gpfs.* | xargs rpm
--verify</font><br><br><br><img align=left src=cid:_1_104A1ACC104A188C00650DCC85257FFE alt="Marc A Kaplan" style="border:0px solid;"><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Bryan Banister <bbanister@jumptrading.com></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug main discussion
list <gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">07/28/2016 01:58 PM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
CCR troubles - CCR and mmXXconfig commands fine with mmshutdown</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><font size=2 color=#004080 face="Calibri">I now see that these </font><font size=2 face="Arial">mmccrmonitor
and mmsdrserv daemons </font><font size=2 color=#004080 face="Calibri">are
required for the CCR operations to work. This is just not clear in
the error output. Even the GPFS 4.2 Problem Determination Guide doesn’t
have anything explaining the “Not enough CCR quorum nodes available”
or “Unexpected error from ccr fget mmsdrfs” error messages. Thus
there is no clear direction on how to fix this issue from the command output,
the man pages, nor the Admin Guides.</font><br><font size=2 color=#004080 face="Calibri"> </font><br><font size=2 color=#004080 face="Courier New">[root@fpia-gpfs-jcsdr01
~]# man -E ascii mmccr</font><br><font size=2 color=#004080 face="Courier New">No manual entry for mmccr</font><br><font size=2 color=#004080 face="Courier New"> </font><br><font size=2 color=#004080 face="Cambria">There isn’t a help for mmccr
either, but at least it does print some usage info:</font><br><font size=2 color=#004080 face="Cambria"> </font><br><font size=2 color=#004080 face="Courier New">[root@fpia-gpfs-jcsdr01
~]# mmccr -h</font><br><font size=2 color=#004080 face="Courier New">Unknown subcommand: '-h'Usage:
mmccr subcommand common-options subcommand-options...</font><br><font size=2 color=#004080 face="Courier New"> </font><br><font size=2 color=#004080 face="Courier New">Subcommands:</font><br><font size=2 color=#004080 face="Courier New"> </font><br><font size=2 color=#004080 face="Courier New">Setup and Initialization:</font><br><font size=2 color=#004080 face="Courier New">[snip]</font><br><font size=2 color=#004080 face="Calibri"> </font><br><font size=2 color=#004080 face="Calibri">I’m still not sure how to
start these </font><font size=2 face="Arial">mmccrmonitor and mmsdrserv
</font><font size=2 color=#004080 face="Calibri">daemons without starting
GPFS… could you tell me how it would be possible?</font><br><font size=2 color=#004080 face="Calibri"> </font><br><font size=2 color=#004080 face="Calibri">Thanks for sharing details
about how this all works Marc, I do appreciate your response!</font><br><font size=2 color=#004080 face="Calibri">-Bryan</font><br><font size=2 color=#004080 face="Calibri"> </font><br><font size=2 face="Tahoma"><b>From:</b> gpfsug-discuss-bounces@spectrumscale.org
[</font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><font size=2 face="Tahoma">mailto:gpfsug-discuss-bounces@spectrumscale.org</font></a><font size=2 face="Tahoma">]
<b>On Behalf Of </b>Marc A Kaplan<b><br>Sent:</b> Thursday, July 28, 2016 12:25 PM<b><br>To:</b> gpfsug main discussion list<b><br>Subject:</b> Re: [gpfsug-discuss] CCR troubles - CCR and mmXXconfig commands
fine with mmshutdown</font><br><font size=3 face="Times New Roman"> </font><br><font size=2 face="Arial">Based on experiments on my test cluster,
I can assure you that you can list and change GPFS configuration parameters
with CCR enabled while GPFS is down.</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br>I understand you are having a problem with your cluster, but you are incorrectly
disparaging the CCR.</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br>In fact you can mmshutdown -a AND kill all GPFS related processes, including
mmsdrserv and mmcrmonitor and then issue commands like:</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br>mmlscluster, mmlsconfig, mmchconfig</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br>Those will work correctly and by-the-way re-start mmsdrserv and mmcrmonitor...
<br>(Use command like `ps auxw | grep mm` to find the relevenat processes).</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br>But that will not start the main GPFS file manager process mmfsd. GPFS
"proper" remains down...</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br>For the following commands Linux was "up" on all nodes, but GPFS
was shutdown.<br>[root@n2 gpfs-git]# mmgetstate -a</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br> Node number Node name GPFS state<br>------------------------------------------<br> 1 n2
down<br> 3 n4
down<br> 4 n5
down<br> 6 n3
down</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br>However if a majority of the quorum nodes can not be obtained, you WILL
see a sequence of messages like this, after a noticeable "timeout":<br>(For the following test I had three quorum nodes and did a Linux shutdown
on two of them...)</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br>[root@n2 gpfs-git]# mmlsconfig<br>get file failed: Not enough CCR quorum nodes available (err 809)<br>gpfsClusterInit: Unexpected error from ccr fget mmsdrfs. Return code:
158<br>mmlsconfig: Command failed. Examine previous error messages to determine
cause.</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br>[root@n2 gpfs-git]# mmchconfig worker1Threads=1022<br>mmchconfig: Unable to obtain the GPFS configuration file lock.<br>mmchconfig: GPFS was unable to obtain a lock from node n2.frozen.<br>mmchconfig: Command failed. Examine previous error messages to determine
cause.</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br>[root@n2 gpfs-git]# mmgetstate -a<br>get file failed: Not enough CCR quorum nodes available (err 809)<br>gpfsClusterInit: Unexpected error from ccr fget mmsdrfs. Return code:
158<br>mmgetstate: Command failed. Examine previous error messages to determine
cause.</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br>HMMMM.... notice mmgetstate needs a quorum even to "know" what
nodes it should check!</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br>Then re-starting Linux... So I have two of three quorum nodes active, but
GPFS still down...</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br>## From n2, login to node n3 that I just rebooted...<br>[root@n2 gpfs-git]# ssh n3<br>Last login: Thu Jul 28 09:50:53 2016 from n2.frozen</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br>## See if any mm processes are running? ... NOPE!</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br>[root@n3 ~]# ps auxw | grep mm<br>ps auxw | grep mm<br>root 3834 0.0 0.0 112640 972 pts/0
S+ 10:12 0:00 grep --color=auto mm</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br>## Check the state... notice n4 is powered off...<br>[root@n3 ~]# mmgetstate -a<br>mmgetstate -a</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br> Node number Node name GPFS state<br>------------------------------------------<br> 1 n2
down<br> 3 n4
unknown<br> 4 n5
down<br> 6 n3
down</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br>## Examine the cluster configuration<br>[root@n3 ~]# mmlscluster<br>mmlscluster</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br>GPFS cluster information<br>========================<br> GPFS cluster name: madagascar.frozen<br> GPFS cluster id: 7399668614468035547<br> GPFS UID domain: madagascar.frozen<br> Remote shell command: /usr/bin/ssh<br> Remote file copy command: /usr/bin/scp<br> Repository type: CCR</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br>GPFS cluster configuration servers:<br>-----------------------------------<br> Primary server: n2.frozen (not in use)<br> Secondary server: n4.frozen (not in use)</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br> Node Daemon node name IP address Admin node name Designation<br>-------------------------------------------------------------------<br> 1 n2.frozen 172.20.0.21 n2.frozen
quorum-manager-perfmon<br> 3 n4.frozen 172.20.0.23 n4.frozen
quorum-manager-perfmon<br> 4 n5.frozen 172.20.0.24 n5.frozen
perfmon<br> 6 n3.frozen 172.20.0.22 n3.frozen
quorum-manager-perfmon</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br>## notice that mmccrmonitor and mmsdrserv are running but not mmfsd</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br>[root@n3 ~]# ps auxw | grep mm<br>ps auxw | grep mm<br>root 3882 0.0 0.0 114376 1720 pts/0
S 10:13 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor
15<br>root 3954 0.0 0.0 491244 13040 ?
Ssl 10:13 0:00 /usr/lpp/mmfs/bin/mmsdrserv
1191 10 10 /var/adm/ras/mmsdrserv.log 128 yes<br>root 4339 0.0 0.0 114376 796 pts/0
S 10:15 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor
15<br>root 4345 0.0 0.0 112640 972 pts/0
S+ 10:16 0:00 grep --color=auto mm</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br>## Now I can mmchconfig ... while GPFS remains down.</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br>[root@n3 ~]# mmchconfig worker1Threads=1022<br>mmchconfig worker1Threads=1022<br>mmchconfig: Command successfully completed<br>mmchconfig: Propagating the cluster configuration data to all<br> affected nodes. This is an asynchronous process.<br>[root@n3 ~]# Thu Jul 28 10:18:16 PDT 2016: mmcommon pushSdr_async: mmsdrfs
propagation started<br>Thu Jul 28 10:18:21 PDT 2016: mmcommon pushSdr_async: mmsdrfs propagation
completed; mmdsh rc=0</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br>[root@n3 ~]# mmgetstate -a<br>mmgetstate -a</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br> Node number Node name GPFS state<br>------------------------------------------<br> 1 n2
down<br> 3 n4
unknown<br> 4 n5
down<br> 6 n3
down</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br>## Quorum node n4 remains unreachable... But n2 and n3 are running
Linux.<br>[root@n3 ~]# ping -c 1 n4<br>ping -c 1 n4<br>PING n4.frozen (172.20.0.23) 56(84) bytes of data.<br>From n3.frozen (172.20.0.22) icmp_seq=1 Destination Host Unreachable</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br>--- n4.frozen ping statistics ---<br>1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br>[root@n3 ~]# exit<br>exit<br>logout<br>Connection to n3 closed.<br>[root@n2 gpfs-git]# ps auwx | grep mm<br>root 3264 0.0 0.0 114376 812 pts/1
S 10:21 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor
15<br>root 3271 0.0 0.0 112640 980 pts/1
S+ 10:21 0:00 grep --color=auto mm<br>root 31820 0.0 0.0 114376 1728 pts/1
S 09:42 0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor
15<br>root 32058 0.0 0.0 493264 12000 ?
Ssl 09:42 0:00 /usr/lpp/mmfs/bin/mmsdrserv 1191
10 10 /var/adm/ras/mmsdrserv.log 1<br>root 32263 0.0 0.0 1700732 17600 ?
Sl 09:42 0:00 python /usr/lpp/mmfs/bin/mmsysmon.py<br>[root@n2 gpfs-git]#</font><font size=3 face="Times New Roman"><br></font><p><font size=3><br></font><hr><font size=1 color=#808080 face="Arial"><br>Note: This email is for the confidential use of the named addressee(s)
only and may contain proprietary, confidential or privileged information.
If you are not the intended recipient, you are hereby notified that any
review, dissemination or copying of this email is strictly prohibited,
and to please notify the sender immediately and destroy this email and
any attachments. Email transmission cannot be guaranteed to be secure or
error-free. The Company, therefore, does not make any guarantees as to
the completeness or accuracy of this email or any attachments. This email
is for informational purposes only and does not constitute a recommendation,
offer, request or solicitation of any kind to buy, sell, subscribe, redeem
or perform any type of transaction of a financial product.</font><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br></font></tt><p><BR>