<font size=2 face="sans-serif">Based on experiments on my test cluster,
I can assure you that you can list and change GPFS configuration parameters
with CCR enabled while GPFS is down.</font><br><br><font size=2 face="sans-serif">I understand you are having a problem
with your cluster, but you are incorrectly disparaging the CCR.</font><br><br><font size=2 face="sans-serif">In fact you can mmshutdown -a AND kill
all GPFS related processes, including mmsdrserv and mmcrmonitor and then
issue commands like:</font><br><br><font size=2 face="sans-serif">mmlscluster, mmlsconfig, mmchconfig</font><br><br><font size=2 face="sans-serif">Those will work correctly and by-the-way
re-start mmsdrserv and mmcrmonitor... </font><br><font size=2 face="sans-serif">(Use command like `ps auxw | grep mm`
to find the relevenat processes).</font><br><br><font size=2 face="sans-serif">But that will not start the main GPFS
file manager process mmfsd. GPFS "proper" remains down...</font><br><br><font size=2 face="sans-serif">For the following commands Linux was
"up" on all nodes, but GPFS was shutdown.</font><br><font size=2 face="sans-serif">[root@n2 gpfs-git]# mmgetstate -a</font><br><br><font size=2 face="sans-serif"> Node number Node name
GPFS state</font><br><font size=2 face="sans-serif">------------------------------------------</font><br><font size=2 face="sans-serif"> 1
n2 down</font><br><font size=2 face="sans-serif"> 3
n4 down</font><br><font size=2 face="sans-serif"> 4
n5 down</font><br><font size=2 face="sans-serif"> 6
n3 down</font><br><br><font size=2 face="sans-serif">However if a majority of the quorum
nodes can not be obtained, you WILL see a sequence of messages like this,
after a noticeable "timeout":</font><br><font size=2 face="sans-serif">(For the following test I had three
quorum nodes and did a Linux shutdown on two of them...)</font><br><br><font size=2 face="sans-serif">[root@n2 gpfs-git]# mmlsconfig</font><br><font size=2 face="sans-serif">get file failed: Not enough CCR quorum
nodes available (err 809)</font><br><font size=2 face="sans-serif">gpfsClusterInit: Unexpected error from
ccr fget mmsdrfs. Return code: 158</font><br><font size=2 face="sans-serif">mmlsconfig: Command failed. Examine
previous error messages to determine cause.</font><br><br><font size=2 face="sans-serif">[root@n2 gpfs-git]# mmchconfig worker1Threads=1022</font><br><font size=2 face="sans-serif">mmchconfig: Unable to obtain the GPFS
configuration file lock.</font><br><font size=2 face="sans-serif">mmchconfig: GPFS was unable to obtain
a lock from node n2.frozen.</font><br><font size=2 face="sans-serif">mmchconfig: Command failed. Examine
previous error messages to determine cause.</font><br><br><font size=2 face="sans-serif">[root@n2 gpfs-git]# mmgetstate -a</font><br><font size=2 face="sans-serif">get file failed: Not enough CCR quorum
nodes available (err 809)</font><br><font size=2 face="sans-serif">gpfsClusterInit: Unexpected error from
ccr fget mmsdrfs. Return code: 158</font><br><font size=2 face="sans-serif">mmgetstate: Command failed. Examine
previous error messages to determine cause.</font><br><br><font size=2 face="sans-serif">HMMMM.... notice mmgetstate needs a
quorum even to "know" what nodes it should check!</font><br><br><font size=2 face="sans-serif">Then re-starting Linux... So I have
two of three quorum nodes active, but GPFS still down...</font><br><br><font size=2 face="sans-serif">## From n2, login to node n3 that
I just rebooted...</font><br><font size=2 face="sans-serif">[root@n2 gpfs-git]# ssh n3</font><br><font size=2 face="sans-serif">Last login: Thu Jul 28 09:50:53 2016
from n2.frozen</font><br><br><font size=2 face="sans-serif">## See if any mm processes are running?
... NOPE!</font><br><br><font size=2 face="sans-serif">[root@n3 ~]# ps auxw | grep mm</font><br><font size=2 face="sans-serif">ps auxw | grep mm</font><br><font size=2 face="sans-serif">root 3834 0.0
0.0 112640 972 pts/0 S+ 10:12 0:00
grep --color=auto mm</font><br><br><font size=2 face="sans-serif">## Check the state... notice n4
is powered off...</font><br><font size=2 face="sans-serif">[root@n3 ~]# mmgetstate -a</font><br><font size=2 face="sans-serif">mmgetstate -a</font><br><br><font size=2 face="sans-serif"> Node number Node name
GPFS state</font><br><font size=2 face="sans-serif">------------------------------------------</font><br><font size=2 face="sans-serif"> 1
n2 down</font><br><font size=2 face="sans-serif"> 3
n4 unknown</font><br><font size=2 face="sans-serif"> 4
n5 down</font><br><font size=2 face="sans-serif"> 6
n3 down</font><br><br><font size=2 face="sans-serif">## Examine the cluster configuration</font><br><font size=2 face="sans-serif">[root@n3 ~]# mmlscluster</font><br><font size=2 face="sans-serif">mmlscluster</font><br><br><font size=2 face="sans-serif">GPFS cluster information</font><br><font size=2 face="sans-serif">========================</font><br><font size=2 face="sans-serif"> GPFS cluster name:
madagascar.frozen</font><br><font size=2 face="sans-serif"> GPFS cluster id:
7399668614468035547</font><br><font size=2 face="sans-serif"> GPFS UID domain:
madagascar.frozen</font><br><font size=2 face="sans-serif"> Remote shell command:
/usr/bin/ssh</font><br><font size=2 face="sans-serif"> Remote file copy command: /usr/bin/scp</font><br><font size=2 face="sans-serif"> Repository type:
CCR</font><br><br><font size=2 face="sans-serif">GPFS cluster configuration servers:</font><br><font size=2 face="sans-serif">-----------------------------------</font><br><font size=2 face="sans-serif"> Primary server: n2.frozen
(not in use)</font><br><font size=2 face="sans-serif"> Secondary server: n4.frozen
(not in use)</font><br><br><font size=2 face="sans-serif"> Node Daemon node name IP
address Admin node name Designation</font><br><font size=2 face="sans-serif">-------------------------------------------------------------------</font><br><font size=2 face="sans-serif"> 1 n2.frozen
172.20.0.21 n2.frozen quorum-manager-perfmon</font><br><font size=2 face="sans-serif"> 3 n4.frozen
172.20.0.23 n4.frozen quorum-manager-perfmon</font><br><font size=2 face="sans-serif"> 4 n5.frozen
172.20.0.24 n5.frozen perfmon</font><br><font size=2 face="sans-serif"> 6 n3.frozen
172.20.0.22 n3.frozen quorum-manager-perfmon</font><br><br><font size=2 face="sans-serif">## notice that mmccrmonitor and mmsdrserv
are running but not mmfsd</font><br><br><font size=2 face="sans-serif">[root@n3 ~]# ps auxw | grep mm</font><br><font size=2 face="sans-serif">ps auxw | grep mm</font><br><font size=2 face="sans-serif">root 3882 0.0
0.0 114376 1720 pts/0 S 10:13
0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15</font><br><font size=2 face="sans-serif">root 3954 0.0
0.0 491244 13040 ? Ssl 10:13
0:00 /usr/lpp/mmfs/bin/mmsdrserv 1191 10 10 /var/adm/ras/mmsdrserv.log
128 yes</font><br><font size=2 face="sans-serif">root 4339 0.0
0.0 114376 796 pts/0 S 10:15
0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15</font><br><font size=2 face="sans-serif">root 4345 0.0
0.0 112640 972 pts/0 S+ 10:16 0:00
grep --color=auto mm</font><br><br><font size=2 face="sans-serif">## Now I can mmchconfig ... while GPFS
remains down.</font><br><br><font size=2 face="sans-serif">[root@n3 ~]# mmchconfig worker1Threads=1022</font><br><font size=2 face="sans-serif">mmchconfig worker1Threads=1022</font><br><font size=2 face="sans-serif">mmchconfig: Command successfully completed</font><br><font size=2 face="sans-serif">mmchconfig: Propagating the cluster
configuration data to all</font><br><font size=2 face="sans-serif"> affected nodes. This is
an asynchronous process.</font><br><font size=2 face="sans-serif">[root@n3 ~]# Thu Jul 28 10:18:16 PDT
2016: mmcommon pushSdr_async: mmsdrfs propagation started</font><br><font size=2 face="sans-serif">Thu Jul 28 10:18:21 PDT 2016: mmcommon
pushSdr_async: mmsdrfs propagation completed; mmdsh rc=0</font><br><br><font size=2 face="sans-serif">[root@n3 ~]# mmgetstate -a</font><br><font size=2 face="sans-serif">mmgetstate -a</font><br><br><font size=2 face="sans-serif"> Node number Node name
GPFS state</font><br><font size=2 face="sans-serif">------------------------------------------</font><br><font size=2 face="sans-serif"> 1
n2 down</font><br><font size=2 face="sans-serif"> 3
n4 unknown</font><br><font size=2 face="sans-serif"> 4
n5 down</font><br><font size=2 face="sans-serif"> 6
n3 down</font><br><br><font size=2 face="sans-serif">## Quorum node n4 remains unreachable...
But n2 and n3 are running Linux.</font><br><font size=2 face="sans-serif">[root@n3 ~]# ping -c 1 n4</font><br><font size=2 face="sans-serif">ping -c 1 n4</font><br><font size=2 face="sans-serif">PING n4.frozen (172.20.0.23) 56(84)
bytes of data.</font><br><font size=2 face="sans-serif">From n3.frozen (172.20.0.22) icmp_seq=1
Destination Host Unreachable</font><br><br><font size=2 face="sans-serif">--- n4.frozen ping statistics ---</font><br><font size=2 face="sans-serif">1 packets transmitted, 0 received, +1
errors, 100% packet loss, time 0ms</font><br><br><font size=2 face="sans-serif">[root@n3 ~]# exit</font><br><font size=2 face="sans-serif">exit</font><br><font size=2 face="sans-serif">logout</font><br><font size=2 face="sans-serif">Connection to n3 closed.</font><br><font size=2 face="sans-serif">[root@n2 gpfs-git]# ps auwx | grep mm</font><br><font size=2 face="sans-serif">root 3264 0.0
0.0 114376 812 pts/1 S 10:21
0:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15</font><br><font size=2 face="sans-serif">root 3271 0.0
0.0 112640 980 pts/1 S+ 10:21 0:00
grep --color=auto mm</font><br><font size=2 face="sans-serif">root 31820 0.0 0.0
114376 1728 pts/1 S 09:42 0:00 /usr/lpp/mmfs/bin/mmksh
/usr/lpp/mmfs/bin/mmccrmonitor 15</font><br><font size=2 face="sans-serif">root 32058 0.0 0.0
493264 12000 ? Ssl 09:42 0:00 /usr/lpp/mmfs/bin/mmsdrserv
1191 10 10 /var/adm/ras/mmsdrserv.log 1</font><br><font size=2 face="sans-serif">root 32263 0.0 0.0
1700732 17600 ? Sl 09:42 0:00 python
/usr/lpp/mmfs/bin/mmsysmon.py</font><br><font size=2 face="sans-serif">[root@n2 gpfs-git]#</font><br><br><BR>