<font size=2 face="sans-serif">Bryan , </font><br><br><font size=2 face="sans-serif">there some limited configuration changes,
which are not possible with CCR enabled , just like changing a node's daemon
interface </font><br><font size=2 face="sans-serif">but changes, like you described
(workerthreads and so on) .. , are all possible with CCR enabled</font><br><br><font size=2 face="sans-serif">being "blind" (no details
about your current setup) ... it is hard to tell what's wrong in your case,
but these kind of configuration changes must work with CCR enabled
and nodes being down. </font><br><font size=2 face="sans-serif">...and like Marc already stated out,
the communication between the nodes needed to be working..and just check,
that on every node, you can see at least the following process - for
e.g. </font><br><br><font size=2 face="sans-serif">/usr/lpp/mmfs/bin/mmsdrserv 1191 10
10 /var/adm/ras/mmsdrserv.log 128 no</font><br><font size=2 face="sans-serif">/usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor
15</font><br><br><div><font size=2 face="sans-serif">Mit freundlichen Gr¨¹ßen / Kind regards</font><br><br><font size=2 face="sans-serif"> <br>Olaf Weiser<br> <br>EMEA Storage Competence Center Mainz, German / IBM Systems, Storage Platform,<br>-------------------------------------------------------------------------------------------------------------------------------------------<br>IBM Deutschland<br>IBM Allee 1<br>71139 Ehningen<br>Phone: +49-170-579-44-66<br>E-Mail: olaf.weiser@de.ibm.com<br>-------------------------------------------------------------------------------------------------------------------------------------------<br>IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter<br>Geschäftsf¨¹hrung: Martina Koederitz (Vorsitzende), Susanne Peter, Norbert
Janzen, Dr. Christian Keller, Ivo Koerner, Markus Koerner<br>Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart,
HRB 14562 / WEEE-Reg.-Nr. DE 99369940 </font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Bryan Banister <bbanister@jumptrading.com></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug main discussion
list <gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">07/28/2016 12:44 AM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
CCR troubles</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><font size=2 color=#004080 face="Calibri">Right, I know that I can
disable CCR, and I¡¯m asking if this seemingly broken behavior of GPFS
commands when the cluster is down was the expected mode of operation with
CCR enabled. Sounds like it from the responses thus far.</font><br><font size=2 color=#004080 face="Calibri">-Bryan</font><br><font size=2 color=#004080 face="Calibri"> </font><br><font size=2 face="Tahoma"><b>From:</b> gpfsug-discuss-bounces@spectrumscale.org
[</font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><font size=2 face="Tahoma">mailto:gpfsug-discuss-bounces@spectrumscale.org</font></a><font size=2 face="Tahoma">]
<b>On Behalf Of </b>McPheeters, Gordon<b><br>Sent:</b> Wednesday, July 27, 2016 5:35 PM<b><br>To:</b> gpfsug main discussion list<b><br>Subject:</b> Re: [gpfsug-discuss] CCR troubles</font><br><font size=3 face="Times New Roman"> </font><br><font size=3 face="Times New Roman">mmchcluster has an option: </font><br><font size=3>©\©\</font><font size=3 face="Times New Roman">ccr</font><font size=3>©\</font><font size=3 face="Times New Roman">disable</font><br><font size=3 face="Times New Roman"> Reverts
to the traditional primary or backup</font><br><font size=3 face="Times New Roman"> configuration
server semantics and destroys the CCR</font><br><font size=3 face="Times New Roman"> environment.
All nodes must be shut down before</font><br><font size=3 face="Times New Roman"> disabling
CCR.</font><br><font size=1 face="Helvetica"> </font><br><font size=1 face="Helvetica">-Gordon</font><br><font size=3 face="Times New Roman"> </font><br><font size=3 face="Times New Roman"> </font><br><font size=3 face="Times New Roman">On Jul 27, 2016, at 5:29 PM, Bryan
Banister <</font><a href=mailto:bbanister@jumptrading.com><font size=3 color=blue face="Times New Roman"><u>bbanister@jumptrading.com</u></font></a><font size=3 face="Times New Roman">>
wrote:</font><br><font size=3 face="Times New Roman"> </font><br><font size=2 color=#004080 face="Calibri">Hi Marc,</font><br><font size=2 color=#004080 face="Calibri"> </font><br><font size=2 color=#004080 face="Calibri">I do understand the principal
you describe. The quorum nodes are accessible over TCP/IP but GPFS
happens to be down. I think that CCR should would work regardless
of whether GPFS is up or down, so that you can change the configuration
on a down cluster. I could even imagine a scenario where a config
parameter was set incorrectly and prevents GPFS from starting at all. If
you have to have GPFS up to make config changes because of CCR then how
can you fix this issue?</font><br><font size=2 color=#004080 face="Calibri"> </font><br><font size=2 color=#004080 face="Calibri">Thanks for the response!</font><br><font size=2 color=#004080 face="Calibri">-Bryan</font><br><font size=2 color=#004080 face="Calibri"> </font><br><font size=2 face="Tahoma"><b>From:</b> </font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><font size=2 color=#800080 face="Tahoma"><u>gpfsug-discuss-bounces@spectrumscale.org</u></font></a><font size=2 face="Tahoma">[</font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><font size=2 color=#800080 face="Tahoma"><u>mailto:gpfsug-discuss-bounces@spectrumscale.org</u></font></a><font size=2 face="Tahoma">]
<b>On Behalf Of </b>Marc A Kaplan<b><br>Sent:</b> Wednesday, July 27, 2016 1:03 PM<b><br>To:</b> gpfsug main discussion list<b><br>Subject:</b> Re: [gpfsug-discuss] CCR troubles</font><br><font size=3 face="Times New Roman"> </font><br><font size=2 face="Arial">I understand you are having problems with
your cluster, but you do NOT need to have GPFS "started" to <br>display and/or change configuration paramters. You do need at least
a majority of the nodes to be up and in communcation (e.g. can talk to
each other by tcp/ip)</font><font size=3 face="Times New Roman"><br></font><font size=1 face="Courier New"><b><br>--ccr-enable</b></font><font size=1 face="Arial"><br>Enables the configuration server repository (CCR), which stores redundant
copies of configuration data files on all quorum nodes. The advantage of
CCR over the traditional primary or backup configuration server semantics
is that when using CCR, all GPFS administration commands as well as file
system mounts and daemon startups work normally as long as a majority of
quorum nodes are accessible.</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br>Think about how this must work (I have the advantage of actually NOT knowing
the details, but one can reason...)<br>to maintain a consistent single configuration database, a majority of quorum
nodes MUST agree on every bit of data in the configuration database.</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br>Even to query the database and get a correct answer, you'd have to know
that a majority agree on the answer.</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br>(You could ask 1 guy, but then how would you know if he was telling you
what the majority opinion is? The minority need not lie to mislead you,
<br>I don't think CCR guards against Byzantine failures...<br>The minority guy could just be out of touch for a while...)</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br>I advise that you do some testing on a test cluster (could be virtual)...
</font><font size=3 face="Times New Roman"><br><br><br></font><font size=1 color=#5f5f5f face="Arial"><br>From: </font><font size=1 face="Arial">Bryan
Banister <</font><a href=mailto:bbanister@jumptrading.com><font size=1 color=#800080 face="Arial"><u>bbanister@jumptrading.com</u></font></a><font size=1 face="Arial">></font><font size=1 color=#5f5f5f face="Arial"><br>To: </font><font size=1 face="Arial">"gpfsug
main discussion list (</font><a href="mailto:gpfsug-discuss@spectrumscale.org"><font size=1 color=#800080 face="Arial"><u>gpfsug-discuss@spectrumscale.org</u></font></a><font size=1 face="Arial">)"
<</font><a href="mailto:gpfsug-discuss@spectrumscale.org"><font size=1 color=#800080 face="Arial"><u>gpfsug-discuss@spectrumscale.org</u></font></a><font size=1 face="Arial">></font><font size=1 color=#5f5f5f face="Arial"><br>Date: </font><font size=1 face="Arial">07/27/2016
01:37 PM</font><font size=1 color=#5f5f5f face="Arial"><br>Subject: </font><font size=1 face="Arial">[gpfsug-discuss]
CCR troubles</font><font size=1 color=#5f5f5f face="Arial"><br>Sent by: </font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><font size=1 color=#800080 face="Arial"><u>gpfsug-discuss-bounces@spectrumscale.org</u></font></a><div align=center><hr noshade></div><br><font size=3 face="Times New Roman"><br><br></font><font size=2 face="Calibri"><br>When I have the GPFS cluster down, some GPFS commands no longer work like
they should, or at least they did work without CCR:<br> </font><font size=2 face="Courier New"><br># mmgetstate -aL
# Which stalls for a really stupid amount of time and then
spits out:<br>get file failed: Not enough CCR quorum nodes available (err 809)<br>gpfsClusterInit: Unexpected error from ccr fget mmsdrfs. Return code:
158<br>mmgetstate: Command failed. Examine previous error messages to determine
cause.</font><font size=2 face="Calibri"><br> <br>And trying to change tuning parameters now also barfs when GPFS is down:</font><font size=2 face="Courier New"><br># [root@fpia-gpfs-jcsdr01 ~]# mmlsconfig<br>get file failed: Not enough CCR quorum nodes available (err 809)<br>gpfsClusterInit: Unexpected error from ccr fget mmsdrfs. Return code:
158<br>mmlsconfig: Command failed. Examine previous error messages to determine
cause.<br> <br># mmchconfig worker1Threads=128,prefetchThreads=128<br>mmchconfig: Unable to obtain the GPFS configuration file lock.<br>mmchconfig: GPFS was unable to obtain a lock from node </font><a href="http://fpia-gpfs-jcsdr01.grid.jumptrading.com/"><font size=2 color=#800080 face="Courier New"><u>fpia-gpfs-jcsdr01.grid.jumptrading.com</u></font></a><font size=2 face="Courier New">.<br>mmchconfig: Command failed. Examine previous error messages to determine
cause.</font><font size=2 face="Calibri"><br> <br>Which means I will have to start GPFS, change the parameter, shut GPFS
down again, and start GPFS up again just to get the new setting.<br> <br>Is this really the new mode of operation for CCR enabled clusters?<br> <br>I searched CCR in the Concepts, Planning, and Install Guide and also the
Adv. Admin Guide, with explanation. <br> <br>If so, then maybe I¡¯ll go back to non CCR,<br>-Bryan</font><br><font size=3 face="Times New Roman"> </font><div align=center><hr></div><br><font size=1 color=#808080 face="Arial"><br>Note: This email is for the confidential use of the named addressee(s)
only and may contain proprietary, confidential or privileged information.
If you are not the intended recipient, you are hereby notified that any
review, dissemination or copying of this email is strictly prohibited,
and to please notify the sender immediately and destroy this email and
any attachments. Email transmission cannot be guaranteed to be secure or
error-free. The Company, therefore, does not make any guarantees as to
the completeness or accuracy of this email or any attachments. This email
is for informational purposes only and does not constitute a recommendation,
offer, request or solicitation of any kind to buy, sell, subscribe, redeem
or perform any type of transaction of a financial product.</font><font size=2 face="Courier New">_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href=http://spectrumscale.org/><font size=2 color=#800080 face="Courier New"><u>spectrumscale.org</u></font></a><font size=3 color=blue face="Times New Roman"><u><br></u></font><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><font size=2 color=#800080 face="Courier New"><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></a><br><font size=3 face="Times New Roman"> </font><br><font size=3 face="Times New Roman"> </font><div align=center><hr></div><br><font size=1 color=#808080 face="Arial"><br>Note: This email is for the confidential use of the named addressee(s)
only and may contain proprietary, confidential or privileged information.
If you are not the intended recipient, you are hereby notified that any
review, dissemination or copying of this email is strictly prohibited,
and to please notify the sender immediately and destroy this email and
any attachments. Email transmission cannot be guaranteed to be secure or
error-free. The Company, therefore, does not make any guarantees as to
the completeness or accuracy of this email or any attachments. This email
is for informational purposes only and does not constitute a recommendation,
offer, request or solicitation of any kind to buy, sell, subscribe, redeem
or perform any type of transaction of a financial product.</font><font size=1 face="Helvetica"><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href=http://spectrumscale.org/><font size=1 color=#800080 face="Helvetica"><u>spectrumscale.org</u></font></a><font size=3 color=blue face="Times New Roman"><u><br></u></font><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><font size=1 color=#800080 face="Helvetica"><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></a><br><font size=3 face="Times New Roman"> </font><p><font size=3><br></font><hr><font size=1 color=#808080 face="Arial"><br>Note: This email is for the confidential use of the named addressee(s)
only and may contain proprietary, confidential or privileged information.
If you are not the intended recipient, you are hereby notified that any
review, dissemination or copying of this email is strictly prohibited,
and to please notify the sender immediately and destroy this email and
any attachments. Email transmission cannot be guaranteed to be secure or
error-free. The Company, therefore, does not make any guarantees as to
the completeness or accuracy of this email or any attachments. This email
is for informational purposes only and does not constitute a recommendation,
offer, request or solicitation of any kind to buy, sell, subscribe, redeem
or perform any type of transaction of a financial product.</font><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br></font></tt><p></div><BR>