<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Looking at this further, the output says the “</span>The following disks of home will be formatted on node cl003:<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">“
however that node is the node in ‘arbitrating’ state, so I don’t see how that would work,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">-B<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> gpfsug-discuss-bounces@spectrumscale.org [mailto:gpfsug-discuss-bounces@spectrumscale.org]
<b>On Behalf Of </b>Bryan Banister<br>
<b>Sent:</b> Thursday, January 05, 2017 2:27 PM<br>
<b>To:</b> gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
<b>Subject:</b> Re: [gpfsug-discuss] nsd not adding with one quorum node down?<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Removing the quorum designation is an option.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">However I believe the file system manager must be assigned to the file system in order for the mmadddisk to work. If the file system manager is not assigned
(mmlsmgr to check) or continuously is reassigned to nodes but that fails (check /var/adm/ras/mmfs.log.latest on all nodes) or is blocked from being assigned due to the apparent node recovery in the cluster indicated by the one node in the ‘arbitrating’ state,
then the mmadddisk will not succeed.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">-Bryan<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">
<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">gpfsug-discuss-bounces@spectrumscale.org</a> [<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">mailto:gpfsug-discuss-bounces@spectrumscale.org</a>]
<b>On Behalf Of </b>J. Eric Wonderley<br>
<b>Sent:</b> Thursday, January 05, 2017 2:13 PM<br>
<b>To:</b> gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.org</a>><br>
<b>Subject:</b> Re: [gpfsug-discuss] nsd not adding with one quorum node down?<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Bryan:<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Have you ever attempted to do this knowing that one quorum server is down? *all* nsdservers will not see the nsd about to be added?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">How about temporarily removing quorum from a nsd server...?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">Thanks<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Thu, Jan 5, 2017 at 3:06 PM, Bryan Banister <<a href="mailto:bbanister@jumptrading.com" target="_blank">bbanister@jumptrading.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">There may be an issue with one of the other NSDs in the file system according to the “</span>mmadddisk:
File system home has some disks that are in a non-ready state.<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">“ message in our output. Best to check the status of the NSDs in the file system using the `mmlsdisk home` and if any
disks are not ‘up’ then run the `mmchdisk home start -a` command after confirming that all nsdservers can see the disks. I typically use `mmdsh -N nsdnodes tspreparedisk –s | dshbak –c` for that.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hope that helps,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">-Bryan</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">
<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a> [mailto:<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a>]
<b>On Behalf Of </b>J. Eric Wonderley<br>
<b>Sent:</b> Thursday, January 05, 2017 2:01 PM<br>
<b>To:</b> gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a>><br>
<b>Subject:</b> [gpfsug-discuss] nsd not adding with one quorum node down?</span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">I have one quorum node down and attempting to add a nsd to a fs:<br>
[root@cl005 ~]# mmadddisk home -F add_1_flh_home -v no |& tee /root/adddisk_flh_home.out<br>
Verifying file system configuration information ...<br>
<br>
The following disks of home will be formatted on node cl003:<br>
r10f1e5: size 1879610 MB<br>
Extending Allocation Map<br>
Checking Allocation Map for storage pool fc_ssd400G<br>
55 % complete on Thu Jan 5 14:43:31 2017<br>
Lost connection to file system daemon.<br>
mmadddisk: tsadddisk failed.<br>
Verifying file system configuration information ...<br>
mmadddisk: File system home has some disks that are in a non-ready state.<br>
mmadddisk: Propagating the cluster configuration data to all<br>
affected nodes. This is an asynchronous process.<br>
mmadddisk: Command failed. Examine previous error messages to determine cause.<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">Had to use -v no (this failed once before). Anyhow I next see:<br>
[root@cl002 ~]# mmgetstate -aL<br>
<br>
Node number Node name Quorum Nodes up Total nodes GPFS state Remarks
<br>
------------------------------------------------------------------------------------<br>
1 cl001 0 0 8 down quorum node<br>
2 cl002 5 6 8 active quorum node<br>
3 cl003 5 0 8 arbitrating quorum node<br>
4 cl004 5 6 8 active quorum node<br>
5 cl005 5 6 8 active quorum node<br>
6 cl006 5 6 8 active quorum node<br>
7 cl007 5 6 8 active quorum node<br>
8 cl008 5 6 8 active quorum node<br>
[root@cl002 ~]# mmlsdisk home<br>
disk driver sector failure holds holds storage<br>
name type size group metadata data status availability pool<br>
------------ -------- ------ ----------- -------- ----- ------------- ------------ ------------<br>
r10f1e5 nsd 512 1001 No Yes allocmap add up fc_ssd400G
<br>
r6d2e8 nsd 512 1001 No Yes ready up fc_8T
<br>
r6d3e8 nsd 512 1001 No Yes ready up fc_8T<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Do all quorum node have to be up and participating to do these admin type operations?
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" align="center">
</div>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:gray"><br>
Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this
email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness
or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial
product.</span><o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" align="center">
</div>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:gray"><br>
Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this
email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness
or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial
product.</span><o:p></o:p></p>
</div>
<br>
<hr>
<font face="Arial" color="Gray" size="1"><br>
Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this
email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness
or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial
product.<br>
</font>
</body>
</html>