<div dir="ltr">Hi Jan, <div><br></div><div>I am sorry if my post sounded accusatory - I did not mean it that way. We had a very frustrating experience trying to change recoverygroup yesterday morning. I've read the manual you have linked and indeed, you have outlined the correct procedure. </div><div><br></div><div>I am left wondering why the level 2 gpfs support instructed us not to do that in the future. Their support instructions are contradicting what's in the manual. We are running now with the --active recovery group in place and will change it permanently back to the default setting early in the new year. </div><div><br></div><div>Anyway, thanks for your help.</div><div><br></div><div>Damir</div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Dec 20, 2016 at 1:36 PM Jan-Frode Myklebust <<a href="mailto:janfrode@tanso.net">janfrode@tanso.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><div class="gmail_msg">I'm sorry for your trouble, but those 4 steps you got from IBM support does not seem correct. IBM support might not always realize that it's an ESS, and not plain GPFS... If you take down an ESS IO-node without moving its RG to the other node using "--servers othernode,thisnode", or by using --active (which I've never used), you'll take down the whole recoverygroup and need to suffer an uncontrolled failover. Such an uncontrolled failover takes a few minutes of filesystem hang, while a controlled failover should not hang the system.<br class="gmail_msg"><br class="gmail_msg"></div>I don't see why it's a problem that you now have an IO server that is owning both recoverygroups. Once your maintenance of the first IO servers is done, I would just revert the --servers order of that recovergroup, and it should move back.<br class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">The procedure to move RGs around during IO node maintenance is documented on page 10 the quick deployment guide (step 1-3):<br class="gmail_msg"><br class="gmail_msg"><a href="http://www.ibm.com/support/knowledgecenter/en/SSYSP8_4.5.0/c2785801.pdf?view=kc" class="gmail_msg" target="_blank">http://www.ibm.com/support/knowledgecenter/en/SSYSP8_4.5.0/c2785801.pdf?view=kc</a><br class="gmail_msg"></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><br class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg"> -jf<br class="gmail_msg"></div><div class="gmail_msg"><br class="gmail_msg"></div></div><div class="gmail_extra gmail_msg"><br class="gmail_msg"><div class="gmail_quote gmail_msg">On Tue, Dec 20, 2016 at 6:19 PM, Damir Krstic <span dir="ltr" class="gmail_msg"><<a href="mailto:damir.krstic@gmail.com" class="gmail_msg" target="_blank">damir.krstic@gmail.com</a>></span> wrote:<br class="gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg">For sake of everyone else on this listserv, I'll highlight the appropriate procedure here. It turns out, changing recovery group on an active system is not recommended by IBM. We tried following Jan's recommendation this morning, and the system became unresponsive for about 30 minutes. It only became responsive (and recovery group change finished) after we killed couple of processes (ssh and scp) going to couple of clients. <div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">I got a Sev. 1 with IBM opened and they tell me that appropriate steps for IO maintenance are as follows:</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">1. change cluster managers to system that will stay up (mmlsmgr - mmchmgr)</div><div class="gmail_msg">2. unmount gpfs on io node that is going down</div><div class="gmail_msg">3. shutdown gpfs on io node that is going down</div><div class="gmail_msg">4. shutdown os</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">That's it - recovery groups should not be changed. If there is a need to change recovery group, use --active option (not permanent change). </div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">We are now stuck in situation that io2 server is owner of both recovery groups. The way IBM tells us to fix this is to unmount the filesystem on all clients and change recovery groups then. We can't do it now and will have to schedule maintenance sometime in 2017. For now, we have switched recovery groups using --active flag and things (filesystem performance) seems to be OK. Load average on both io servers is quite high (250avg) and does not seem to be going down.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">I really wish that maintenance procedures were documented somewhere on IBM website. This experience this morning has really shaken my confidence in ESS. </div><span class="m_4025470254161590475HOEnZb gmail_msg"><font color="#888888" class="gmail_msg"><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Damir</div></font></span></div><div class="m_4025470254161590475HOEnZb gmail_msg"><div class="m_4025470254161590475h5 gmail_msg"><br class="gmail_msg"><div class="gmail_quote gmail_msg"><div dir="ltr" class="gmail_msg">On Mon, Dec 19, 2016 at 9:53 AM Jan-Frode Myklebust <<a href="mailto:janfrode@tanso.net" class="gmail_msg" target="_blank">janfrode@tanso.net</a>> wrote:<br class="gmail_msg"></div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"><br class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"></div><div class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg">Move its recoverygrops to the other node by putting the other node as primary server for it:</div><div class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"><br class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"></div><div class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg">mmchrecoverygroup rgname --servers otherServer,thisServer</div><div class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"><br class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"></div><div class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg">And verify that it's now active on the other node by "mmlsrecoverygroup rgname -L".</div><div class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"><br class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"></div><div class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg">Move away any filesystem managers or cluster manager role if that's active on it. Check with mmlsmgr, move with mmchmgr/mmchmgr -c.</div><div class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"><br class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"></div><div class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg">Then you can run mmshutdown on it (assuming you also have enough quorum nodes in the remaining cluster).</div><div class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"><br class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"></div><div class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"><br class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"></div><div class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"> -jf</div><div class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"><br class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"><div class="gmail_quote m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"></div></div><div class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"><div class="gmail_quote m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"><div class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg">man. 19. des. 2016 kl. 15.53 skrev Damir Krstic <<a href="mailto:damir.krstic@gmail.com" class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg" target="_blank">damir.krstic@gmail.com</a>>:<br class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"></div></div></div><div class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"><div class="gmail_quote m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"><blockquote class="gmail_quote m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg">We have a single ESS GL6 system running GPFS 4.2.0-1. Last night one of the IO servers phoned home with memory error. IBM is coming out today to replace the faulty DIMM.<div class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"><br class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"></div><div class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg">What is the correct way of taking this system out for maintenance?</div><div class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"><br class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"></div><div class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg">Before ESS we had a large GPFS 3.5 installation with 14 IO servers. When we needed to do maintenance on the old system, we would migrate manager role and also move primary and secondary server roles if one of those systems had to be taken down. </div><div class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"><br class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"></div><div class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg">With ESS and resource pool manager roles etc. is there a correct way of shutting down one of the IO serves for maintenance?</div><div class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"><br class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"></div><div class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg">Thanks,</div><div class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg">Damir</div></div><br class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"><br class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"></blockquote></div></div><div class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"><div class="gmail_quote m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"><blockquote class="gmail_quote m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">_______________________________________________<br class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"><br class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg">gpfsug-discuss mailing list<br class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"><br class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg">gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg" target="_blank">spectrumscale.org</a><br class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"><br class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"><br class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg"></blockquote></div></div>
_______________________________________________<br class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg">
gpfsug-discuss mailing list<br class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg">
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg" target="_blank">spectrumscale.org</a><br class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="m_4025470254161590475m_-1109653483915209351gmail_msg gmail_msg">
</blockquote></div>
</div></div><br class="gmail_msg">_______________________________________________<br class="gmail_msg">
gpfsug-discuss mailing list<br class="gmail_msg">
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" class="gmail_msg" target="_blank">spectrumscale.org</a><br class="gmail_msg">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" class="gmail_msg" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="gmail_msg">
<br class="gmail_msg"></blockquote></div><br class="gmail_msg"></div>
_______________________________________________<br class="gmail_msg">
gpfsug-discuss mailing list<br class="gmail_msg">
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" class="gmail_msg" target="_blank">spectrumscale.org</a><br class="gmail_msg">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" class="gmail_msg" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="gmail_msg">
</blockquote></div>