<html><body><p>What version of Scale/ GPFS code is this cluster on ? <br><br>------------------------------------------<br>Sven Oehme <br>Scalable Storage Research <br>email: oehmes@us.ibm.com <br>Phone: +1 (408) 824-8904 <br>IBM Almaden Research Lab <br>------------------------------------------<br><br><img width="16" height="16" src="cid:1__=0FBB0A22DF848FAB8f9e8a93df938690918c0FB@" border="0" alt="Inactive hide details for Aaron Knister ---01/23/2017 01:31:29 AM---I was afraid someone would ask :) One possible use would be"><font color="#424282">Aaron Knister ---01/23/2017 01:31:29 AM---I was afraid someone would ask :) One possible use would be testing how monitoring reacts to and/or</font><br><br><font size="2" color="#5F5F5F">From: </font><font size="2">Aaron Knister <aaron.s.knister@nasa.gov></font><br><font size="2" color="#5F5F5F">To: </font><font size="2"><gpfsug-discuss@spectrumscale.org></font><br><font size="2" color="#5F5F5F">Date: </font><font size="2">01/23/2017 01:31 AM</font><br><font size="2" color="#5F5F5F">Subject: </font><font size="2">Re: [gpfsug-discuss] forcibly panic stripegroup everywhere?</font><br><font size="2" color="#5F5F5F">Sent by: </font><font size="2">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br><br><br><tt>I was afraid someone would ask :)<br><br>One possible use would be testing how monitoring reacts to and/or <br>corrects stale filesystems.<br><br>The use in my case is there's an issue we see quite often where a <br>filesystem won't unmount when trying to shut down gpfs. Linux insists <br>its still busy despite every process being killed on the node just about <br>except init. It's a real pain because it complicates maintenance, <br>requiring a reboot of some nodes prior to patching for example.<br><br>I dug into it and it appears as though when this happens the <br>filesystem's mnt_count is ridiculously high (300,000+ in one case). I'm <br>trying to debug it further but I need to actually be able to make the <br>condition happen a few more times to debug it. A stripegroup panic isn't <br>a surefire way but it's the only way I've found so far to trigger this <br>behavior somewhat on demand.<br><br>One way I've found to trigger a mass stripegroup panic is to induce what <br>I call a "301 error":<br><br>loremds07: Sun Jan 22 00:30:03.367 2017: [X] File System ttest unmounted <br>by the system with return code 301 reason code 0<br>loremds07: Sun Jan 22 00:30:03.368 2017: Invalid argument<br><br>and tickle a known race condition between nodes being expelled from the <br>cluster and a manager node joining the cluster. When this happens it <br>seems to cause a mass stripe group panic that's over in a few minutes. <br>The trick there is that it doesn't happen every time I go through the <br>exercise and when it does there's no guarantee the filesystem that <br>panics is the one in use. If it's not an fs in use then it doesn't help <br>me reproduce the error condition. I was trying to use the "mmfsadm test <br>panic" command to try a more direct approach.<br><br>Hope that helps shed some light.<br><br>-Aaron<br><br>On 1/22/17 8:16 PM, Andrew Beattie wrote:<br>> Out of curiosity -- why would you want to?<br>> Andrew Beattie<br>> Software Defined Storage - IT Specialist<br>> Phone: 614-2133-7927<br>> E-mail: abeattie@au1.ibm.com <</tt><tt><a href="mailto:abeattie@au1.ibm.com">mailto:abeattie@au1.ibm.com</a></tt><tt>><br>><br>><br>><br>> ----- Original message -----<br>> From: Aaron Knister <aaron.s.knister@nasa.gov><br>> Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>> To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>> Cc:<br>> Subject: [gpfsug-discuss] forcibly panic stripegroup everywhere?<br>> Date: Mon, Jan 23, 2017 11:11 AM<br>><br>> This is going to sound like a ridiculous request, but, is there a way to<br>> cause a filesystem to panic everywhere in one "swell foop"? I'm assuming<br>> the answer will come with an appropriate disclaimer of "don't ever do<br>> this, we don't support it, it might eat your data, summon cthulu, etc.".<br>> I swear I've seen the fs manager initiate this type of operation before.<br>><br>> I can seem to do it on a per-node basis with "mmfsadm test panic <fs><br>> <error code>" but if I do that over all 1k nodes in my test cluster at<br>> once it results in about 45 minutes of almost total deadlock while each<br>> panic is processed by the fs manager.<br>><br>> -Aaron<br>><br>> --<br>> Aaron Knister<br>> NASA Center for Climate Simulation (Code 606.2)<br>> Goddard Space Flight Center<br>> (301) 286-2776<br>> _______________________________________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at spectrumscale.org<br>> </tt><tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></tt><tt><br>><br>><br>><br>><br>><br>><br>> _______________________________________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at spectrumscale.org<br>> </tt><tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></tt><tt><br>><br><br>-- <br>Aaron Knister<br>NASA Center for Climate Simulation (Code 606.2)<br>Goddard Space Flight Center<br>(301) 286-2776<br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></tt><tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></tt><tt><br><br></tt><br><br><BR>
</body></html>