<div dir="ltr">Aaron, <div><br></div><div>hold a bit with the upgrade , i just got word that while 4.2.1+ most likely addresses the issues i mentioned, there was a defect in the initial release of the parallel log recovery code. i will get the exact minimum version you need to deploy and send another update to this thread. </div><div><br></div><div>sven</div></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Jan 23, 2017 at 5:03 AM Sven Oehme <<a href="mailto:oehmes@gmail.com">oehmes@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg">Then i would suggest to move up to at least 4.2.1.LATEST , there is a high chance your problem might already be fixed. <div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">i see 2 potential area that got significant improvements , Token Manager recovery and Log Recovery, both are in latest 4.2.1 code enabled : </div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">2 significant improvements on Token Recovery in 4.2.1 : </div><div class="gmail_msg">
<p class="m_-4981009086590633322inbox-inbox-p1 gmail_msg"><span class="m_-4981009086590633322inbox-inbox-Apple-converted-space gmail_msg"> </span>1. Extendible hashing for token hash table.<span class="m_-4981009086590633322inbox-inbox-Apple-converted-space gmail_msg"> This</span> speeds up token lookup and thereby reduce tcMutex hold times for configurations with a large ratio of clients to token servers.<br class="gmail_msg">
<span class="m_-4981009086590633322inbox-inbox-Apple-converted-space gmail_msg"> </span>2. Cleaning up tokens held by failed nodes was making multiple passes over the whole token table, one for each failed node.<span class="m_-4981009086590633322inbox-inbox-Apple-converted-space gmail_msg"> </span>The loops are now inverted, so it makes a single pass over the able, and for each token found, does cleanup for all failed nodes.<br class="gmail_msg">
</p><p class="m_-4981009086590633322inbox-inbox-p1 gmail_msg">there are multiple smaller enhancements beyond 4.2.1 but thats the minimum level you want to be. i have seen token recovery of 10's of minutes similar to what you described going down to a minute with this change. </p><p class="m_-4981009086590633322inbox-inbox-p1 gmail_msg">on Log Recovery - in case of an unclean unmount/shutdown of a node prior 4.2.1 the Filesystem manager would only recover one Log file at a time, using a single thread, with 4.2.1 this is now done with multiple threads and multiple log files in parallel . </p></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><p class="m_-4981009086590633322inbox-inbox-p1 gmail_msg">Sven</p></div></div><br class="gmail_msg"><div class="gmail_quote gmail_msg"><div dir="ltr" class="gmail_msg">On Mon, Jan 23, 2017 at 4:22 AM Aaron Knister <<a href="mailto:aaron.s.knister@nasa.gov" class="gmail_msg" target="_blank">aaron.s.knister@nasa.gov</a>> wrote:<br class="gmail_msg"></div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">It's at 4.1.1.10.<br class="gmail_msg">
<br class="gmail_msg">
On 1/22/17 11:12 PM, Sven Oehme wrote:<br class="gmail_msg">
> What version of Scale/ GPFS code is this cluster on ?<br class="gmail_msg">
><br class="gmail_msg">
> ------------------------------------------<br class="gmail_msg">
> Sven Oehme<br class="gmail_msg">
> Scalable Storage Research<br class="gmail_msg">
> email: <a href="mailto:oehmes@us.ibm.com" class="gmail_msg" target="_blank">oehmes@us.ibm.com</a><br class="gmail_msg">
> Phone: <a href="tel:(408)%20824-8904" value="+14088248904" class="gmail_msg" target="_blank">+1 (408) 824-8904</a><br class="gmail_msg">
> IBM Almaden Research Lab<br class="gmail_msg">
> ------------------------------------------<br class="gmail_msg">
><br class="gmail_msg">
> Inactive hide details for Aaron Knister ---01/23/2017 01:31:29 AM---I<br class="gmail_msg">
> was afraid someone would ask :) One possible use would beAaron Knister<br class="gmail_msg">
> ---01/23/2017 01:31:29 AM---I was afraid someone would ask :) One<br class="gmail_msg">
> possible use would be testing how monitoring reacts to and/or<br class="gmail_msg">
><br class="gmail_msg">
> From: Aaron Knister <<a href="mailto:aaron.s.knister@nasa.gov" class="gmail_msg" target="_blank">aaron.s.knister@nasa.gov</a>><br class="gmail_msg">
> To: <<a href="mailto:gpfsug-discuss@spectrumscale.org" class="gmail_msg" target="_blank">gpfsug-discuss@spectrumscale.org</a>><br class="gmail_msg">
> Date: 01/23/2017 01:31 AM<br class="gmail_msg">
> Subject: Re: [gpfsug-discuss] forcibly panic stripegroup everywhere?<br class="gmail_msg">
> Sent by: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" class="gmail_msg" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a><br class="gmail_msg">
><br class="gmail_msg">
> ------------------------------------------------------------------------<br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
> I was afraid someone would ask :)<br class="gmail_msg">
><br class="gmail_msg">
> One possible use would be testing how monitoring reacts to and/or<br class="gmail_msg">
> corrects stale filesystems.<br class="gmail_msg">
><br class="gmail_msg">
> The use in my case is there's an issue we see quite often where a<br class="gmail_msg">
> filesystem won't unmount when trying to shut down gpfs. Linux insists<br class="gmail_msg">
> its still busy despite every process being killed on the node just about<br class="gmail_msg">
> except init. It's a real pain because it complicates maintenance,<br class="gmail_msg">
> requiring a reboot of some nodes prior to patching for example.<br class="gmail_msg">
><br class="gmail_msg">
> I dug into it and it appears as though when this happens the<br class="gmail_msg">
> filesystem's mnt_count is ridiculously high (300,000+ in one case). I'm<br class="gmail_msg">
> trying to debug it further but I need to actually be able to make the<br class="gmail_msg">
> condition happen a few more times to debug it. A stripegroup panic isn't<br class="gmail_msg">
> a surefire way but it's the only way I've found so far to trigger this<br class="gmail_msg">
> behavior somewhat on demand.<br class="gmail_msg">
><br class="gmail_msg">
> One way I've found to trigger a mass stripegroup panic is to induce what<br class="gmail_msg">
> I call a "301 error":<br class="gmail_msg">
><br class="gmail_msg">
> loremds07: Sun Jan 22 00:30:03.367 2017: [X] File System ttest unmounted<br class="gmail_msg">
> by the system with return code 301 reason code 0<br class="gmail_msg">
> loremds07: Sun Jan 22 00:30:03.368 2017: Invalid argument<br class="gmail_msg">
><br class="gmail_msg">
> and tickle a known race condition between nodes being expelled from the<br class="gmail_msg">
> cluster and a manager node joining the cluster. When this happens it<br class="gmail_msg">
> seems to cause a mass stripe group panic that's over in a few minutes.<br class="gmail_msg">
> The trick there is that it doesn't happen every time I go through the<br class="gmail_msg">
> exercise and when it does there's no guarantee the filesystem that<br class="gmail_msg">
> panics is the one in use. If it's not an fs in use then it doesn't help<br class="gmail_msg">
> me reproduce the error condition. I was trying to use the "mmfsadm test<br class="gmail_msg">
> panic" command to try a more direct approach.<br class="gmail_msg">
><br class="gmail_msg">
> Hope that helps shed some light.<br class="gmail_msg">
><br class="gmail_msg">
> -Aaron<br class="gmail_msg">
><br class="gmail_msg">
> On 1/22/17 8:16 PM, Andrew Beattie wrote:<br class="gmail_msg">
>> Out of curiosity -- why would you want to?<br class="gmail_msg">
>> Andrew Beattie<br class="gmail_msg">
>> Software Defined Storage - IT Specialist<br class="gmail_msg">
>> Phone: 614-2133-7927<br class="gmail_msg">
>> E-mail: <a href="mailto:abeattie@au1.ibm.com" class="gmail_msg" target="_blank">abeattie@au1.ibm.com</a> <mailto:<a href="mailto:abeattie@au1.ibm.com" class="gmail_msg" target="_blank">abeattie@au1.ibm.com</a>><br class="gmail_msg">
>><br class="gmail_msg">
>><br class="gmail_msg">
>><br class="gmail_msg">
>> ----- Original message -----<br class="gmail_msg">
>> From: Aaron Knister <<a href="mailto:aaron.s.knister@nasa.gov" class="gmail_msg" target="_blank">aaron.s.knister@nasa.gov</a>><br class="gmail_msg">
>> Sent by: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" class="gmail_msg" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a><br class="gmail_msg">
>> To: gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" class="gmail_msg" target="_blank">gpfsug-discuss@spectrumscale.org</a>><br class="gmail_msg">
>> Cc:<br class="gmail_msg">
>> Subject: [gpfsug-discuss] forcibly panic stripegroup everywhere?<br class="gmail_msg">
>> Date: Mon, Jan 23, 2017 11:11 AM<br class="gmail_msg">
>><br class="gmail_msg">
>> This is going to sound like a ridiculous request, but, is there a way to<br class="gmail_msg">
>> cause a filesystem to panic everywhere in one "swell foop"? I'm assuming<br class="gmail_msg">
>> the answer will come with an appropriate disclaimer of "don't ever do<br class="gmail_msg">
>> this, we don't support it, it might eat your data, summon cthulu, etc.".<br class="gmail_msg">
>> I swear I've seen the fs manager initiate this type of operation before.<br class="gmail_msg">
>><br class="gmail_msg">
>> I can seem to do it on a per-node basis with "mmfsadm test panic <fs><br class="gmail_msg">
>> <error code>" but if I do that over all 1k nodes in my test cluster at<br class="gmail_msg">
>> once it results in about 45 minutes of almost total deadlock while each<br class="gmail_msg">
>> panic is processed by the fs manager.<br class="gmail_msg">
>><br class="gmail_msg">
>> -Aaron<br class="gmail_msg">
>><br class="gmail_msg">
>> --<br class="gmail_msg">
>> Aaron Knister<br class="gmail_msg">
>> NASA Center for Climate Simulation (Code 606.2)<br class="gmail_msg">
>> Goddard Space Flight Center<br class="gmail_msg">
>> <a href="tel:(301)%20286-2776" value="+13012862776" class="gmail_msg" target="_blank">(301) 286-2776</a><br class="gmail_msg">
>> _______________________________________________<br class="gmail_msg">
>> gpfsug-discuss mailing list<br class="gmail_msg">
>> gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" class="gmail_msg" target="_blank">spectrumscale.org</a><br class="gmail_msg">
>> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" class="gmail_msg" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="gmail_msg">
>><br class="gmail_msg">
>><br class="gmail_msg">
>><br class="gmail_msg">
>><br class="gmail_msg">
>><br class="gmail_msg">
>><br class="gmail_msg">
>> _______________________________________________<br class="gmail_msg">
>> gpfsug-discuss mailing list<br class="gmail_msg">
>> gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" class="gmail_msg" target="_blank">spectrumscale.org</a><br class="gmail_msg">
>> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" class="gmail_msg" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="gmail_msg">
>><br class="gmail_msg">
><br class="gmail_msg">
> --<br class="gmail_msg">
> Aaron Knister<br class="gmail_msg">
> NASA Center for Climate Simulation (Code 606.2)<br class="gmail_msg">
> Goddard Space Flight Center<br class="gmail_msg">
> <a href="tel:(301)%20286-2776" value="+13012862776" class="gmail_msg" target="_blank">(301) 286-2776</a><br class="gmail_msg">
> _______________________________________________<br class="gmail_msg">
> gpfsug-discuss mailing list<br class="gmail_msg">
> gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" class="gmail_msg" target="_blank">spectrumscale.org</a><br class="gmail_msg">
> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" class="gmail_msg" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
> _______________________________________________<br class="gmail_msg">
> gpfsug-discuss mailing list<br class="gmail_msg">
> gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" class="gmail_msg" target="_blank">spectrumscale.org</a><br class="gmail_msg">
> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" class="gmail_msg" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="gmail_msg">
><br class="gmail_msg">
<br class="gmail_msg">
--<br class="gmail_msg">
Aaron Knister<br class="gmail_msg">
NASA Center for Climate Simulation (Code 606.2)<br class="gmail_msg">
Goddard Space Flight Center<br class="gmail_msg">
<a href="tel:(301)%20286-2776" value="+13012862776" class="gmail_msg" target="_blank">(301) 286-2776</a><br class="gmail_msg">
_______________________________________________<br class="gmail_msg">
gpfsug-discuss mailing list<br class="gmail_msg">
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" class="gmail_msg" target="_blank">spectrumscale.org</a><br class="gmail_msg">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" class="gmail_msg" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="gmail_msg">
</blockquote></div></blockquote></div>