<div dir="ltr">Hi,<div><br></div><div>now i need to swap back in a lot of information about GPFS i tried to swap out :-)</div><div><br></div><div>i bet kswapd is not doing anything you think the name suggest here, which is handling swap space. i claim the kswapd thread is trying to throw dentries out of the cache and what it tries to actually get rid of are entries of directories very high up in the tree which GPFS still has a refcount on so it can't free it. when it does this there is a single thread (unfortunate was never implemented with multiple threads) walking down the tree to find some entries to steal, it it can't find any it goes to the next , next , etc and on a bus system it can take forever to free anything up. there have been multiple fixes in this area in 5.0.1.x and 5.0.2 which i pushed for the weeks before i left IBM. you never see this in a trace with default traces which is why nobody would have ever suspected this, you need to set special trace levels to even see this. </div><div>i don't know the exact version the changes went into, but somewhere in the 5.0.1.X timeframe. the change was separating the cache list to prefer stealing files before directories, also keep a minimum percentages of directories in the cache (10 % by default) before it would ever try to get rid of a directory. it also tries to keep a list of free entries all the time (means pro active cleaning them) and also allows to go over the hard limit compared to just block as in previous versions. so i assume you run a version prior to 5.0.1.x and what you see is kspwapd desperately get rid of entries, but can't find one its already at the limit so it blocks and doesn't allow a new entry to be created or promoted from the statcache . </div><div><br></div><div>again all this is without source code access and speculation on my part based on experience :-)</div><div><br></div><div>what version are you running and also share mmdiag --stats of that node </div><div><br></div><div>sven</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Nov 27, 2018 at 9:54 AM Simon Thompson <<a href="mailto:S.J.Thompson@bham.ac.uk">S.J.Thompson@bham.ac.uk</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-GB" link="#0563C1" vlink="#954F72">
<div class="m_674267759528597120WordSection1">
<p class="MsoNormal">Thanks Sven …<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">We found a node with kswapd running 100% (and swap was off)…<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Killing that node made access to the FS spring into life.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Simon<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black"><<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a>> on behalf of "<a href="mailto:oehmes@gmail.com" target="_blank">oehmes@gmail.com</a>" <<a href="mailto:oehmes@gmail.com" target="_blank">oehmes@gmail.com</a>><br>
<b>Reply-To: </b>"<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a>" <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a>><br>
<b>Date: </b>Tuesday, 27 November 2018 at 16:14<br>
<b>To: </b>"<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a>" <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a>><br>
<b>Subject: </b>Re: [gpfsug-discuss] Hanging file-systems<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:"-webkit-standard",serif;color:black">1. are you under memory pressure or even worse started swapping . </span><u></u><u></u></p>
</div>
</div>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
</blockquote></div>