<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p>Hmm ... mmdiag --tokenmgr shows:</p>
<p><br>
</p>
<p></p>
<div> Server stats: requests 195417431 ServerSideRevokes 120140</div>
<div> nTokens 2146923 nranges 4124507</div>
<div> designated mnode appointed 55481 mnode thrashing detected 1036</div>
<br>
<p></p>
<p>So how do I convert "1036" to a node?</p>
<p><br>
</p>
<p>Simon</p>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> gpfsug-discuss-bounces@spectrumscale.org <gpfsug-discuss-bounces@spectrumscale.org> on behalf of Simon Thompson <S.J.Thompson@bham.ac.uk><br>
<b>Sent:</b> 20 February 2020 19:45:02<br>
<b>To:</b> gpfsug main discussion list<br>
<b>Subject:</b> [gpfsug-discuss] Unkillable snapshots</font>
<div> </div>
</div>
<div>
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p>Hi,</p>
<p><br>
</p>
<p>We have a snapshot which is stuck in the state "<span>DeleteRequired". When deleting, it goes through the motions but eventually gives up with:</span></p>
<p><span></p>
<div>Unable to quiesce all nodes; some processes are busy or holding required resources.</div>
<div>mmdelsnapshot: Command failed. Examine previous error messages to determine cause.</div>
<br>
</span>
<p></p>
<p><span>And in the mmfslog on the FS manager there are a bunch of retries and "failure to quesce" on nodes. However in each retry its never the same set of nodes. I suspect we have one HPC job somewhere killing us.</span></p>
<p><span><br>
</span></p>
<p><span>What's interesting is that we can delete other snapshots OK, it appears to be one particular fileset.</span></p>
<p><span><br>
</span></p>
<p><span>My old goto "<span>mmfsadm dump tscomm" isn't showing any particular node, and waiters around just tend to point to the FS manager node.</span></span></p>
<p><span><span><br>
</span></span></p>
<p><span><span>So ... any suggestions? I'm assuming its some workload holding a lock open or some such, but tracking it down is proving elusive!</span></span></p>
<p><span><span><br>
</span></span></p>
<p><span><span>Generally the FS is also "lumpy" ... at times it feels like a wifi connection on a train using a terminal, I guess its all related though.</span></span></p>
<p><span><span><br>
</span></span></p>
<p><span><span>Thanks</span></span></p>
<p><span><span><br>
</span></span></p>
<p><span><span>Simon </span></span></p>
</div>
</div>
</body>
</html>