<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
We use NHC here (Node Health Check) from LBNL and our SS clients are almost all using NFS root. We have a check where we look for access to a couple of dotfiles (we have multiple SS file systems) and will mark a node offline if the checks fail.
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Many things can contribute to the failure of a single client node as we all know. Our checks are for actual node health on the clients, NOT to assess the health of the File Systems themselves. I will normally see MANY other problems from other monitoring
sources long before I normally see stale file handles at the client level. <br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
We did have to turn up the timeout for a check of the file to return on very busy clients, but we've haven't seen slowdowns due to hundreds of nodes all checking the file at the same time. Localized node slowdowns will occasionally mark a node offline for
this check here and there (normally a node that is extremely busy), but the next check will put the node right back online in the batch system.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Ed Wahl</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Ohio Supercomputer Center</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
ewahl@osc.edu<br>
</div>
<div>
<div id="appendonsend"></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> gpfsug-discuss-bounces@spectrumscale.org <gpfsug-discuss-bounces@spectrumscale.org> on behalf of Alexander John Mamach <alex.mamach@northwestern.edu><br>
<b>Sent:</b> Friday, August 9, 2019 1:46 PM<br>
<b>To:</b> gpfsug-discuss@spectrumscale.org <gpfsug-discuss@spectrumscale.org><br>
<b>Subject:</b> [gpfsug-discuss] Checking for Stale File Handles</font>
<div> </div>
</div>
<div lang="EN-US">
<div class="x_WordSection1">
<p class="x_MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
Hi folks,</p>
<p class="x_MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
</p>
<p class="x_MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
We’re currently investigating a way to check for stale file handles on the nodes across our cluster in a way that minimizes impact to the filesystem and performance.</p>
<p class="x_MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
</p>
<p class="x_MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
Has anyone found a direct way of doing so? We considered a few methods, including simply attempting to ls a GPFS filesystem from each node, but that might have false positives, (detecting slowdowns as stale file handles), and could negatively impact performance
with hundreds of nodes doing this simultaneously.</p>
<p class="x_MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
</p>
<p class="x_MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
Thanks,</p>
<p class="x_MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
</p>
<p class="x_MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
Alex</p>
<p class="x_MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
</p>
<p class="x_MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
Senior Systems Administrator<br>
<br>
Research Computing Infrastructure<br>
Northwestern University Information Technology (NUIT)<br>
<br>
2020 Ridge Ave<br>
Evanston, IL 60208-4311<br>
<br>
O: (847) 491-2219<br>
M: (312) 887-1881<br>
www.it.northwestern.edu</p>
<p class="x_MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: "Calibri", sans-serif;">
</p>
</div>
</div>
</div>
</body>
</html>