<span style=" font-size:10pt;font-family:sans-serif">Hi Alex,</span><br><br><span style=" font-size:10pt;font-family:sans-serif">did you try mmhealth
? It should detect stale file handles of the gpfs filesystems already and
report a "stale_mount" event. </span><br><br><span style=" font-size:10pt;font-family:sans-serif"> </span><br><span style=" font-size:10pt;font-family:sans-serif">Mit freundlichen
Grüßen / Kind regards<br><br>Mathias Dietz<br><br>Spectrum Scale Development - Release Lead Architect (4.2.x)<br>Spectrum Scale RAS Architect<br>---------------------------------------------------------------------------<br>IBM Deutschland<br>Am Weiher 24<br>65451 Kelsterbach<br>Phone: +49 70342744105<br>Mobile: +49-15152801035<br>E-Mail: mdietz@de.ibm.com<br>-----------------------------------------------------------------------------<br>IBM Deutschland Research & Development GmbH<br>Vorsitzender des Aufsichtsrats: Martina Koederitz, Geschäftsführung: Dirk
WittkoppSitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht
Stuttgart, HRB 243294</span><br><br><br><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">From:
</span><span style=" font-size:9pt;font-family:sans-serif">Alexander
John Mamach <alex.mamach@northwestern.edu></span><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">To:
</span><span style=" font-size:9pt;font-family:sans-serif">gpfsug
main discussion list <gpfsug-discuss@spectrumscale.org></span><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Cc:
</span><span style=" font-size:9pt;font-family:sans-serif">"gpfsug-discuss@spectrumscale.org"
<gpfsug-discuss@spectrumscale.org></span><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Date:
</span><span style=" font-size:9pt;font-family:sans-serif">09/08/2019
22:33</span><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Subject:
</span><span style=" font-size:9pt;font-family:sans-serif">[EXTERNAL]
Re: [gpfsug-discuss] Checking for Stale File Handles</span><br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Sent
by: </span><span style=" font-size:9pt;font-family:sans-serif">gpfsug-discuss-bounces@spectrumscale.org</span><br><hr noshade><br><br><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:sans-serif">Hi
Fred,</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:sans-serif"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:sans-serif">We
sometimes find a node will show that GPFS is active when running mmgetstate,
but one of our GPFS filesystems, (such as our home or projects filesystems)
are inaccessible to users, while the other GPFS-mounted filesystems behave
as expected. Our current node health checks don’t always detect this,
especially when it’s for a resource-based mount that doesn’t impact the
node but would impact jobs trying to run on the node.</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:sans-serif"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:sans-serif">If
there is something native to GPFS that can detect this, all the better,
but I’m simply unaware of how to do so.</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:sans-serif"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:sans-serif">Thanks,</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:sans-serif"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:sans-serif">Alex</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:sans-serif"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:sans-serif">Senior
Systems Administrator<br><br>Research Computing Infrastructure<br>Northwestern University Information Technology (NUIT)<br><br>2020 Ridge Ave<br>Evanston, IL 60208-4311<br><br>O: (847) 491-2219<br>M: (312) 887-1881<br></span><a href="www.it.northwestern.edu"><span style=" font-size:12pt;font-family:sans-serif">www.it.northwestern.edu</span></a></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:sans-serif"> </span></p><br><hr><br><span style=" font-size:11pt;font-family:sans-serif"><b>From:</b> gpfsug-discuss-bounces@spectrumscale.org
<gpfsug-discuss-bounces@spectrumscale.org> on behalf of Frederick
Stock <stockf@us.ibm.com><b><br>Sent:</b> Friday, August 9, 2019 1:03:09 PM<b><br>To:</b> gpfsug-discuss@spectrumscale.org <gpfsug-discuss@spectrumscale.org><b><br>Cc:</b> gpfsug-discuss@spectrumscale.org <gpfsug-discuss@spectrumscale.org><b><br>Subject:</b> Re: [gpfsug-discuss] Checking for Stale File Handles</span><span style=" font-size:12pt"></span><br><span style=" font-size:12pt"> </span><br><span style=" font-size:12pt;font-family:Arial">Are you able to explain
why you want to check for stale file handles? Are you attempting
to detect failures of some sort, and why do the existing mechanisms in
GPFS not provide the functionality you require?</span><br><span style=" font-size:10pt;font-family:sans-serif"><br>Fred<br>__________________________________________________<br>Fred Stock | IBM Pittsburgh Lab | 720-430-8821<br>stockf@us.ibm.com</span><br><span style=" font-size:12pt;font-family:Arial"> </span><br><span style=" font-size:12pt;font-family:Arial"> </span><br><span style=" font-size:12pt;font-family:Arial">----- Original message
-----<br>From: Alexander John Mamach <alex.mamach@northwestern.edu><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: "gpfsug-discuss@spectrumscale.org" <gpfsug-discuss@spectrumscale.org><br>Cc:<br>Subject: [EXTERNAL] [gpfsug-discuss] Checking for Stale File Handles<br>Date: Fri, Aug 9, 2019 1:46 PM<br> </span><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:Arial">Hi
folks,</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:Arial"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:Arial">We’re
currently investigating a way to check for stale file handles on the nodes
across our cluster in a way that minimizes impact to the filesystem and
performance.</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:Arial"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:Arial">Has
anyone found a direct way of doing so? We considered a few methods, including
simply attempting to ls a GPFS filesystem from each node, but that might
have false positives, (detecting slowdowns as stale file handles), and
could negatively impact performance with hundreds of nodes doing this simultaneously.</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:Arial"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:Arial">Thanks,</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:Arial"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:Arial">Alex</span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:Arial"> </span></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:Arial">Senior
Systems Administrator<br><br>Research Computing Infrastructure<br>Northwestern University Information Technology (NUIT)<br><br>2020 Ridge Ave<br>Evanston, IL 60208-4311<br><br>O: (847) 491-2219<br>M: (312) 887-1881<br></span><a href="www.it.northwestern.edu"><span style=" font-size:12pt;font-family:Arial">www.it.northwestern.edu</span></a></p><p style="margin-top:0px;margin-Bottom:0px"><span style=" font-size:12pt;font-family:Arial"> </span></p><br><tt><span style=" font-size:10pt">_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org</span></tt><tt><span style=" font-size:10pt;color:blue"><u><br></u></span></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><tt><span style=" font-size:10pt;color:blue"><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></span></tt></a><tt><span style=" font-size:10pt"></span></tt><br><span style=" font-size:12pt;font-family:Arial"> </span><br><tt><span style=" font-size:10pt">_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></span></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><span style=" font-size:10pt">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></tt></a><tt><span style=" font-size:10pt"><br></span></tt><br><br><BR>