<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi Bob,
<div class=""><br class="">
</div>
<div class="">We use the nodeLeave callback to detect node expels … for what you’re wanting to do I wonder if nodeJoin might work?? If a node joins the cluster and then has an uptime of a few minutes you could go looking in /tmp/mmfs. HTH...</div>
<div class=""><br class="">
</div>
<div class="">--</div>
<div class="">
<div class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class="">Kevin Buterbaugh - Senior System Administrator</div>
<div class="">Vanderbilt University - Advanced Computing Center for Research and Education</div>
<div class=""><a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" class="">Kevin.Buterbaugh@vanderbilt.edu</a> - (615)875-9633</div>
</div>
</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Jan 30, 2019, at 3:02 PM, Sanchez, Paul <<a href="mailto:Paul.Sanchez@deshaw.com" class="">Paul.Sanchez@deshaw.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; color: rgb(31, 73, 125);" class="">There are some cases which I don’t believe can be caught with callbacks (e.g. DMS = Dead Man Switch). But you could possibly use preStartup to check the host uptime to make an assumption if GPFS
was restarted long after the host booted. You could also peek in /tmp/mmfs and only report if you find something there. That said, the docs say that preStartup fires after the node joins the cluster. So if that means once the node is ‘active’ then you might
miss out on nodes stuck in ‘arbitrating’ for a while due to a waiter problem.<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; color: rgb(31, 73, 125);" class="">We run a script with cron which monitors the myriad things which can go wrong and attempt to right those which are safe to fix, and raise alerts appropriately. Something like that, outside the
reach of GPFS, is often a good choice if you don’t need to know something the moment it happens.<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; color: rgb(31, 73, 125);" class="">Thx<br class="">
Paul<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div>
<div class="">
<div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(225, 225, 225); padding: 3pt 0in 0in;" class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<b class=""><span style="font-size: 11pt;" class="">From:</span></b><span style="font-size: 11pt;" class=""><span class="Apple-converted-space"> </span><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" style="color: rgb(149, 79, 114); text-decoration: underline;" class="">gpfsug-discuss-bounces@spectrumscale.org</a><span class="Apple-converted-space"> </span><<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" style="color: rgb(149, 79, 114); text-decoration: underline;" class="">gpfsug-discuss-bounces@spectrumscale.org</a>><span class="Apple-converted-space"> </span><b class="">On
Behalf Of<span class="Apple-converted-space"> </span></b>Oesterlin, Robert<br class="">
<b class="">Sent:</b><span class="Apple-converted-space"> </span>Wednesday, January 30, 2019 3:52 PM<br class="">
<b class="">To:</b><span class="Apple-converted-space"> </span>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" style="color: rgb(149, 79, 114); text-decoration: underline;" class="">gpfsug-discuss@spectrumscale.org</a>><br class="">
<b class="">Subject:</b><span class="Apple-converted-space"> </span>[gpfsug-discuss] Node ‘crash and restart’ event using GPFS callback?<o:p class=""></o:p></span></div>
</div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class="">Anyone crafted a good way to detect a node ‘crash and restart’ event using GPFS callbacks? I’m thinking “preShutdown” but I’m not sure if that’s the best. What I’m really looking for is did the node shutdown (abort) and
create a dump in /tmp/mmfs<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
Bob Oesterlin<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
Sr Principal Storage Engineer, Nuance<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri, sans-serif;" class="">
<o:p class=""> </o:p></div>
</div>
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">_______________________________________________</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">gpfsug-discuss
mailing list</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">gpfsug-discuss
at<span class="Apple-converted-space"> </span></span><a href="http://spectrumscale.org/" style="color: rgb(149, 79, 114); text-decoration: underline; font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">spectrumscale.org</a><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cccd012a939124326a53908d686f64117%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636844789557921185&sdata=9bMPd%2F%2B%2Babt6IdeFYcdznPBQwPrMLFsXHTBYISlyYGM%3D&reserved=0" style="color: rgb(149, 79, 114); text-decoration: underline; font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cccd012a939124326a53908d686f64117%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636844789557921185&sdata=9bMPd%2F%2B%2Babt6IdeFYcdznPBQwPrMLFsXHTBYISlyYGM%3D&reserved=0</a></div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>