<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;
panose-1:2 11 0 4 2 2 2 2 2 4;}
@font-face
{font-family:"Apple Color Emoji";
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Segoe UI";
panose-1:2 11 5 2 4 2 4 2 2 3;}
@font-face
{font-family:"IBM Plex Sans";
panose-1:2 11 5 3 5 2 3 0 2 3;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:12.0pt;
font-family:"Aptos",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#467886;
text-decoration:underline;}
p.xmsonormal, li.xmsonormal, div.xmsonormal
{mso-style-name:x_msonormal;
margin:0in;
font-size:11.0pt;
font-family:"Aptos",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:254898523;
mso-list-template-ids:299905004;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1
{mso-list-id:402025599;
mso-list-template-ids:30549186;}
@list l1:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2
{mso-list-id:771823461;
mso-list-template-ids:-2122138750;}
@list l2:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level2
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l3
{mso-list-id:1123379320;
mso-list-template-ids:1144945794;}
@list l3:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l3:level2
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l3:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l3:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l3:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l3:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l3:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l3:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l3:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l4
{mso-list-id:1712805092;
mso-list-template-ids:-368664886;}
@list l4:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l4:level2
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l4:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l4:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l4:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l4:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l4:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l4:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l4:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l5
{mso-list-id:2131706328;
mso-list-template-ids:1827027258;}
@list l5:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l5:level2
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l5:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l5:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l5:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l5:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l5:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l5:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l5:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#467886" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Hi All,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Mathias is exactly correct … there was some misconception both on my part _<i>and</i>_ on the part of IBM support in regards to fsstruct errors! Mathias and I discussed this outside of this mailing list and
he was able to clear up my misconceptions, for which I owe him many thanks.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">For those who may not read thru the rest of this e-mail I want to apologize up front for my misconceptions. As I’ll detail below, the truth is out there (<grin>), just not obvious nor easy to find. My rant
against the capabilities of GPFS was misinformed and misguided and for that I apologize. My rant against IBM Support, unfortunately, was not.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">It turns out that with the way I had my ESS configured (default configuration plus e-mail notifications enabled and configured) we _<i>will</i>_ be notified of any fsstruct errors (we did a test which confirmed
this) … and many other things as well that are most definitely _<i>not</i>_ obvious if you just run “mmlscallback” and see what it lists out. For a list of events that are monitored Mathias pointed me to this page in the IBM documentation:
<a href="https://www.ibm.com/docs/en/storage-scale/5.1.8?topic=references-events">
https://www.ibm.com/docs/en/storage-scale/5.1.8?topic=references-events</a> (you can adjust the GPFS version appropriately for your site).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">My RFE will therefore be closed with “feature already implemented.”<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We did agree that the lack of a clear linkage between the callbacks listed and the events that are actually monitored for is an area of potential confusion … as evidently I’m not the only customer who has
had a similar misunderstanding of the way things actually work.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The frustrating thing for me is that I opened up a ticket, worked with IBM Support for 6 months, and then was blown off and told to file an RFE … when what I was wanting _<i>was in GPFS all along!</i>_ Sigh.
Maybe IBM could consider spending a little less money on marketers who change the name of the product every few years and take the money saved and apply it to improving the support organization? To be fair to the support personnel, as the quote from the ticket
below indicates, they were in communication with at least one “dev team” member who had the same lack of knowledge. I’m thankful to have had my misconceptions corrected by Mathias, but that also just makes it even more clear that there are some severe issues
with the IBM Support organization. I hope that IBM can take steps to turn that ship around.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks all…<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Kevin B.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div id="mail-editor-reference-message-container">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-family:"Calibri",sans-serif;color:black">From:
</span></b><span style="font-family:"Calibri",sans-serif;color:black">gpfsug-discuss <gpfsug-discuss-bounces@gpfsug.org> on behalf of Mathias Dietz <MDIETZ@de.ibm.com><br>
<b>Date: </b>Thursday, January 25, 2024 at 10:52 AM<br>
<b>To: </b>gpfsug main discussion list <gpfsug-discuss@gpfsug.org><br>
<b>Subject: </b>[EXTERNAL] Re: [gpfsug-discuss] Wouldn't you like to know if you had filesystem corruption?<o:p></o:p></span></p>
</div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left" width="100%" style="width:100.0%;display:table;border-collapse:seperate;float:none">
<tbody>
<tr>
<td style="background:#A6A6A6;padding:5.25pt 1.5pt 5.25pt 1.5pt"></td>
<td width="100%" style="width:100.0%;background:#EAEAEA;padding:5.25pt 3.75pt 5.25pt 11.25pt">
<div>
<p class="MsoNormal" style="mso-element:frame;mso-element-frame-hspace:2.25pt;mso-element-wrap:around;mso-element-anchor-vertical:paragraph;mso-element-anchor-horizontal:column;mso-height-rule:exactly">
<span style="font-size:9.0pt;font-family:"Segoe UI",sans-serif;color:#212121">You don't often get email from mdietz@de.ibm.com.
</span><span style="color:black"><a href="https://aka.ms/LearnAboutSenderIdentification"><span style="font-size:9.0pt;font-family:"Segoe UI",sans-serif">Learn why this is important</span></a></span><span style="font-size:9.0pt;font-family:"Segoe UI",sans-serif;color:#212121"><o:p></o:p></span></p>
</div>
</td>
<td width="75" style="width:56.25pt;background:#EAEAEA;padding:5.25pt 3.75pt 5.25pt 3.75pt">
</td>
</tr>
</tbody>
</table>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Hi Kevin,</span> </p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">I think there is some misconception about how FSStruct errors are detected and handled. </span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">All nodes in a Storage Scale cluster have a health monitoring daemon running (backend for mmhealth cmd) which monitors the individual components and listens to callbacks to detect issues like FSStruct
errors.</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">As you correctly mentioned, the FSStruct callbacks will be fired on the Filesystem-Manager nodes only and therefore raise a new mmhealth event on that node. </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">You can see those events running
<b><i>mmhealth node show</i></b> on that node. <br>
<br>
Irrespective of the fact if this is an EMS node or an IO node, mmhealth will forward any event to the cluster manager to provide a consolidated cluster wide state view (<b>mmhealth cluster show</b>) </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">In addition, all events will be forwarded to the GUI, which will show those events as alerts. </span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Since many customers have their own monitoring system we provide multiple ways to get notified about new events:</span></p>
<ul type="disc">
<li class="MsoNormal" style="color:black;mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l3 level1 lfo3">
<span style="font-size:11.0pt">Scale GUI allows to configure Email notifications or SNMP traps <br>
</span><a href="https://www.ibm.com/docs/en/storage-scale/5.1.9?topic=gui-event-notifications"><span style="font-size:11.0pt">https://www.ibm.com/docs/en/storage-scale/5.1.9?topic=gui-event-notifications</span></a><span style="font-size:11.0pt"><o:p></o:p></span></li></ul>
<p class="MsoNormal"><o:p> </o:p></p>
<ul type="disc">
<li class="MsoNormal" style="color:black;mso-margin-top-alt:auto;margin-bottom:12.0pt;mso-list:l2 level1 lfo6">
<span style="font-size:11.0pt">mmhealth offers a modern webhook interface <br>
</span><a href="https://www.ibm.com/docs/en/storage-scale/5.1.9?topic=command-configuring-webhook-by-using-mmhealth"><span style="font-size:11.0pt">https://www.ibm.com/docs/en/storage-scale/5.1.9?topic=command-configuring-webhook-by-using-mmhealth</span></a><span style="font-size:11.0pt"><o:p></o:p></span></li><li class="MsoNormal" style="color:black;mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l2 level1 lfo6;align-self:start">
<span style="font-size:11.0pt">mmhealth can call user defined scripts to trigger any custom notification tool<o:p></o:p></span></li></ul>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:11.0pt;color:black">
</span><a href="https://www.ibm.com/docs/en/storage-scale/5.1.9?topic=mhn-running-user-defined-script-when-event-is-raised"><span style="font-size:11.0pt">https://www.ibm.com/docs/en/storage-scale/5.1.9?topic=mhn-running-user-defined-script-when-event-is-raised</span></a></p>
<ul type="disc">
<li class="MsoNormal" style="color:black;mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l4 level1 lfo9">
<span style="font-size:11.0pt">3<sup>rd</sup> party monitoring tools can use the REST API or mmhealth CLIs to poll the system status <o:p></o:p></span></li></ul>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt;margin-left:.5in">
<a href="https://www.ibm.com/docs/en/storage-scale/5.1.9?topic=endpoints-nodesnamehealthstates-get"><span style="font-size:11.0pt">https://www.ibm.com/docs/en/storage-scale/5.1.9?topic=endpoints-nodesnamehealthstates-get</span></a><span style="font-size:11.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Depending on which option you choose and where your external monitoring system is running you need to ensure that there is a network route to the system.<br>
(e.g. GUI Email & SNMP need the EMS node to talk to the server, webhook/custom script will need any node to talk to the server) </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black;background:white">ESS IO nodes are not necessarily restricted to an internal network. We have many customers who attach their ESS to their campus network for central management and monitoring.</span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">If you have further questions or want to hear more about monitoring & notifications, I can offer to schedule a webex session with you. </span></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div id="Signature">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">best regards<br>
<br>
Mathias Dietz</span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Storage Scale RAS Architect <br>
<br>
IBM Deutschland Research & Development GmbH</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Vorsitzender des Aufsichtsrats: Wolfgang Wendt</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Geschäftsführung: David Faller</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294</span></p>
</div>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="divRplyFwdMsg">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> gpfsug-discuss <gpfsug-discuss-bounces@gpfsug.org> on behalf of
Buterbaugh, Kevin Lynn <klbuter@sandia.gov><br>
<b>Sent:</b> Wednesday, January 24, 2024 6:08 PM<br>
<b>To:</b> gpfsug-discuss@spectrumscale.org <gpfsug-discuss@spectrumscale.org><br>
<b>Subject:</b> [EXTERNAL] [gpfsug-discuss] Wouldn't you like to know if you had filesystem corruption?</span>
</p>
<div>
<p class="MsoNormal"> </p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">Hi All, Wouldn’t you like to know if your IBM ESS had filesystem corruption? If you answered “no” my guess is that you’ve never experienced undetected filesystem
corruption! </span><span style="font-size:1.0pt;font-family:"Apple Color Emoji";color:white">😉</span><span style="font-size:1.0pt;color:white"> Did you know that if you’ve got an IBM ESS set up in its’
<o:p></o:p></span></p>
</div>
<div>
<p class="xmsonormal">Hi All,</p>
<p class="xmsonormal"> </p>
<p class="xmsonormal">Wouldn’t you like to know if your IBM ESS had filesystem corruption? If you answered “no” my guess is that you’ve never experienced undetected filesystem corruption!
<span style="font-family:"Apple Color Emoji"">😉</span></p>
<p class="xmsonormal"> </p>
<p class="xmsonormal">Did you know that if you’ve got an IBM ESS set up in its’ default configuration, which also matches the recommended configuration in every last piece of IBM documentation that I’ve ever come across, you WILL NOT be notified of filesystem
corruption?!?</p>
<p class="xmsonormal"> </p>
<p class="xmsonormal">Do you think IBM should fix this ASAP? If so, please up vote
<a href="https://ideas.ibm.com/ideas/ESS-I-61">https://ideas.ibm.com/ideas/ESS-I-61</a>.</p>
<p class="xmsonormal"> </p>
<p class="xmsonormal">If you, like me, consider this a bug in the existing product and not a “feature enhancement” to maybe be included in some future release if we’re lucky, then please keep reading.</p>
<p class="xmsonormal"> </p>
<p class="xmsonormal">Here’s the gory details to the best of my understanding…</p>
<p class="xmsonormal"> </p>
<p class="xmsonormal">Your IBM ESS can and will detect filesystem corruption (FS_STRUCT errors). But it currently will NOT, and cannot, let you know that it’s happened. The reason is that FS_STRUCT errors are detected only on the filesystem manager node,
which makes sense. But if you’re running in the default and recommended configuration your filesystem manager node is one of the I/O nodes, not the EMS node. The I/O nodes have no way to communicate anything out to you unless IBM decides to configure them
to do so – like they ALREADY DO with other things like hardware events – by routing the error thru the EMS node which can send it on to you.</p>
<p class="xmsonormal"> </p>
<p class="xmsonormal">You could fix this problem yourself by writing a custom callback script to send you an e-mail (or a text) whenever an FS_STRUCT error is detected by the filesystem manager node … EXCEPT that you’d need mailx / postfix or something like
that and IBM doesn’t provide you with a way to install them on the I/O nodes. As an aside, if you’re NOT on an ESS (i.e. running GPFS on some sort of commodity hardware) you can and should do this!</p>
<p class="xmsonormal"> </p>
<p class="xmsonormal">There is a workaround for this issue, which is to run your filesystem manager(s) on the EMS node. However, 1) this goes against IBM’s recommendations (and defaults), and 2) is not possible for larger ESS systems as the EMS node doesn’t
have enough RAM to handle the filesystem manager function.</p>
<p class="xmsonormal"> </p>
<p class="xmsonormal">Personally, I think it’s absolutely crazy that an I/O node can tell you that you’ve got a pdisk failure but can’t tell you that you’ve got filesystem corruption! If you agree, then please up vote the RFE above.</p>
<p class="xmsonormal"> </p>
<p class="xmsonormal"><rant></p>
<p class="xmsonormal">Even if you don’t agree, let me ask you to consider up voting the RFE anyway. Why? To send a message to IBM that you consider it unacceptable for them to allow a customer (me, obviously) to open up a support ticket for this very issue
(again, I consider this a very serious bug, not a feature enhancement) in July of 2023, work with the customer for 6 months, and then blow the customer off by telling them, and I quote:</p>
<p class="xmsonormal"> </p>
<p style="line-height:18.0pt;vertical-align:baseline">“<span style="font-size:9.0pt;font-family:"IBM Plex Sans",sans-serif;color:#161616">As per the dev team, this feature has been in this way since really old versions and has not changed which means that is
not going to change soon. You can request an RFE with your idea for the development team to take it into account. Below I share the link where you can share your idea (RFE):”</span></p>
<p class="xmsonormal"> </p>
<p class="xmsonormal">“Not going to change soon.” Thanks for nothing, IBM … well, I do appreciate your honesty. I’ve got one other RFE out there - submitted in August of 2022 - and its’ status is still “Future Consideration.” I guess I’ll just keep my fingers
crossed that I never have filesystem corruption on an ESS. But if I do, let me highly recommend to you that you not assign me one of your support personnel who does not understand that 1 plus 4 does not equal 6 … or that October comes before November on the
calendar (both of which I have actually had happen to me in the last 6 months; no, sadly, I am not joking or exaggerating in the least).</p>
<p class="xmsonormal"> </p>
<p class="xmsonormal">To all the IBMers reading this I want you to know that I personally consider the ESS and GPFS to be the best storage solution out there from a technical perspective … I truly do. But that is rapidly becoming irrelevant when you are also
doing things like the above, especially when you are overly proud (I think you know what I mean) of your support even if it was good, which it used to be but sadly no longer is.</p>
<p class="xmsonormal"> </p>
<p class="xmsonormal">IBMers, I’m sure you don’t like this bit of public shaming. Guess what? I don’t like doing it. But I have complained directly to IBM about these things for quite some time now (ask my sales rep if you don’t believe me) and it’s done
no good whatsoever. Not only did I count to 100 before composing this e-mail, I slept on it. I don’t know what else to do when things aren’t changing. But I promise you this, if you’ll stop doing stuff like this I will absolutely be more than glad to never
have to send another e-mail like this one again. Deal?</p>
<p class="xmsonormal"></rant></p>
<p class="xmsonormal"> </p>
<p class="xmsonormal">Thank you, all…</p>
<p class="xmsonormal"> </p>
<p class="xmsonormal">Kevin B.</p>
<p class="xmsonormal"> </p>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>