<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI Emoji";
panose-1:2 11 5 2 4 2 4 2 2 3;}
@font-face
{font-family:Menlo-Regular;}
@font-face
{font-family:Helvetica-Bold;
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.apple-tab-span
{mso-style-name:apple-tab-span;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle23
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.fontstyle01
{mso-style-name:fontstyle01;
font-family:"Helvetica",sans-serif;
color:black;
font-weight:normal;
font-style:normal;}
span.fontstyle21
{mso-style-name:fontstyle21;
font-family:"Helvetica-Bold",serif;
color:black;
font-weight:bold;
font-style:normal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="DE-AT" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Hi Felipe!<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Yes . I am very sorry that I answer with such a delay!!<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">This was the response from the Support
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">“<o:p></o:p></span></p>
<p class="MsoNormal"><span class="fontstyle01"><span lang="EN-US" style="font-size:9.0pt">The IP pair connectivity detection is to address this issue:</span></span><span lang="EN-US" style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:black"><br>
<span class="fontstyle01">If there are more than one IP pair between a pair of nodes, if one of the IP pair has some problems, and disk lease request or reply</span><br>
<span class="fontstyle01">happens be sent on this IP pair, disk lease overdue could happen since TCP has very long retransmit timeout, but since the other IP</span><br>
<span class="fontstyle01">pairs are in good condition, we should avoid the node expel and send disk lease via other good IP pairs. So, when sending disk</span><br>
<span class="fontstyle01">lease and reply, we will detect the connectivity of the IP pair, if it’s in good condition, disk lease and reply will be sent, otherwise, we</span><br>
<span class="fontstyle01">will try other IP pairs for sending.</span><br>
<span class="fontstyle01">detectIpPairAggressiveness (undocumented configuration parameter) control whether we do detection,
</span></span><span class="fontstyle21"><span lang="EN-US" style="font-size:9.0pt">here there is only ONE IP</span></span><b><span lang="EN-US" style="font-size:9.0pt;font-family:"Helvetica-Bold",serif;color:black"><br>
<span class="fontstyle21">pair, actually, we don't need to check the IP pair connectivity.
</span></span></b><span class="fontstyle01"><span lang="EN-US" style="font-size:9.0pt">But we still need ping to work since we have other places to do</span></span><span lang="EN-US" style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:black"><br>
<span class="fontstyle01">ping check, like when disk lease overdue happens.</span><br>
<span class="fontstyle01">---------------------------<o:p></o:p></span></span></p>
<p class="MsoNormal"><span class="fontstyle01"><span lang="EN-US" style="font-size:9.0pt"><o:p> </o:p></span></span></p>
<p class="MsoNormal"><span class="fontstyle01"><span lang="EN-US" style="font-size:9.0pt">until the 5.1.7 code will be available early march detectIpPairAggressiveness may can be disabled.</span></span><span lang="EN-US" style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:black"><br>
<span class="fontstyle01">echo 999 | mmchconfig detectIpPairAggressiveness=0 -i<o:p></o:p></span></span></p>
<p class="MsoNormal"><span class="fontstyle01"><span lang="EN-US" style="font-size:9.0pt"><o:p> </o:p></span></span></p>
<p class="MsoNormal"><span class="fontstyle01"><span lang="EN-US" style="font-size:9.0pt"><o:p> </o:p></span></span></p>
<p class="MsoNormal"><span class="fontstyle01"><b><span lang="EN-US" style="font-size:9.0pt;color:red;background:yellow;mso-highlight:yellow">THANK YOU VERY MUCH, Felipe!!!</span></b></span><span class="fontstyle01"><b><span lang="EN-US" style="font-size:9.0pt;color:red"><o:p></o:p></span></b></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Best regards<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Walter
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> gpfsug-discuss <gpfsug-discuss-bounces@gpfsug.org>
<b>On Behalf Of </b>Felipe Knop<br>
<b>Sent:</b> Donnerstag, 2. März 2023 04:33<br>
<b>To:</b> gpfsug main discussion list <gpfsug-discuss@gpfsug.org><br>
<b>Subject:</b> Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span lang="EN-US">Walter,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Just following up. I just realized that the SalesForce case below has been closed. The support case owner was correctly able to identify the root cause as being the same problem as the one I mentioned below. The fix will
be in the upcoming 5.1.7.0 release.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Thanks for opening the case and working with the support team on this one.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Felipe<o:p></o:p></span></p>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">----<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Felipe Knop <a href="mailto:knop@us.ibm.com">knop@us.ibm.com</a><br>
GPFS Development and Security<br>
IBM Systems<br>
IBM Building 008<br>
2455 South Rd, Poughkeepsie, NY 12601<o:p></o:p></span></p>
</div>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:12.0pt;color:black">From:
</span></b><span lang="EN-US" style="font-size:12.0pt;color:black">gpfsug-discuss <<a href="mailto:gpfsug-discuss-bounces@gpfsug.org">gpfsug-discuss-bounces@gpfsug.org</a>> on behalf of Walter Sklenka <<a href="mailto:Walter.Sklenka@EDV-Design.at">Walter.Sklenka@EDV-Design.at</a>><br>
<b>Reply-To: </b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@gpfsug.org">gpfsug-discuss@gpfsug.org</a>><br>
<b>Date: </b>Wednesday, February 22, 2023 at 5:23 AM<br>
<b>To: </b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@gpfsug.org">gpfsug-discuss@gpfsug.org</a>><br>
<b>Subject: </b>[EXTERNAL] Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span lang="EN-US" style="font-size:1.0pt;color:white">Hi ; sorry for the delay Our case is TS012184140 They are still analizing As soon as I get feedback I will update you Mit freundlichen Grüßen Walter
Sklenka Technical Consultant EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210
<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span lang="EN-US" style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerStart<o:p></o:p></span></p>
</div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;border-radius:4px">
<tbody>
<tr>
<td style="padding:12.0pt 0cm 12.0pt 0cm">
<table class="MsoNormalTable" border="1" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;background:#D0D8DC;border:none;border-top:solid #90A4AE 3.0pt">
<tbody>
<tr>
<td valign="top" style="border:none;padding:0cm 7.5pt 3.75pt 4.5pt">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left">
<tbody>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<p class="MsoNormal"><b><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">This Message Is From an External Sender
<o:p></o:p></span></b></p>
</td>
</tr>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:black">This message came from outside your organization.
<o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span lang="EN-US" style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerEnd<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US">Hi ; sorry for the delay <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Our case is TS012184140 <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">They are still analizing <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">As soon as I get feedback I will update you
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">Mit freundlichen Grüßen<br>
<b><i>Walter Sklenka</i></b><br>
</span><b><i><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:gray">Technical Consultant</span></i></b><span style="color:black">
</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif;color:black"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:black">EDV-Design Informationstechnologie GmbH<br>
Giefinggasse 6/1/2, A-1210 Wien<br>
Tel: +43 1 29 22 165-31<br>
Fax: +43 1 29 22 165-90<br>
E-Mail: <a href="mailto:sklenka@edv-design.at" target="_blank"><span style="color:#0563C1">sklenka@edv-design.at</span></a></span><span style="font-size:8.0pt;color:black"><br>
</span><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:black">Internet:
<a href="http://www.edv-design.at/" target="_blank"><span style="color:#0563C1">www.edv-design.at</span></a></span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span lang="DE"> </span><span lang="EN-US"><o:p></o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="DE">Von:</span></b><span lang="DE"> gpfsug-discuss <<a href="mailto:gpfsug-discuss-bounces@gpfsug.org">gpfsug-discuss-bounces@gpfsug.org</a>>
<b>Im Auftrag von </b>Ryan Novosielski<br>
<b>Gesendet:</b> Friday, February 17, 2023 11:52 PM<br>
<b>An:</b> gpfsug main discussion list <<a href="mailto:gpfsug-discuss@gpfsug.org">gpfsug-discuss@gpfsug.org</a>><br>
<b>Betreff:</b> Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I talked about it a lot in TS011616986. <o:p>
</o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Part of the problem is we’re having a lot of strange problems at the same time, and so the different issues we’re having often come together (like one cause shows two symptoms).<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">I can’t remember if there was a case where I specifically mentioned the watchdog, or whether it was unexpectedly late lease times in general.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="color:black">--<br>
#BlackLivesMatter</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="color:black">____<br>
|| <a href="file:///UTGERS">\\UTGERS</a>, <span class="apple-tab-span"> </span> |---------------------------*O*---------------------------<br>
||_// the State<span class="apple-tab-span"> </span> | Ryan Novosielski - <a href="mailto:novosirj@rutgers.edu">novosirj@rutgers.edu</a><br>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus<br>
|| \\ of NJ<span class="apple-tab-span"> </span> | Office of Advanced Research Computing - MSB C630, Newark<br>
`'</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US"><br>
<br>
<o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"><span lang="EN-US">On Feb 17, 2023, at 04:43, Walter Sklenka <<a href="mailto:Walter.Sklenka@EDV-Design.at">Walter.Sklenka@EDV-Design.at</a>> wrote:<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Hi Ryan and Felipe!<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Could you eventually tell me the case number if you remember it?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">I opened the case and would reference to your case ID<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Or shall I send you mine ?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<div>
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span class="apple-converted-space"><span lang="EN-US"> </span></span><span lang="EN-US">gpfsug-discuss <<a href="mailto:gpfsug-discuss-bounces@gpfsug.org">gpfsug-discuss-bounces@gpfsug.org</a>><span class="apple-converted-space"> </span><b>On
Behalf Of<span class="apple-converted-space"> </span></b>Ryan Novosielski<br>
<b>Sent:</b><span class="apple-converted-space"> </span>Freitag, 17. Februar 2023 06:43<br>
<b>To:</b><span class="apple-converted-space"> </span>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@gpfsug.org">gpfsug-discuss@gpfsug.org</a>><br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded<o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Thanks for this, Felipe.<span class="apple-converted-space"> </span><o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">We’ve started seeing intermittent overdue leases in large numbers and don’t otherwise have an explanation for it, other than “look at your network,” which actually does show occasional signs of strange behavior/higher-than-normal
RTO values, but we’re not necessarily seeing those things happen at the same times as the lease issues. We’ve also seen “GPFS Critical Thread Watchdog” recently.<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">We had a case open about it, but didn’t draw any real conclusions. If any of our data might be helpful/if there’s a case we could reference to see if we’re also running into that, we could provide a gpfs.snap.<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">FWIW, we are running 5.1.3-1 on the storage side (except one system that’s about to be upgraded that runs a combination of 5.0.3-2 and 5.0.5-1), and 5.1.6-0 (soon to be 5.1.6-1) on the remote/client cluster side.<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">--<br>
#BlackLivesMatter<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">____<br>
|| <a href="file:///utgers/">\\UTGERS</a>, <span class="apple-converted-space"> </span> |---------------------------*O*---------------------------<br>
||_// the State<span class="apple-converted-space"> </span> | Ryan Novosielski - <a href="mailto:novosirj@rutgers.edu">novosirj@rutgers.edu</a><br>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus<br>
|| \\ of NJ<span class="apple-converted-space"> </span> | Office of Advanced Research Computing - MSB C630, Newark<br>
`'<o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US"><br>
<br>
<br>
<o:p></o:p></span></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">On Feb 16, 2023, at 12:02, Felipe Knop <<a href="mailto:knop@us.ibm.com">knop@us.ibm.com</a>> wrote:<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Walter,<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Thanks for the detailed description.<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">I don’t yet see anything glaringly incorrect on your configuration, but perhaps others might find something out of place.<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">I’d encourage you to open a case, since I spoke with a colleague yesterday, and he mentioned that he is working on a problem that may cause the lease thread to “loop” for a while. That might cause the critical thread
watchdog to flag the lease thread as taking too long to “check in”.<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Capturing gpfs.snap is important, since we’d be looking into all the<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">[W] ------------------[GPFS Critical Thread Watchdog]------------------</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">instances.<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Thanks,<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> Felipe<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">----<o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Felipe Knop <a href="mailto:knop@us.ibm.com"><span style="color:#0563C1">knop@us.ibm.com</span></a><br>
GPFS Development and Security<br>
IBM Systems<br>
IBM Building 008<br>
2455 South Rd, Poughkeepsie, NY 12601<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<div>
<div>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:12.0pt">From:<span class="apple-converted-space"> </span></span></b><span lang="EN-US" style="font-size:12.0pt">gpfsug-discuss <<a href="mailto:gpfsug-discuss-bounces@gpfsug.org"><span style="color:#0563C1">gpfsug-discuss-bounces@gpfsug.org</span></a>>
on behalf of Walter Sklenka <<a href="mailto:Walter.Sklenka@EDV-Design.at"><span style="color:#0563C1">Walter.Sklenka@EDV-Design.at</span></a>><br>
<b>Reply-To:<span class="apple-converted-space"> </span></b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@gpfsug.org"><span style="color:#0563C1">gpfsug-discuss@gpfsug.org</span></a>><br>
<b>Date:<span class="apple-converted-space"> </span></b>Thursday, February 16, 2023 at 9:16 AM<br>
<b>To:<span class="apple-converted-space"> </span></b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@gpfsug.org"><span style="color:#0563C1">gpfsug-discuss@gpfsug.org</span></a>><br>
<b>Subject:<span class="apple-converted-space"> </span></b>[EXTERNAL] Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:1.0pt;color:white">Hi Felipe! Once again me. Thank you very much for the hint I did not open a PMR yet because I fear they will ask me/us if we are cracy<span class="apple-converted-space"> </span></span><span lang="EN-US" style="font-size:1.0pt;font-family:"Segoe UI Emoji",sans-serif;color:white">☹</span><span class="apple-converted-space"><span lang="EN-US" style="font-size:1.0pt;color:white"> </span></span><span lang="EN-US" style="font-size:1.0pt;color:white">I
did not tell the full story yet We have a 3 node cluster, 2 NSD servers o1,o2 (same site ) and g1 (different</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerStart</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
</div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;border-radius: 4px">
<tbody>
<tr>
<td style="padding:12.0pt 0cm 12.0pt 0cm">
<table class="MsoNormalTable" border="1" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;background:#D0D8DC;border:none;border-top:solid #90A4AE 3.0pt">
<tbody>
<tr>
<td valign="top" style="border:none;padding:0cm 7.5pt 3.75pt 4.5pt">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left">
<tbody>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<div>
<div>
<p class="MsoNormal"><b><span style="font-size:10.5pt;font-family:"Arial",sans-serif">This Message Is From an External Sender</span></b><o:p></o:p></p>
</div>
</div>
</td>
</tr>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial",sans-serif">This message came from outside your organization.</span><o:p></o:p></p>
</div>
</div>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerEnd</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Hi Felipe!<span class="apple-converted-space"> </span><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Once again me. Thank you very much for the hint<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">I did not open a PMR yet because I fear they will ask me/us if we are cracy<span class="apple-converted-space"> </span></span><span lang="EN-US" style="font-family:"Segoe UI Emoji",sans-serif">☹</span><span class="apple-converted-space"><span lang="EN-US"> </span></span><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">I did not tell the full story yet<span class="apple-converted-space"> </span><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">We have a 3 node cluster, 2 NSD servers o1,o2 (same site ) and g1 (different site). (rhel 8.7)<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">All of them are Vmware VMs<span class="apple-converted-space"> </span><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">O1 and o2 have each 4 NVME drives passed through , there is a software raid 5 made over these NVMEs , and from them made a single NSD , for a filesystem fs4vm (m,r=2 )<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New""> </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">[root@ogpfs1 ras]# mmlscluster</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New""> </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">GPFS cluster information</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">========================</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New""> GPFS cluster name: edvdesign-cluster.local</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New""> GPFS cluster id: 12147978822727803186</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New""> GPFS UID domain: edvdesign-cluster.local</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New""> Remote shell command: /usr/bin/ssh</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New""> Remote file copy command: /usr/bin/scp</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New""> Repository type: CCR</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New""> </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">Node Daemon node name IP address Admin node name Designation</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">----------------------------------------------------------------------------</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New""> 1 ogpfs1-hs.local 10.20.30.1 ogpfs1-hs.local quorum-manager-perfmon</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New""> 2 ogpfs2-hs.local 10.20.30.2 ogpfs2-hs.local quorum-manager-perfmon</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New""> 3 ggpfsq.mgmt.cloudia <span class="apple-converted-space"> </span><a href="http://xxxx.other.net/"><span style="color:#0563C1">xxxx.other.net</span></a><span class="apple-converted-space"> </span>ggpfsq.mgmt.
a quorum-perfmon</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New""> </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">[root@ogpfs1 ras]# mmlsconfig</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">Configuration data for cluster edvdesign-cluster.local:</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">-------------------------------------------------------</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">clusterName edvdesign-cluster.local</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">clusterId 12147978822727803186</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">autoload yes</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">profile gpfsProtocolRandomIO</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">dmapiFileHandleSize 32</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">minReleaseLevel 5.1.6.0</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">tscCmdAllowRemoteConnections no</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">ccrEnabled yes</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">cipherList AUTHONLY</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">sdrNotifyAuthEnabled yes</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">maxblocksize 16M</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">[cesNodes]</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">maxMBpS 5000</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">numaMemoryInterleave yes</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">enforceFilesetQuotaOnRoot yes</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">workerThreads 512</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">[common]</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">tscCmdPortRange 60000-61000</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">[srv]</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">verbsPorts mlx5_0/1 mlx5_1/1</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">[common]</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">cesSharedRoot /fs4vmware/cesSharedRoot</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">[srv]</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">maxFilesToCache 10000</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">maxStatCache 20000</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">[common]</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">verbsRdma enable</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">[ggpfsq]</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">verbsRdma disable</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">[common]</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">verbsRdmaSend yes</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">[ggpfsq]</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">verbsRdmaSend no</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">[common]</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">verbsRdmaCm enable</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">[ggpfsq]</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">verbsRdmaCm disable</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">[srv]</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">pagepool 32G</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">[common]</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">adminMode central</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New""> </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">File systems in cluster edvdesign-cluster.local:</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">------------------------------------------------</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">/dev/fs4vm</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New""> </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New""> </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">[root@ogpfs1 ras]# mmlsdisk fs4vm -L</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">disk driver sector failure holds holds storage</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">name type size group metadata data status availability disk id pool remarks</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">------------ -------- ------ ----------- -------- ----- ------------- ------------ ------- ------------ ---------</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">ogpfs1_1 nsd 512 1 yes yes ready up 1 system desc</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">ogpfs2_1 nsd 512 2 yes yes ready up 2 system desc</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">ggpfsq_qdisk nsd 512 -1 no no ready up 3 system desc</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">Number of quorum disks: 3</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">Read quorum value: 2</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">Write quorum value: 2</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">And the two nodes o1 and o2 export the filesystem via CES NFS functions ( for VMware)<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">I think this isn´supported , that a NSD Server is also a CES Node?<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">And finally the RDMA Network :<span class="apple-converted-space"> </span><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">The both NSD servers also have a Mellanox ConnectX-6 Lx dual port 25Gb adapter also via passthrough<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">And these interfaces we configured for rdma (RoCE) ,<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Last but not least: this network is not switched but direct attached ( 2x25Gb directly connected between the NSD nodes )<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">RDMA Connections between nodes:</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New""> Fabric 0 - Device mlx5_0 Port 1 Width 1x Speed EDR lid 0</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New""> hostname idx CM state VS buff RDMA_CT(ERR) RDMA_RCV_MB RDMA_SND_MB VS_CT(ERR) VS_SND_MB VS_RCV_MB WAIT_CON_SLOT WAIT_NODE_SLOT</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New""> ogpfs2-hs.local 0 Y RTS (Y)256 478202 (0 ) 12728 67024 8864789(0 ) 22776 4643 0 0</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New""> Fabric 0 - Device mlx5_1 Port 1 Width 1x Speed EDR lid 0</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New""> hostname idx CM state VS buff RDMA_CT(ERR) RDMA_RCV_MB RDMA_SND_MB VS_CT(ERR) VS_SND_MB VS_RCV_MB WAIT_CON_SLOT WAIT_NODE_SLOT</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New""> ogpfs2-hs.local 1 Y RTS (Y)256 477659 (0 ) 12489 67034 8864773(0 ) 22794 4639 0 0</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">[root@ogpfs1 ras]#</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New""> </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">You mentioned that it might be a cpu contention : Maybe due to the VM layer (scheduling with other VMS) ? And wrong layout of VMs ( 8 vCPUs and 64GB Mem) [ esxis only single socket with 32/64 cores HT)<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">And also the direct attached RDMA ( +DAEMON) network is also not good?<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Do you think IBM would say no to check such a configuration ?<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Best regards<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Walter<span class="apple-converted-space"> </span><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<div>
<div>
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span class="apple-converted-space"><span lang="EN-US"> </span></span><span lang="EN-US">gpfsug-discuss <<a href="mailto:gpfsug-discuss-bounces@gpfsug.org"><span style="color:#0563C1">gpfsug-discuss-bounces@gpfsug.org</span></a>><span class="apple-converted-space"> </span><b>On
Behalf Of<span class="apple-converted-space"> </span></b>Felipe Knop<br>
<b>Sent:</b><span class="apple-converted-space"> </span>Mittwoch, 15. Februar 2023 15:59<br>
<b>To:</b><span class="apple-converted-space"> </span>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@gpfsug.org"><span style="color:#0563C1">gpfsug-discuss@gpfsug.org</span></a>><br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Walter,<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Thanks for the details.<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">The stack trace below captures the lease thread in the middle of sending the “lease” RPC. This operation normally is not blocking, and we do not often block while sending the RPC. But the stack trace “does not show” whether
there was anything blocking the thread prior to the point where the RPCs are sent.<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">At a first glance:<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">2023-02-14_19:44:07.430+0100: [W] counter: 0 (mark-idle: 0 mark-active: 0 pre-work: 0 post-work: 0) sched: (nvcsw: 0 nivcsw: 10)</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">I believe<span class="apple-converted-space"><b> </b></span><b>nivcsw: 10</b><span class="apple-converted-space"> </span>means that the thread was scheduled out of the CPU involuntarily, possibly indicating that there
is some CPU contention going on.<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Could you open a case to get debug data collected? If the problem can be recreated, I think we’ll need a recreate of the problem with traces enabled.<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Thanks,<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> Felipe<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">----<o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Felipe Knop <a href="mailto:knop@us.ibm.com"><span style="color:#0563C1">knop@us.ibm.com</span></a><br>
GPFS Development and Security<br>
IBM Systems<br>
IBM Building 008<br>
2455 South Rd, Poughkeepsie, NY 12601<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<div>
<div>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:12.0pt">From:<span class="apple-converted-space"> </span></span></b><span lang="EN-US" style="font-size:12.0pt">gpfsug-discuss <<a href="mailto:gpfsug-discuss-bounces@gpfsug.org"><span style="color:#0563C1">gpfsug-discuss-bounces@gpfsug.org</span></a>>
on behalf of Walter Sklenka <<a href="mailto:Walter.Sklenka@EDV-Design.at"><span style="color:#0563C1">Walter.Sklenka@EDV-Design.at</span></a>><br>
<b>Reply-To:<span class="apple-converted-space"> </span></b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@gpfsug.org"><span style="color:#0563C1">gpfsug-discuss@gpfsug.org</span></a>><br>
<b>Date:<span class="apple-converted-space"> </span></b>Wednesday, February 15, 2023 at 4:23 AM<br>
<b>To:<span class="apple-converted-space"> </span></b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@gpfsug.org"><span style="color:#0563C1">gpfsug-discuss@gpfsug.org</span></a>><br>
<b>Subject:<span class="apple-converted-space"> </span></b>[EXTERNAL] Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:1.0pt;color:white">Hi! This is a „full“ sequence in mmfs. log. latest Fortunately this was also the last event until now (yesterday evening) Maybe you can have a look? 2023-02-14_19: 43: 51. 474+0100:
[N] Disk lease period expired 0. 030 seconds ago in cluster<span class="apple-converted-space"> </span></span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerStart</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
</div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;border-radius: 4px">
<tbody>
<tr>
<td style="padding:12.0pt 0cm 12.0pt 0cm">
<table class="MsoNormalTable" border="1" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;background:#D0D8DC;border:none;border-top:solid #90A4AE 3.0pt">
<tbody>
<tr>
<td valign="top" style="border:none;padding:0cm 7.5pt 3.75pt 4.5pt">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left">
<tbody>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<div>
<div>
<p class="MsoNormal"><b><span style="font-size:10.5pt;font-family:"Arial",sans-serif">This Message Is From an External Sender</span></b><o:p></o:p></p>
</div>
</div>
</td>
</tr>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial",sans-serif">This message came from outside your organization.</span><o:p></o:p></p>
</div>
</div>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerEnd</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Hi!<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">This is a „full“ sequence in mmfs.log.latest<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Fortunately this was also the last event until now (yesterday evening)<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Maybe you can have a look?<span class="apple-converted-space"> </span><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">2023-02-14_19:43:51.474+0100: [N] Disk lease period expired 0.030 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">2023-02-14_19:44:07.430+0100: [W] ------------------[GPFS Critical Thread Watchdog]------------------</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">2023-02-14_19:44:07.430+0100: [W] PID: 7294 State: R (DiskLeaseThread) is overloaded for more than 8 seconds</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">2023-02-14_19:44:07.430+0100: [W] counter: 0 (mark-idle: 0 mark-active: 0 pre-work: 0 post-work: 0) sched: (nvcsw: 0 nivcsw: 10)</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">2023-02-14_19:44:07.430+0100: [W] Call Trace(PID: 7294):</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">2023-02-14_19:44:07.431+0100: [W] #0: 0x000055CABE4A56AB NodeConn::sendMessage(TcpConn**, iovec*, int, unsigned char, int, int, int, unsigned int, DestTag*, int*, unsigned
long long*, unsigned long long*, unsi</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">gned int*, CondvarName, vsendCallback_t*) + 0x42B at ??:0</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">2023-02-14_19:44:07.432+0100: [W] #1: 0x000055CABE4A595F llc_send_msg(ClusterConfiguration*, NodeAddr, iovec*, int, unsigned char, int, int, int, unsigned int, DestTag*,
int*, TcpConn**, unsigned long long*, u</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">nsigned long long*, unsigned int*, CondvarName, vsendCallback_t*, int, unsigned int) + 0xDF at ??:0</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">2023-02-14_19:44:07.437+0100: [W] #2: 0x000055CABE479A55 MsgRecord::send() + 0x1345 at ??:0</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">2023-02-14_19:44:07.438+0100: [W] #3: 0x000055CABE47A169 tscSendInternal(ClusterConfiguration*, unsigned int, unsigned char, int, int, NodeAddr*, TscReply*, TscScatteredBuff*,
int, int (*)(void*, ClusterConfig</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">uration*, int, NodeAddr*, TscReply*), void*, ChainedCallback**, __va_list_tag*) + 0x339 at ??:0</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">2023-02-14_19:44:07.439+0100: [W] #4: 0x000055CABE47C39A tscSendWithCallback(ClusterConfiguration*, unsigned int, unsigned char, int, NodeAddr*, TscReply*, int (*)(void*,
ClusterConfiguration*, int, NodeAddr*,</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">TscReply*), void*, void**, int, ...) + 0x1DA at ??:0</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">2023-02-14_19:44:07.440+0100: [W] #5: 0x000055CABE5F9853 MyLeaseState::renewLease(NodeAddr, TickTime) + 0x6E3 at ??:0</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">2023-02-14_19:44:07.440+0100: [W] #6: 0x000055CABE5FA682 ClusterConfiguration::checkAndRenewLease(TickTime) + 0x192 at ??:0</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">2023-02-14_19:44:07.441+0100: [W] #7: 0x000055CABE5FAAC6 ClusterConfiguration::RunLeaseChecks(void*) + 0x366 at ??:0</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">2023-02-14_19:44:07.441+0100: [W] #8: 0x000055CABDF2B662 Thread::callBody(Thread*) + 0x42 at ??:0</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">2023-02-14_19:44:07.441+0100: [W] #9: 0x000055CABDF18680 Thread::callBodyWrapper(Thread*) + 0xA0 at ??:0</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">2023-02-14_19:44:07.441+0100: [W] #10: 0x00007F3B7563D1CA start_thread + 0xEA at ??:0</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">2023-02-14_19:44:07.441+0100: [W] #11: 0x00007F3B7435BE73 __GI___clone + 0x43 at ??:0</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">2023-02-14_19:44:10.512+0100: [N] Disk lease reacquired in cluster xxx-cluster.</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">2023-02-14_19:44:10.512+0100: [N] Disk lease period expired 7.970 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">2023-02-14_19:44:12.563+0100: [N] Disk lease reacquired in cluster xxx-cluster.</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Thank you very much!<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Best regards<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Walter<span class="apple-converted-space"> </span><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<div>
<div>
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span class="apple-converted-space"><span lang="EN-US"> </span></span><span lang="EN-US">gpfsug-discuss <<a href="mailto:gpfsug-discuss-bounces@gpfsug.org"><span style="color:#0563C1">gpfsug-discuss-bounces@gpfsug.org</span></a>><span class="apple-converted-space"> </span><b>On
Behalf Of<span class="apple-converted-space"> </span></b>Felipe Knop<br>
<b>Sent:</b><span class="apple-converted-space"> </span>Mittwoch, 15. Februar 2023 00:06<br>
<b>To:</b><span class="apple-converted-space"> </span>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@gpfsug.org"><span style="color:#0563C1">gpfsug-discuss@gpfsug.org</span></a>><br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">All,<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">These messages like<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">[W] ------------------[GPFS Critical Thread Watchdog]------------------</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">indicate that a “critical thread”, in this case the lease thread, was apparently blocked for longer than expected. This is usually not caused by delays in the network, but possibly by excessive CPU load, blockage while
accessing the local file system, or possible mutex contention.<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Do you have other samples of the message, with a more complete stack trace? Or was the instance below the only one?<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> Felipe<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">----<o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Felipe Knop <a href="mailto:knop@us.ibm.com"><span style="color:#0563C1">knop@us.ibm.com</span></a><br>
GPFS Development and Security<br>
IBM Systems<br>
IBM Building 008<br>
2455 South Rd, Poughkeepsie, NY 12601<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<div>
<div>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:12.0pt">From:<span class="apple-converted-space"> </span></span></b><span lang="EN-US" style="font-size:12.0pt">gpfsug-discuss <<a href="mailto:gpfsug-discuss-bounces@gpfsug.org"><span style="color:#0563C1">gpfsug-discuss-bounces@gpfsug.org</span></a>>
on behalf of Walter Sklenka <<a href="mailto:Walter.Sklenka@EDV-Design.at"><span style="color:#0563C1">Walter.Sklenka@EDV-Design.at</span></a>><br>
<b>Reply-To:<span class="apple-converted-space"> </span></b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@gpfsug.org"><span style="color:#0563C1">gpfsug-discuss@gpfsug.org</span></a>><br>
<b>Date:<span class="apple-converted-space"> </span></b>Tuesday, February 14, 2023 at 10:49 AM<br>
<b>To:<span class="apple-converted-space"> </span></b>"<a href="mailto:gpfsug-discuss@gpfsug.org"><span style="color:#0563C1">gpfsug-discuss@gpfsug.org</span></a>" <<a href="mailto:gpfsug-discuss@gpfsug.org"><span style="color:#0563C1">gpfsug-discuss@gpfsug.org</span></a>><br>
<b>Subject:<span class="apple-converted-space"> </span></b>[EXTERNAL] Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:1.0pt;color:white">Hi! I started with 5. 1. 6. 0 and now am at [root@ ogpfs1 ~]# mmfsadm dump version Dump level: verbose Build branch "5. 1. 6. 1 ". the messages started from the beginning From: gpfsug-discuss
<gpfsug-discuss-bounces@ gpfsug. org> On<span class="apple-converted-space"> </span></span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerStart</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
</div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;border-radius: 4px">
<tbody>
<tr>
<td style="padding:12.0pt 0cm 12.0pt 0cm">
<table class="MsoNormalTable" border="1" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;background:#D0D8DC;border:none;border-top:solid #90A4AE 3.0pt">
<tbody>
<tr>
<td valign="top" style="border:none;padding:0cm 7.5pt 3.75pt 4.5pt">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left">
<tbody>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<div>
<div>
<p class="MsoNormal"><b><span style="font-size:10.5pt;font-family:"Arial",sans-serif">This Message Is From an External Sender</span></b><o:p></o:p></p>
</div>
</div>
</td>
</tr>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial",sans-serif">This message came from outside your organization.</span><o:p></o:p></p>
</div>
</div>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerEnd</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Hi!<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">I started with 5.1.6.0 and now am at [root@ogpfs1 ~]# mmfsadm dump version<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Dump level: verbose<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Build branch "5.1.6.1 ".<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">the messages started from the beginning<span class="apple-converted-space"> </span><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<div>
<div>
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span class="apple-converted-space"><span lang="EN-US"> </span></span><span lang="EN-US">gpfsug-discuss <<a href="mailto:gpfsug-discuss-bounces@gpfsug.org"><span style="color:#0563C1">gpfsug-discuss-bounces@gpfsug.org</span></a>><span class="apple-converted-space"> </span><b>On
Behalf Of<span class="apple-converted-space"> </span></b>Christian Vieser<br>
<b>Sent:</b><span class="apple-converted-space"> </span>Dienstag, 14. Februar 2023 15:34<br>
<b>To:</b><span class="apple-converted-space"> </span><a href="mailto:gpfsug-discuss@gpfsug.org"><span style="color:#0563C1">gpfsug-discuss@gpfsug.org</span></a><br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<p><span lang="EN-US" style="font-size:8.5pt;font-family:Menlo-Regular">What version of Spectrum Scale is running there? Do these errors appear since your last version update?</span><span lang="EN-US"><o:p></o:p></span></p>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Am 14.02.23 um 14:09 schrieb Walter Sklenka:<o:p></o:p></span></p>
</div>
</div>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Dear Collegues!<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">May I ask if anyone has a hint what could be the reason for Critical Thread Watchdog warnings for Disk Leases Threads?<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Is this a “local node” Problem or a network problem ?<span class="apple-converted-space"> </span><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">I see these messages sometimes arriving when NSD Servers which also serve as NFS servers when they get under heavy NFS load<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""> </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""> </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""> </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">Following is an excerpt from mmfs.log.latest</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""> </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">2023-02-14_12:06:53.235+0100: [N] Disk lease period expired 0.040 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">2023-02-14_12:06:53.600+0100: [W] ------------------[GPFS Critical Thread Watchdog]------------------</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">2023-02-14_12:06:53.600+0100: [W] PID: 7294 State: R (DiskLeaseThread) is overloaded for more than 8 seconds</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">2023-02-14_12:06:53.600+0100: [W] counter: 0 (mark-idle: 0 mark-active: 0 pre-work: 0 post-work: 0) sched: (nvcsw: 0 nivcsw: 8)</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">2023-02-14_12:06:53.600+0100: [W] Call Trace(PID: 7294):</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">2023-02-14_12:06:53.600+0100: [W] #0: 0x000055CABDF49521 BaseMutexClass::release() + 0x12 at ??:0</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">2023-02-14_12:06:53.600+0100: [W] #1: 0xB1557721BBABD900 _etext + 0xB154F7E646041C0E at ??:0</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">2023-02-14_12:07:09.554+0100: [N] Disk lease reacquired in cluster xxx-cluster.</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">2023-02-14_12:07:09.554+0100: [N] Disk lease period expired 5.680 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">2023-02-14_12:07:11.605+0100: [N] Disk lease reacquired in cluster xxx-cluster.</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">2023-02-14_12:10:55.990+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">2023-02-14_12:10:55.990+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">2023-02-14_12:30:58.756+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">2023-02-14_12:30:58.756+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">2023-02-14_13:10:55.988+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">2023-02-14_13:10:55.989+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">2023-02-14_13:21:40.892+0100: [N] Node 10.20.30.2 (ogpfs2-hs.local) lease renewal is overdue. Pinging to check if it is alive</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">2023-02-14_13:21:40.892+0100: [I] The TCP connection to IP address 10.20.30.2 ogpfs2-hs.local <c0n1>:[1] (socket 106) state: state=1 ca_state=0 snd_cwnd=10 snd_ssthresh=2147483647 unacked=0
probes=0 backoff=0 retransmits=0 rto=201000 rcv_ssthresh=1219344 rtt=121 rttvar=69 sacked=0 retrans=0 reordering=3 lost=0</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">2023-02-14_13:22:00.220+0100: [N] Disk lease period expired 0.010 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">2023-02-14_13:22:08.298+0100: [N] Disk lease reacquired in cluster xxx-cluster.</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">2023-02-14_13:30:58.760+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">2023-02-14_13:30:58.760+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">Mit freundlichen Grüßen<br>
<b><i>Walter Sklenka</i></b><br>
</span><b><i><span lang="EN-US" style="font-size:8.0pt;font-family:"Arial",sans-serif;color:gray">Technical Consultant</span></i></b><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
</blockquote>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.5pt;font-family:Menlo-Regular">_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at<span class="apple-converted-space"> </span></span><span lang="EN-US"><a href="http://gpfsug.org/"><span style="font-size:8.5pt;font-family:Menlo-Regular;color:#0563C1">gpfsug.org</span></a></span><span lang="EN-US" style="font-size:8.5pt;font-family:Menlo-Regular"><br>
</span><span lang="EN-US"><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org"><span style="font-size:8.5pt;font-family:Menlo-Regular;color:#0563C1">http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org</span></a><o:p></o:p></span></p>
</div>
</div>
</blockquote>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.5pt;font-family:Menlo-Regular">_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at gpfsug.org<br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org">http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org</a></span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
</body>
</html>