<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
I talked about it a lot in TS011616986.
<div><br>
</div>
<div>Part of the problem is we’re having a lot of strange problems at the same time, and so the different issues we’re having often come together (like one cause shows two symptoms).</div>
<div><br>
</div>
<div>I can’t remember if there was a case where I specifically mentioned the watchdog, or whether it was unexpectedly late lease times in general.</div>
<div><br>
<div>
<div style="color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
--<br>
#BlackLivesMatter</div>
<div style="color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
____<br>
|| \\UTGERS, <span class="Apple-tab-span" style="white-space: pre;"> </span> |---------------------------*O*---------------------------<br>
||_// the State<span class="Apple-tab-span" style="white-space: pre;"> </span> | Ryan Novosielski - novosirj@rutgers.edu<br>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus<br>
|| \\ of NJ<span class="Apple-tab-span" style="white-space: pre;"> </span> | Office of Advanced Research Computing - MSB C630, Newark<br>
`'</div>
</div>
<div><br>
<blockquote type="cite">
<div>On Feb 17, 2023, at 04:43, Walter Sklenka <Walter.Sklenka@EDV-Design.at> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span>Hi Ryan and Felipe!</span><span lang="EN-US"><o:p></o:p></span></div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US">Could you eventually tell me the case number if you remember it?<o:p></o:p></span></div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US">I opened the case and would reference to your case ID<o:p></o:p></span></div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US">Or shall I send you mine ?<o:p></o:p></span></div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US"><o:p> </o:p></span></div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="EN-US"><o:p> </o:p></span></div>
<div>
<div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(225, 225, 225); padding: 3pt 0cm 0cm;">
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><b><span lang="EN-US">From:</span></b><span lang="EN-US"><span class="Apple-converted-space"> </span>gpfsug-discuss <gpfsug-discuss-bounces@gpfsug.org><span class="Apple-converted-space"> </span><b>On
Behalf Of<span class="Apple-converted-space"> </span></b>Ryan Novosielski<br>
<b>Sent:</b><span class="Apple-converted-space"> </span>Freitag, 17. Februar 2023 06:43<br>
<b>To:</b><span class="Apple-converted-space"> </span>gpfsug main discussion list <gpfsug-discuss@gpfsug.org><br>
<b>Subject:</b><span class="Apple-converted-space"> </span>Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded<o:p></o:p></span></div>
</div>
</div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Thanks for this, Felipe.<span class="Apple-converted-space"> </span><o:p></o:p></div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">We’ve started seeing intermittent overdue leases in large numbers and don’t otherwise have an explanation for it, other than “look at your network,” which actually does show occasional
signs of strange behavior/higher-than-normal RTO values, but we’re not necessarily seeing those things happen at the same times as the lease issues. We’ve also seen “GPFS Critical Thread Watchdog” recently.<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">We had a case open about it, but didn’t draw any real conclusions. If any of our data might be helpful/if there’s a case we could reference to see if we’re also running into that,
we could provide a gpfs.snap.<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">FWIW, we are running 5.1.3-1 on the storage side (except one system that’s about to be upgraded that runs a combination of 5.0.3-2 and 5.0.5-1), and 5.1.6-0 (soon to be 5.1.6-1) on
the remote/client cluster side.<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div>
<div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="">--<br>
#BlackLivesMatter<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="">____<br>
|| <a href="file://utgers/" style="color: blue; text-decoration: underline;">\\UTGERS</a>, <span class="apple-tab-span"><span class="Apple-converted-space"> </span></span> |---------------------------*O*---------------------------<br>
||_// the State<span class="apple-tab-span"><span class="Apple-converted-space"> </span></span> | Ryan Novosielski - <a href="mailto:novosirj@rutgers.edu" style="color: blue; text-decoration: underline;">novosirj@rutgers.edu</a><br>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus<br>
|| \\ of NJ<span class="apple-tab-span"><span class="Apple-converted-space"> </span></span> | Office of Advanced Research Computing - MSB C630, Newark<br>
`'<o:p></o:p></span></div>
</div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><br>
<br>
<o:p></o:p></div>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;">
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">On Feb 16, 2023, at 12:02, Felipe Knop <<a href="mailto:knop@us.ibm.com" style="color: blue; text-decoration: underline;">knop@us.ibm.com</a>> wrote:<o:p></o:p></div>
</div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div>
<div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Walter,<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Thanks for the detailed description.<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">I don’t yet see anything glaringly incorrect on your configuration, but perhaps others might find something out of place.<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">I’d encourage you to open a case, since I spoke with a colleague yesterday, and he mentioned that he is working on a problem that may cause the lease thread to “loop” for a while.
That might cause the critical thread watchdog to flag the lease thread as taking too long to “check in”.<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Capturing gpfs.snap is important, since we’d be looking into all the<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">[W] ------------------[GPFS Critical Thread Watchdog]------------------</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">instances.<o:p></o:p></div>
</div>
<div>
<div>
<div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Thanks,<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> Felipe<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">----<o:p></o:p></div>
</div>
</div>
<div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Felipe Knop <a href="mailto:knop@us.ibm.com" style="color: blue; text-decoration: underline;"><span style="color: rgb(5, 99, 193);">knop@us.ibm.com</span></a><br>
GPFS Development and Security<br>
IBM Systems<br>
IBM Building 008<br>
2455 South Rd, Poughkeepsie, NY 12601<o:p></o:p></div>
</div>
</div>
</div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0cm 0cm;">
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><b><span style="font-size: 12pt;">From:<span class="apple-converted-space"> </span></span></b><span style="font-size: 12pt;">gpfsug-discuss <<a href="mailto:gpfsug-discuss-bounces@gpfsug.org" style="color: blue; text-decoration: underline;"><span style="color: rgb(5, 99, 193);">gpfsug-discuss-bounces@gpfsug.org</span></a>>
on behalf of Walter Sklenka <<a href="mailto:Walter.Sklenka@EDV-Design.at" style="color: blue; text-decoration: underline;"><span style="color: rgb(5, 99, 193);">Walter.Sklenka@EDV-Design.at</span></a>><br>
<b>Reply-To:<span class="apple-converted-space"> </span></b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@gpfsug.org" style="color: blue; text-decoration: underline;"><span style="color: rgb(5, 99, 193);">gpfsug-discuss@gpfsug.org</span></a>><br>
<b>Date:<span class="apple-converted-space"> </span></b>Thursday, February 16, 2023 at 9:16 AM<br>
<b>To:<span class="apple-converted-space"> </span></b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@gpfsug.org" style="color: blue; text-decoration: underline;"><span style="color: rgb(5, 99, 193);">gpfsug-discuss@gpfsug.org</span></a>><br>
<b>Subject:<span class="apple-converted-space"> </span></b>[EXTERNAL] Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded</span><o:p></o:p></div>
</div>
</div>
<div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
</div>
<div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 1pt; color: white;">Hi Felipe! Once again me. Thank you very much for the hint I did not open a PMR yet because I fear they will ask me/us if we are cracy<span class="apple-converted-space"> </span></span><span style="font-size: 1pt; font-family: "Segoe UI Emoji", sans-serif; color: white;">☹</span><span class="apple-converted-space"><span style="font-size: 1pt; color: white;"> </span></span><span style="font-size: 1pt; color: white;">I
did not tell the full story yet We have a 3 node cluster, 2 NSD servers o1,o2 (same site ) and g1 (different</span><o:p></o:p></div>
</div>
</div>
<div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 1pt; color: white;">ZjQcmQRYFpfptBannerStart</span><o:p></o:p></div>
</div>
</div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="100%" style="width: 418px; border-radius: 4px;">
<tbody>
<tr>
<td style="padding: 12pt 0cm;">
<table class="MsoNormalTable" border="1" cellspacing="0" cellpadding="0" width="100%" style="width: 418px; background-color: rgb(208, 216, 220); border-style: solid none none; border-top-width: 3pt; border-top-color: rgb(144, 164, 174);">
<tbody>
<tr>
<td valign="top" style="border: none; padding: 0cm 7.5pt 3.75pt 4.5pt;">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left">
<tbody>
<tr>
<td style="padding: 3pt 6pt;">
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><b><span style="font-size: 10.5pt; font-family: Arial, sans-serif;">This Message Is From an External Sender</span></b><o:p></o:p></div>
</div>
</td>
</tr>
<tr>
<td style="padding: 3pt 6pt;">
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 9pt; font-family: Arial, sans-serif;">This message came from outside your organization.</span><o:p></o:p></div>
</div>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 1pt; color: white;">ZjQcmQRYFpfptBannerEnd</span><o:p></o:p></div>
</div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Hi Felipe!<span class="apple-converted-space"> </span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Once again me. Thank you very much for the hint<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">I did not open a PMR yet because I fear they will ask me/us if we are cracy<span class="apple-converted-space"> </span><span style="font-family: "Segoe UI Emoji", sans-serif;">☹</span><span class="apple-converted-space"> </span> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">I did not tell the full story yet<span class="apple-converted-space"> </span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">We have a 3 node cluster, 2 NSD servers o1,o2 (same site ) and g1 (different site). (rhel 8.7)<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">All of them are Vmware VMs<span class="apple-converted-space"> </span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">O1 and o2 have each 4 NVME drives passed through , there is a software raid 5 made over these NVMEs , and from them made a single NSD , for a filesystem fs4vm (m,r=2 )<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";"> </span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">[root@ogpfs1 ras]# mmlscluster</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";"> </span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">GPFS cluster information</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">========================</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";"> GPFS cluster name: edvdesign-cluster.local</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";"> GPFS cluster id: 12147978822727803186</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";"> GPFS UID domain: edvdesign-cluster.local</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";"> Remote shell command: /usr/bin/ssh</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";"> Remote file copy command: /usr/bin/scp</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";"> Repository type: CCR</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";"> </span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">Node Daemon node name IP address Admin node name Designation</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">----------------------------------------------------------------------------</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";"> 1 ogpfs1-hs.local 10.20.30.1 ogpfs1-hs.local quorum-manager-perfmon</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";"> 2 ogpfs2-hs.local 10.20.30.2 ogpfs2-hs.local quorum-manager-perfmon</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";"> 3 ggpfsq.mgmt.cloudia <span class="apple-converted-space"> </span><a href="http://xxxx.other.net/" style="color: blue; text-decoration: underline;"><span style="color: rgb(5, 99, 193);">xxxx.other.net</span></a><span class="apple-converted-space"> </span>ggpfsq.mgmt.
a quorum-perfmon</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";"> </span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">[root@ogpfs1 ras]# mmlsconfig</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">Configuration data for cluster edvdesign-cluster.local:</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">-------------------------------------------------------</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">clusterName edvdesign-cluster.local</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">clusterId 12147978822727803186</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">autoload yes</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">profile gpfsProtocolRandomIO</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">dmapiFileHandleSize 32</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">minReleaseLevel 5.1.6.0</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">tscCmdAllowRemoteConnections no</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">ccrEnabled yes</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">cipherList AUTHONLY</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">sdrNotifyAuthEnabled yes</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">maxblocksize 16M</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">[cesNodes]</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">maxMBpS 5000</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">numaMemoryInterleave yes</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">enforceFilesetQuotaOnRoot yes</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">workerThreads 512</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">[common]</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">tscCmdPortRange 60000-61000</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">[srv]</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">verbsPorts mlx5_0/1 mlx5_1/1</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">[common]</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">cesSharedRoot /fs4vmware/cesSharedRoot</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">[srv]</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">maxFilesToCache 10000</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">maxStatCache 20000</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">[common]</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">verbsRdma enable</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">[ggpfsq]</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">verbsRdma disable</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">[common]</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">verbsRdmaSend yes</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">[ggpfsq]</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">verbsRdmaSend no</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">[common]</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">verbsRdmaCm enable</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">[ggpfsq]</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">verbsRdmaCm disable</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">[srv]</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">pagepool 32G</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">[common]</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">adminMode central</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";"> </span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">File systems in cluster edvdesign-cluster.local:</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">------------------------------------------------</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">/dev/fs4vm</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";"> </span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";"> </span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">[root@ogpfs1 ras]# mmlsdisk fs4vm -L</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">disk driver sector failure holds holds storage</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">name type size group metadata data status availability disk id pool remarks</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">------------ -------- ------ ----------- -------- ----- ------------- ------------ ------- ------------ ---------</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">ogpfs1_1 nsd 512 1 yes yes ready up 1 system desc</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">ogpfs2_1 nsd 512 2 yes yes ready up 2 system desc</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">ggpfsq_qdisk nsd 512 -1 no no ready up 3 system desc</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">Number of quorum disks: 3</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">Read quorum value: 2</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">Write quorum value: 2</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">And the two nodes o1 and o2 export the filesystem via CES NFS functions ( for VMware)<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">I think this isn´supported , that a NSD Server is also a CES Node?<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">And finally the RDMA Network :<span class="apple-converted-space"> </span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">The both NSD servers also have a Mellanox ConnectX-6 Lx dual port 25Gb adapter also via passthrough<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">And these interfaces we configured for rdma (RoCE) ,<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Last but not least: this network is not switched but direct attached ( 2x25Gb directly connected between the NSD nodes )<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">RDMA Connections between nodes:</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";"> Fabric 0 - Device mlx5_0 Port 1 Width 1x Speed EDR lid 0</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";"> hostname idx CM state VS buff RDMA_CT(ERR) RDMA_RCV_MB RDMA_SND_MB VS_CT(ERR) VS_SND_MB VS_RCV_MB
WAIT_CON_SLOT WAIT_NODE_SLOT</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";"> ogpfs2-hs.local 0 Y RTS (Y)256 478202 (0 ) 12728 67024 8864789(0 ) 22776 4643
0 0</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";"> Fabric 0 - Device mlx5_1 Port 1 Width 1x Speed EDR lid 0</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";"> hostname idx CM state VS buff RDMA_CT(ERR) RDMA_RCV_MB RDMA_SND_MB VS_CT(ERR) VS_SND_MB VS_RCV_MB
WAIT_CON_SLOT WAIT_NODE_SLOT</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";"> ogpfs2-hs.local 1 Y RTS (Y)256 477659 (0 ) 12489 67034 8864773(0 ) 22794 4639
0 0</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">[root@ogpfs1 ras]#</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";"> </span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">You mentioned that it might be a cpu contention : Maybe due to the VM layer (scheduling with other VMS) ? And wrong layout of VMs ( 8 vCPUs and 64GB Mem) [ esxis only single socket
with 32/64 cores HT)<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">And also the direct attached RDMA ( +DAEMON) network is also not good?<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Do you think IBM would say no to check such a configuration ?<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Best regards<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Walter<span class="apple-converted-space"> </span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(225, 225, 225); padding: 3pt 0cm 0cm;">
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><b>From:</b><span class="apple-converted-space"> </span>gpfsug-discuss <<a href="mailto:gpfsug-discuss-bounces@gpfsug.org" style="color: blue; text-decoration: underline;"><span style="color: rgb(5, 99, 193);">gpfsug-discuss-bounces@gpfsug.org</span></a>><span class="apple-converted-space"> </span><b>On
Behalf Of<span class="apple-converted-space"> </span></b>Felipe Knop<br>
<b>Sent:</b><span class="apple-converted-space"> </span>Mittwoch, 15. Februar 2023 15:59<br>
<b>To:</b><span class="apple-converted-space"> </span>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@gpfsug.org" style="color: blue; text-decoration: underline;"><span style="color: rgb(5, 99, 193);">gpfsug-discuss@gpfsug.org</span></a>><br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded<o:p></o:p></div>
</div>
</div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Walter,<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Thanks for the details.<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">The stack trace below captures the lease thread in the middle of sending the “lease” RPC. This operation normally is not blocking, and we do not often block while sending the RPC.
But the stack trace “does not show” whether there was anything blocking the thread prior to the point where the RPCs are sent.<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">At a first glance:<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">2023-02-14_19:44:07.430+0100: [W] counter: 0 (mark-idle: 0 mark-active: 0 pre-work: 0 post-work: 0) sched: (nvcsw: 0 nivcsw:
10)</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">I believe<span class="apple-converted-space"><b> </b></span><b>nivcsw: 10</b><span class="apple-converted-space"> </span>means that the thread was scheduled out of the CPU involuntarily,
possibly indicating that there is some CPU contention going on.<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Could you open a case to get debug data collected? If the problem can be recreated, I think we’ll need a recreate of the problem with traces enabled.<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Thanks,<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> Felipe<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div>
<div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">----<o:p></o:p></div>
</div>
</div>
<div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Felipe Knop <a href="mailto:knop@us.ibm.com" style="color: blue; text-decoration: underline;"><span style="color: rgb(5, 99, 193);">knop@us.ibm.com</span></a><br>
GPFS Development and Security<br>
IBM Systems<br>
IBM Building 008<br>
2455 South Rd, Poughkeepsie, NY 12601<o:p></o:p></div>
</div>
</div>
</div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0cm 0cm;">
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><b><span style="font-size: 12pt;">From:<span class="apple-converted-space"> </span></span></b><span style="font-size: 12pt;">gpfsug-discuss <<a href="mailto:gpfsug-discuss-bounces@gpfsug.org" style="color: blue; text-decoration: underline;"><span style="color: rgb(5, 99, 193);">gpfsug-discuss-bounces@gpfsug.org</span></a>>
on behalf of Walter Sklenka <<a href="mailto:Walter.Sklenka@EDV-Design.at" style="color: blue; text-decoration: underline;"><span style="color: rgb(5, 99, 193);">Walter.Sklenka@EDV-Design.at</span></a>><br>
<b>Reply-To:<span class="apple-converted-space"> </span></b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@gpfsug.org" style="color: blue; text-decoration: underline;"><span style="color: rgb(5, 99, 193);">gpfsug-discuss@gpfsug.org</span></a>><br>
<b>Date:<span class="apple-converted-space"> </span></b>Wednesday, February 15, 2023 at 4:23 AM<br>
<b>To:<span class="apple-converted-space"> </span></b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@gpfsug.org" style="color: blue; text-decoration: underline;"><span style="color: rgb(5, 99, 193);">gpfsug-discuss@gpfsug.org</span></a>><br>
<b>Subject:<span class="apple-converted-space"> </span></b>[EXTERNAL] Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded</span><o:p></o:p></div>
</div>
</div>
<div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
</div>
<div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 1pt; color: white;">Hi! This is a „full“ sequence in mmfs. log. latest Fortunately this was also the last event until now (yesterday evening) Maybe you can
have a look? 2023-02-14_19: 43: 51. 474+0100: [N] Disk lease period expired 0. 030 seconds ago in cluster<span class="apple-converted-space"> </span></span><o:p></o:p></div>
</div>
</div>
<div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 1pt; color: white;">ZjQcmQRYFpfptBannerStart</span><o:p></o:p></div>
</div>
</div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="100%" style="width: 418px; border-radius: 4px;">
<tbody>
<tr>
<td style="padding: 12pt 0cm;">
<table class="MsoNormalTable" border="1" cellspacing="0" cellpadding="0" width="100%" style="width: 418px; background-color: rgb(208, 216, 220); border-style: solid none none; border-top-width: 3pt; border-top-color: rgb(144, 164, 174);">
<tbody>
<tr>
<td valign="top" style="border: none; padding: 0cm 7.5pt 3.75pt 4.5pt;">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left">
<tbody>
<tr>
<td style="padding: 3pt 6pt;">
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><b><span style="font-size: 10.5pt; font-family: Arial, sans-serif;">This Message Is From an External Sender</span></b><o:p></o:p></div>
</div>
</td>
</tr>
<tr>
<td style="padding: 3pt 6pt;">
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 9pt; font-family: Arial, sans-serif;">This message came from outside your organization.</span><o:p></o:p></div>
</div>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 1pt; color: white;">ZjQcmQRYFpfptBannerEnd</span><o:p></o:p></div>
</div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Hi!<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">This is a „full“ sequence in mmfs.log.latest<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Fortunately this was also the last event until now (yesterday evening)<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Maybe you can have a look?<span class="apple-converted-space"> </span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">2023-02-14_19:43:51.474+0100: [N] Disk lease period expired 0.030 seconds ago in cluster xxx-cluster. Attempting to reacquire
the lease.</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">2023-02-14_19:44:07.430+0100: [W] ------------------[GPFS Critical Thread Watchdog]------------------</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">2023-02-14_19:44:07.430+0100: [W] PID: 7294 State: R (DiskLeaseThread) is overloaded for more than 8 seconds</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">2023-02-14_19:44:07.430+0100: [W] counter: 0 (mark-idle: 0 mark-active: 0 pre-work: 0 post-work: 0) sched: (nvcsw: 0 nivcsw:
10)</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">2023-02-14_19:44:07.430+0100: [W] Call Trace(PID: 7294):</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">2023-02-14_19:44:07.431+0100: [W] #0: 0x000055CABE4A56AB NodeConn::sendMessage(TcpConn**, iovec*, int, unsigned char, int,
int, int, unsigned int, DestTag*, int*, unsigned long long*, unsigned long long*, unsi</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">gned int*, CondvarName, vsendCallback_t*) + 0x42B at ??:0</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">2023-02-14_19:44:07.432+0100: [W] #1: 0x000055CABE4A595F llc_send_msg(ClusterConfiguration*, NodeAddr, iovec*, int, unsigned
char, int, int, int, unsigned int, DestTag*, int*, TcpConn**, unsigned long long*, u</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">nsigned long long*, unsigned int*, CondvarName, vsendCallback_t*, int, unsigned int) + 0xDF at ??:0</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">2023-02-14_19:44:07.437+0100: [W] #2: 0x000055CABE479A55 MsgRecord::send() + 0x1345 at ??:0</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">2023-02-14_19:44:07.438+0100: [W] #3: 0x000055CABE47A169 tscSendInternal(ClusterConfiguration*, unsigned int, unsigned char,
int, int, NodeAddr*, TscReply*, TscScatteredBuff*, int, int (*)(void*, ClusterConfig</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">uration*, int, NodeAddr*, TscReply*), void*, ChainedCallback**, __va_list_tag*) + 0x339 at ??:0</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">2023-02-14_19:44:07.439+0100: [W] #4: 0x000055CABE47C39A tscSendWithCallback(ClusterConfiguration*, unsigned int, unsigned
char, int, NodeAddr*, TscReply*, int (*)(void*, ClusterConfiguration*, int, NodeAddr*,</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">TscReply*), void*, void**, int, ...) + 0x1DA at ??:0</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">2023-02-14_19:44:07.440+0100: [W] #5: 0x000055CABE5F9853 MyLeaseState::renewLease(NodeAddr, TickTime) + 0x6E3 at ??:0</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">2023-02-14_19:44:07.440+0100: [W] #6: 0x000055CABE5FA682 ClusterConfiguration::checkAndRenewLease(TickTime) + 0x192 at ??:0</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">2023-02-14_19:44:07.441+0100: [W] #7: 0x000055CABE5FAAC6 ClusterConfiguration::RunLeaseChecks(void*) + 0x366 at ??:0</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">2023-02-14_19:44:07.441+0100: [W] #8: 0x000055CABDF2B662 Thread::callBody(Thread*) + 0x42 at ??:0</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">2023-02-14_19:44:07.441+0100: [W] #9: 0x000055CABDF18680 Thread::callBodyWrapper(Thread*) + 0xA0 at ??:0</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">2023-02-14_19:44:07.441+0100: [W] #10: 0x00007F3B7563D1CA start_thread + 0xEA at ??:0</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">2023-02-14_19:44:07.441+0100: [W] #11: 0x00007F3B7435BE73 __GI___clone + 0x43 at ??:0</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">2023-02-14_19:44:10.512+0100: [N] Disk lease reacquired in cluster xxx-cluster.</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">2023-02-14_19:44:10.512+0100: [N] Disk lease period expired 7.970 seconds ago in cluster xxx-cluster. Attempting to reacquire
the lease.</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8pt; font-family: "Courier New";">2023-02-14_19:44:12.563+0100: [N] Disk lease reacquired in cluster xxx-cluster.</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Thank you very much!<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Best regards<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Walter<span class="apple-converted-space"> </span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(225, 225, 225); padding: 3pt 0cm 0cm;">
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><b>From:</b><span class="apple-converted-space"> </span>gpfsug-discuss <<a href="mailto:gpfsug-discuss-bounces@gpfsug.org" style="color: blue; text-decoration: underline;"><span style="color: rgb(5, 99, 193);">gpfsug-discuss-bounces@gpfsug.org</span></a>><span class="apple-converted-space"> </span><b>On
Behalf Of<span class="apple-converted-space"> </span></b>Felipe Knop<br>
<b>Sent:</b><span class="apple-converted-space"> </span>Mittwoch, 15. Februar 2023 00:06<br>
<b>To:</b><span class="apple-converted-space"> </span>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@gpfsug.org" style="color: blue; text-decoration: underline;"><span style="color: rgb(5, 99, 193);">gpfsug-discuss@gpfsug.org</span></a>><br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded<o:p></o:p></div>
</div>
</div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">All,<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">These messages like<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: "Courier New";">[W] ------------------[GPFS Critical Thread Watchdog]------------------</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">indicate that a “critical thread”, in this case the lease thread, was apparently blocked for longer than expected. This is usually not caused by delays in the network, but possibly
by excessive CPU load, blockage while accessing the local file system, or possible mutex contention.<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Do you have other samples of the message, with a more complete stack trace? Or was the instance below the only one?<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> Felipe<o:p></o:p></div>
</div>
<div>
<div>
<div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">----<o:p></o:p></div>
</div>
</div>
<div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Felipe Knop <a href="mailto:knop@us.ibm.com" style="color: blue; text-decoration: underline;"><span style="color: rgb(5, 99, 193);">knop@us.ibm.com</span></a><br>
GPFS Development and Security<br>
IBM Systems<br>
IBM Building 008<br>
2455 South Rd, Poughkeepsie, NY 12601<o:p></o:p></div>
</div>
</div>
</div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0cm 0cm;">
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><b><span style="font-size: 12pt;">From:<span class="apple-converted-space"> </span></span></b><span style="font-size: 12pt;">gpfsug-discuss <<a href="mailto:gpfsug-discuss-bounces@gpfsug.org" style="color: blue; text-decoration: underline;"><span style="color: rgb(5, 99, 193);">gpfsug-discuss-bounces@gpfsug.org</span></a>>
on behalf of Walter Sklenka <<a href="mailto:Walter.Sklenka@EDV-Design.at" style="color: blue; text-decoration: underline;"><span style="color: rgb(5, 99, 193);">Walter.Sklenka@EDV-Design.at</span></a>><br>
<b>Reply-To:<span class="apple-converted-space"> </span></b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@gpfsug.org" style="color: blue; text-decoration: underline;"><span style="color: rgb(5, 99, 193);">gpfsug-discuss@gpfsug.org</span></a>><br>
<b>Date:<span class="apple-converted-space"> </span></b>Tuesday, February 14, 2023 at 10:49 AM<br>
<b>To:<span class="apple-converted-space"> </span></b>"<a href="mailto:gpfsug-discuss@gpfsug.org" style="color: blue; text-decoration: underline;"><span style="color: rgb(5, 99, 193);">gpfsug-discuss@gpfsug.org</span></a>" <<a href="mailto:gpfsug-discuss@gpfsug.org" style="color: blue; text-decoration: underline;"><span style="color: rgb(5, 99, 193);">gpfsug-discuss@gpfsug.org</span></a>><br>
<b>Subject:<span class="apple-converted-space"> </span></b>[EXTERNAL] Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded</span><o:p></o:p></div>
</div>
</div>
<div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
</div>
<div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 1pt; color: white;">Hi! I started with 5. 1. 6. 0 and now am at [root@ ogpfs1 ~]# mmfsadm dump version Dump level: verbose Build branch "5. 1. 6. 1 ". the messages
started from the beginning From: gpfsug-discuss <gpfsug-discuss-bounces@ gpfsug. org> On<span class="apple-converted-space"> </span></span><o:p></o:p></div>
</div>
</div>
<div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 1pt; color: white;">ZjQcmQRYFpfptBannerStart</span><o:p></o:p></div>
</div>
</div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="100%" style="width: 418px; border-radius: 4px;">
<tbody>
<tr>
<td style="padding: 12pt 0cm;">
<table class="MsoNormalTable" border="1" cellspacing="0" cellpadding="0" width="100%" style="width: 418px; background-color: rgb(208, 216, 220); border-style: solid none none; border-top-width: 3pt; border-top-color: rgb(144, 164, 174);">
<tbody>
<tr>
<td valign="top" style="border: none; padding: 0cm 7.5pt 3.75pt 4.5pt;">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left">
<tbody>
<tr>
<td style="padding: 3pt 6pt;">
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><b><span style="font-size: 10.5pt; font-family: Arial, sans-serif;">This Message Is From an External Sender</span></b><o:p></o:p></div>
</div>
</td>
</tr>
<tr>
<td style="padding: 3pt 6pt;">
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 9pt; font-family: Arial, sans-serif;">This message came from outside your organization.</span><o:p></o:p></div>
</div>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 1pt; color: white;">ZjQcmQRYFpfptBannerEnd</span><o:p></o:p></div>
</div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Hi!<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">I started with 5.1.6.0 and now am at [root@ogpfs1 ~]# mmfsadm dump version<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Dump level: verbose<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Build branch "5.1.6.1 ".<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">the messages started from the beginning<span class="apple-converted-space"> </span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div>
<div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(225, 225, 225); padding: 3pt 0cm 0cm;">
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><b>From:</b><span class="apple-converted-space"> </span>gpfsug-discuss <<a href="mailto:gpfsug-discuss-bounces@gpfsug.org" style="color: blue; text-decoration: underline;"><span style="color: rgb(5, 99, 193);">gpfsug-discuss-bounces@gpfsug.org</span></a>><span class="apple-converted-space"> </span><b>On
Behalf Of<span class="apple-converted-space"> </span></b>Christian Vieser<br>
<b>Sent:</b><span class="apple-converted-space"> </span>Dienstag, 14. Februar 2023 15:34<br>
<b>To:</b><span class="apple-converted-space"> </span><a href="mailto:gpfsug-discuss@gpfsug.org" style="color: blue; text-decoration: underline;"><span style="color: rgb(5, 99, 193);">gpfsug-discuss@gpfsug.org</span></a><br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded<o:p></o:p></div>
</div>
</div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<p><span style="font-size: 8.5pt; font-family: Menlo-Regular, serif;">What version of Spectrum Scale is running there? Do these errors appear since your last version update?<o:p></o:p></span></p>
<div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Am 14.02.23 um 14:09 schrieb Walter Sklenka:<o:p></o:p></div>
</div>
</div>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;">
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Dear Collegues!<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">May I ask if anyone has a hint what could be the reason for Critical Thread Watchdog warnings for Disk Leases Threads?<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Is this a “local node” Problem or a network problem ?<span class="apple-converted-space"> </span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">I see these messages sometimes arriving when NSD Servers which also serve as NFS servers when they get under heavy NFS load<o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: "Courier New";"> </span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: "Courier New";"> </span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: "Courier New";"> </span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: "Courier New";">Following is an excerpt from mmfs.log.latest</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: "Courier New";"> </span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: "Courier New";">2023-02-14_12:06:53.235+0100: [N] Disk lease period expired 0.040 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: "Courier New";">2023-02-14_12:06:53.600+0100: [W] ------------------[GPFS Critical Thread Watchdog]------------------</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: "Courier New";">2023-02-14_12:06:53.600+0100: [W] PID: 7294 State: R (DiskLeaseThread) is overloaded for more than 8 seconds</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: "Courier New";">2023-02-14_12:06:53.600+0100: [W] counter: 0 (mark-idle: 0 mark-active: 0 pre-work: 0 post-work: 0) sched: (nvcsw: 0 nivcsw: 8)</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: "Courier New";">2023-02-14_12:06:53.600+0100: [W] Call Trace(PID: 7294):</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: "Courier New";">2023-02-14_12:06:53.600+0100: [W] #0: 0x000055CABDF49521 BaseMutexClass::release() + 0x12 at ??:0</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: "Courier New";">2023-02-14_12:06:53.600+0100: [W] #1: 0xB1557721BBABD900 _etext + 0xB154F7E646041C0E at ??:0</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: "Courier New";">2023-02-14_12:07:09.554+0100: [N] Disk lease reacquired in cluster xxx-cluster.</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: "Courier New";">2023-02-14_12:07:09.554+0100: [N] Disk lease period expired 5.680 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: "Courier New";">2023-02-14_12:07:11.605+0100: [N] Disk lease reacquired in cluster xxx-cluster.</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: "Courier New";">2023-02-14_12:10:55.990+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: "Courier New";">2023-02-14_12:10:55.990+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: "Courier New";">2023-02-14_12:30:58.756+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: "Courier New";">2023-02-14_12:30:58.756+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: "Courier New";">2023-02-14_13:10:55.988+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: "Courier New";">2023-02-14_13:10:55.989+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: "Courier New";">2023-02-14_13:21:40.892+0100: [N] Node 10.20.30.2 (ogpfs2-hs.local) lease renewal is overdue. Pinging to check if it is alive</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: "Courier New";">2023-02-14_13:21:40.892+0100: [I] The TCP connection to IP address 10.20.30.2 ogpfs2-hs.local <c0n1>:[1] (socket 106) state: state=1 ca_state=0
snd_cwnd=10 snd_ssthresh=2147483647 unacked=0 probes=0 backoff=0 retransmits=0 rto=201000 rcv_ssthresh=1219344 rtt=121 rttvar=69 sacked=0 retrans=0 reordering=3 lost=0</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: "Courier New";">2023-02-14_13:22:00.220+0100: [N] Disk lease period expired 0.010 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: "Courier New";">2023-02-14_13:22:08.298+0100: [N] Disk lease reacquired in cluster xxx-cluster.</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: "Courier New";">2023-02-14_13:30:58.760+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-family: "Courier New";">2023-02-14_13:30:58.760+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y</span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 10pt; font-family: Arial, sans-serif;">Mit freundlichen Grüßen<br>
<b><i>Walter Sklenka</i></b><br>
</span><b><i><span style="font-size: 8pt; font-family: Arial, sans-serif; color: gray;">Technical Consultant</span></i></b><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 12pt;"> </span><o:p></o:p></div>
</div>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
</blockquote>
<div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div>
</div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 8.5pt; font-family: Menlo-Regular, serif;">_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at<span class="apple-converted-space"> </span></span><a href="http://gpfsug.org/" style="color: blue; text-decoration: underline;"><span style="font-size: 8.5pt; font-family: Menlo-Regular, serif; color: rgb(5, 99, 193);">gpfsug.org</span></a><span style="font-size: 8.5pt; font-family: Menlo-Regular, serif;"><br>
</span><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org" style="color: blue; text-decoration: underline;"><span style="font-size: 8.5pt; font-family: Menlo-Regular, serif; color: rgb(5, 99, 193);">http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org</span></a><o:p></o:p></div>
</div>
</blockquote>
</div>
<div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div>
</div>
</div>
<span style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">_______________________________________________</span><br style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<span style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">gpfsug-discuss
mailing list</span><br style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<span style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">gpfsug-discuss
at gpfsug.org</span><br style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<span style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org</span></div>
</blockquote>
</div>
<br>
</div>
</body>
</html>