<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:70.85pt 70.85pt 56.7pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">All,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">These messages like<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">[W] ------------------[GPFS Critical Thread Watchdog]------------------</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">indicate that a “critical thread”, in this case the lease thread, was apparently blocked for longer than expected. This is usually not caused by delays in the network, but possibly by excessive CPU load, blockage while accessing the local
file system, or possible mutex contention.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Do you have other samples of the message, with a more complete stack trace? Or was the instance below the only one?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> Felipe<o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">----<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Felipe Knop knop@us.ibm.com<br>
GPFS Development and Security<br>
IBM Systems<br>
IBM Building 008<br>
2455 South Rd, Poughkeepsie, NY 12601<o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">gpfsug-discuss <gpfsug-discuss-bounces@gpfsug.org> on behalf of Walter Sklenka <Walter.Sklenka@EDV-Design.at><br>
<b>Reply-To: </b>gpfsug main discussion list <gpfsug-discuss@gpfsug.org><br>
<b>Date: </b>Tuesday, February 14, 2023 at 10:49 AM<br>
<b>To: </b>"gpfsug-discuss@gpfsug.org" <gpfsug-discuss@gpfsug.org><br>
<b>Subject: </b>[EXTERNAL] Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">Hi! I started with 5. 1. 6. 0 and now am at [root@ ogpfs1 ~]# mmfsadm dump version Dump level: verbose Build branch "5. 1. 6. 1 ". the messages started from the
beginning From: gpfsug-discuss <gpfsug-discuss-bounces@ gpfsug. org> On <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerStart<o:p></o:p></span></p>
</div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;border-radius:4px">
<tbody>
<tr>
<td style="padding:12.0pt 0in 12.0pt 0in">
<table class="MsoNormalTable" border="1" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;background:#D0D8DC;border:none;border-top:solid #90A4AE 3.0pt">
<tbody>
<tr>
<td valign="top" style="border:none;padding:0in 7.5pt 3.75pt 4.5pt">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left">
<tbody>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<p class="MsoNormal"><b><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">This Message Is From an External Sender
<o:p></o:p></span></b></p>
</td>
</tr>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:black">This message came from outside your organization.
<o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerEnd<o:p></o:p></span></p>
</div>
<p class="MsoNormal">Hi!<o:p></o:p></p>
<p class="MsoNormal">I started with 5.1.6.0 and now am at [root@ogpfs1 ~]# mmfsadm dump version<o:p></o:p></p>
<p class="MsoNormal">Dump level: verbose<o:p></o:p></p>
<p class="MsoNormal">Build branch "5.1.6.1 ".<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">the messages started from the beginning <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="mso-fareast-language:DE-AT">From:</span></b><span style="mso-fareast-language:DE-AT"> gpfsug-discuss <gpfsug-discuss-bounces@gpfsug.org>
<b>On Behalf Of </b>Christian Vieser<br>
<b>Sent:</b> Dienstag, 14. Februar 2023 15:34<br>
<b>To:</b> gpfsug-discuss@gpfsug.org<br>
<b>Subject:</b> Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p>What version of Spectrum Scale is running there? Do these errors appear since your last version update?<o:p></o:p></p>
<div>
<p class="MsoNormal">Am 14.02.23 um 14:09 schrieb Walter Sklenka:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Dear Collegues!<o:p></o:p></p>
<p class="MsoNormal">May I ask if anyone has a hint what could be the reason for Critical Thread Watchdog warnings for Disk Leases Threads?<o:p></o:p></p>
<p class="MsoNormal">Is this a “local node” Problem or a network problem ? <o:p></o:p></p>
<p class="MsoNormal">I see these messages sometimes arriving when NSD Servers which also serve as NFS servers when they get under heavy NFS load
<o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">Following is an excerpt from mmfs.log.latest
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">2023-02-14_12:06:53.235+0100: [N] Disk lease period expired 0.040 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">2023-02-14_12:06:53.600+0100: [W] ------------------[GPFS Critical Thread Watchdog]------------------</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">2023-02-14_12:06:53.600+0100: [W] PID: 7294 State: R (DiskLeaseThread) is overloaded for more than 8 seconds</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">2023-02-14_12:06:53.600+0100: [W] counter: 0 (mark-idle: 0 mark-active: 0 pre-work: 0 post-work: 0) sched: (nvcsw: 0 nivcsw: 8)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">2023-02-14_12:06:53.600+0100: [W] Call Trace(PID: 7294):</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">2023-02-14_12:06:53.600+0100: [W] #0: 0x000055CABDF49521 BaseMutexClass::release() + 0x12 at ??:0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">2023-02-14_12:06:53.600+0100: [W] #1: 0xB1557721BBABD900 _etext + 0xB154F7E646041C0E at ??:0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">2023-02-14_12:07:09.554+0100: [N] Disk lease reacquired in cluster xxx-cluster.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">2023-02-14_12:07:09.554+0100: [N] Disk lease period expired 5.680 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">2023-02-14_12:07:11.605+0100: [N] Disk lease reacquired in cluster xxx-cluster.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">2023-02-14_12:10:55.990+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">2023-02-14_12:10:55.990+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">2023-02-14_12:30:58.756+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">2023-02-14_12:30:58.756+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">2023-02-14_13:10:55.988+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">2023-02-14_13:10:55.989+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">2023-02-14_13:21:40.892+0100: [N] Node 10.20.30.2 (ogpfs2-hs.local) lease renewal is overdue. Pinging to check if it is alive</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">2023-02-14_13:21:40.892+0100: [I] The TCP connection to IP address 10.20.30.2 ogpfs2-hs.local <c0n1>:[1] (socket 106) state: state=1 ca_state=0 snd_cwnd=10 snd_ssthresh=2147483647 unacked=0 probes=0
backoff=0 retransmits=0 rto=201000 rcv_ssthresh=1219344 rtt=121 rttvar=69 sacked=0 retrans=0 reordering=3 lost=0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">2023-02-14_13:22:00.220+0100: [N] Disk lease period expired 0.010 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">2023-02-14_13:22:08.298+0100: [N] Disk lease reacquired in cluster xxx-cluster.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">2023-02-14_13:30:58.760+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">2023-02-14_13:30:58.760+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black;mso-fareast-language:DE-AT">Mit freundlichen Grüßen<br>
<b><i>Walter Sklenka</i></b><br>
</span><b><i><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:gray;mso-fareast-language:DE-AT">Technical Consultant</span></i></b><span style="color:black;mso-fareast-language:DE-AT">
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:DE-AT"> </span><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span style="mso-fareast-language:DE-AT"> </span><o:p></o:p></p>
</div>
</body>
</html>