<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="DE-AT" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hi!<o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">I started with 5.1.6.0 and now am at [root@ogpfs1 ~]# mmfsadm dump version<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Dump level: verbose<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Build branch "5.1.6.1 ".<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">the messages started from the beginning <o:p>
</o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="mso-fareast-language:DE-AT">From:</span></b><span lang="EN-US" style="mso-fareast-language:DE-AT"> gpfsug-discuss <gpfsug-discuss-bounces@gpfsug.org>
<b>On Behalf Of </b>Christian Vieser<br>
<b>Sent:</b> Dienstag, 14. Februar 2023 15:34<br>
<b>To:</b> gpfsug-discuss@gpfsug.org<br>
<b>Subject:</b> Re: [gpfsug-discuss] Reasons for DiskLeaseThread Overloaded<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p>What version of Spectrum Scale is running there? Do these errors appear since your last version update?<span style="mso-fareast-language:DE-AT"><o:p></o:p></span></p>
<div>
<p class="MsoNormal">Am 14.02.23 um 14:09 schrieb Walter Sklenka:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span lang="EN-US">Dear Collegues!</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">May I ask if anyone has a hint what could be the reason for Critical Thread Watchdog warnings for Disk Leases Threads?</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">Is this a “local node” Problem or a network problem ?
</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">I see these messages sometimes arriving when NSD Servers which also serve as NFS servers when they get under heavy NFS load
</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New",serif"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New",serif"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New",serif"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New",serif">Following is an excerpt from mmfs.log.latest
</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New",serif"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New",serif">2023-02-14_12:06:53.235+0100: [N] Disk lease period expired 0.040 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New",serif">2023-02-14_12:06:53.600+0100: [W] ------------------[GPFS Critical Thread Watchdog]------------------</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New",serif">2023-02-14_12:06:53.600+0100: [W] PID: 7294 State: R (DiskLeaseThread) is overloaded for more than 8 seconds</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New",serif">2023-02-14_12:06:53.600+0100: [W] counter: 0 (mark-idle: 0 mark-active: 0 pre-work: 0 post-work: 0) sched: (nvcsw: 0 nivcsw: 8)</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New",serif">2023-02-14_12:06:53.600+0100: [W] Call Trace(PID: 7294):</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New",serif">2023-02-14_12:06:53.600+0100: [W] #0: 0x000055CABDF49521 BaseMutexClass::release() + 0x12 at ??:0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New",serif">2023-02-14_12:06:53.600+0100: [W] #1: 0xB1557721BBABD900 _etext + 0xB154F7E646041C0E at ??:0</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New",serif">2023-02-14_12:07:09.554+0100: [N] Disk lease reacquired in cluster xxx-cluster.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New",serif">2023-02-14_12:07:09.554+0100: [N] Disk lease period expired 5.680 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New",serif">2023-02-14_12:07:11.605+0100: [N] Disk lease reacquired in cluster xxx-cluster.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New",serif">2023-02-14_12:10:55.990+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New",serif">2023-02-14_12:10:55.990+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New",serif">2023-02-14_12:30:58.756+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New",serif">2023-02-14_12:30:58.756+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New",serif">2023-02-14_13:10:55.988+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New",serif">2023-02-14_13:10:55.989+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New",serif">2023-02-14_13:21:40.892+0100: [N] Node 10.20.30.2 (ogpfs2-hs.local) lease renewal is overdue. Pinging to check if it is alive</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New",serif">2023-02-14_13:21:40.892+0100: [I] The TCP connection to IP address 10.20.30.2 ogpfs2-hs.local <c0n1>:[1] (socket 106) state: state=1 ca_state=0 snd_cwnd=10 snd_ssthresh=2147483647
unacked=0 probes=0 backoff=0 retransmits=0 rto=201000 rcv_ssthresh=1219344 rtt=121 rttvar=69 sacked=0 retrans=0 reordering=3 lost=0</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New",serif">2023-02-14_13:22:00.220+0100: [N] Disk lease period expired 0.010 seconds ago in cluster xxx-cluster. Attempting to reacquire the lease.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New",serif">2023-02-14_13:22:08.298+0100: [N] Disk lease reacquired in cluster xxx-cluster.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New",serif">2023-02-14_13:30:58.760+0100: [I] Command: mmlspool /dev/fs4vm all -L -Y</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New",serif">2023-02-14_13:30:58.760+0100: [I] Command: successful mmlspool /dev/fs4vm all -L -Y</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black;mso-fareast-language:DE-AT">Mit freundlichen Grüßen<br>
<b><i>Walter Sklenka</i></b><br>
</span><b><i><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:gray;mso-fareast-language:DE-AT">Technical Consultant</span></i></b><span style="color:black;mso-fareast-language:DE-AT">
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:DE-AT"><o:p> </o:p></span></p>
</blockquote>
<p class="MsoNormal"><span style="mso-fareast-language:DE-AT"><o:p> </o:p></span></p>
</div>
</body>
</html>