<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(50, 49, 48); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, system-ui, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255); display: inline !important">Frederick,</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(50, 49, 48); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, system-ui, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255); display: inline !important"><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(50, 49, 48); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, system-ui, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255); display: inline !important">Yes
on both counts! - mmdf is showing pretty uniform (ie 5 NSDs out of 30 report 65% free; All others are uniform at 58% free)...</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(50, 49, 48); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, system-ui, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255); display: inline !important"><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(50, 49, 48); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, system-ui, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255); display: inline !important">NSD
servers per disks are called in round-robin fashion as well, for example:</span></div>
<div style=""><span style="display: inline !important;"><br>
</span></div>
<div style=""><span style="display: inline !important;">
<p style="margin: 0px;"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> </span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">gpfs1
</span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
</span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">tier2_001</span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
</span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">nsd02-ib,nsd03-ib,nsd04-ib,tsm01-ib,nsd01-ib</span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> </span></p>
<p style="margin: 0px;"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> </span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">gpfs1
</span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
</span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">tier2_002</span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
</span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">nsd03-ib,nsd04-ib,tsm01-ib,nsd01-ib,nsd02-ib</span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> </span></p>
<p style="margin: 0px;"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> </span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">gpfs1
</span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
</span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">tier2_003</span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
</span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">nsd04-ib,tsm01-ib,nsd01-ib,nsd02-ib,nsd03-ib</span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> </span></p>
<p style="margin: 0px;"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> </span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">gpfs1
</span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
</span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">tier2_004</span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
</span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">tsm01-ib,nsd01-ib,nsd02-ib,nsd03-ib,nsd04-ib</span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> </span></p>
<br>
</span></div>
<div style=""><span style="display: inline !important;">Any other potential culprits to investigate?</span></div>
<div style=""><span style="display: inline !important;"><br>
</span></div>
<div style=""><span style="display: inline !important;">I do notice nsd03/nsd04 have long waiters, but nsd01 doesn't (nsd02-ib is offline for now):
<p style="font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; line-height: normal; margin: 0px;">
<span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">[nsd03-ib ~]# mmdiag --waiters</span></p>
<p style="margin: 0px;"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">=== mmdiag: waiters ===</span></p>
<p style="margin: 0px;"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Waiting 6.5113 sec since 17:17:33, monitored, thread 4175 NSDThread: for I/O completion</span></p>
<p style="margin: 0px;"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Waiting 6.3810 sec since 17:17:33, monitored, thread 4127 NSDThread: for I/O completion</span></p>
<p style="margin: 0px;"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Waiting 6.1959 sec since 17:17:34, monitored, thread 4144 NSDThread: for I/O completion</span></p>
<p style="margin: 0px;"><br>
</p>
<p style="margin: 0px;"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">nsd04-ib:</span></p>
<p style="margin: 0px;"></p>
<p style="margin: 0px;"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Waiting 13.1386 sec since 17:19:09, monitored, thread 9971 NSDThread: for I/O completion</span></p>
<p style="margin: 0px;"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Waiting 10.3562 sec since 17:19:12, monitored, thread 9958 NSDThread: for I/O completion</span></p>
<p style="margin: 0px;"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Waiting 10.0338 sec since 17:19:12, monitored, thread 9951 NSDThread: for I/O completion</span></p>
<br>
<p></p>
<p style="margin: 0px;"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">tsm01-ib:</span></p>
<p style="margin: 0px;"></p>
<p style="margin: 0px;"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Waiting 8.1211 sec since 17:20:24, monitored, thread 3644 NSDThread: for I/O completion</span></p>
<p style="margin: 0px;"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Waiting 7.6690 sec since 17:20:24, monitored, thread 3641 NSDThread: for I/O completion</span></p>
<p style="margin: 0px;"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Waiting 7.4969 sec since 17:20:24, monitored, thread 3658 NSDThread: for I/O completion</span></p>
<p style="margin: 0px;"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Waiting 7.3573 sec since 17:20:24, monitored, thread 3642 NSDThread: for I/O completion</span></p>
<br>
<p></p>
<p style="margin: 0px;"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">nsd01-ib:</span></p>
<p style="font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; line-height: normal; margin: 0px;">
</p>
<p style="margin: 0px;"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Waiting 0.2548 sec since 17:21:47, monitored, thread 30513 NSDThread: for I/O completion</span></p>
<p style="font-variant-numeric: normal; font-variant-east-asian: normal; font-stretch: normal; line-height: normal; margin: 0px;">
<span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Waiting 0.1502 sec since 17:21:47, monitored, thread 30529 NSDThread: for I/O completion</span></p>
<br>
<p></p>
</span></div>
<div>
<div style=""><br>
</div>
<div id="Signature">
<div></div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Arial,Helvetica,sans-serif">
<p style="margin-top:0px; margin-bottom:0px"></p>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<span style="font-family:Arial,sans-serif; font-size:13.3333px">Thanks,</span><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<font face="Calibri,Arial,Helvetica,sans-serif"><br>
</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<font face="Calibri,Arial,Helvetica,sans-serif">
<p style="color:rgb(32,31,30); text-align:start; background-color:rgb(255,255,255); font-size:11pt; font-family:Calibri,sans-serif; margin:0px">
<b><span style="margin:0px; font-size:10pt; font-family:Arial,sans-serif; color:black">Oluwasijibomi (Siji) Saula</span></b></p>
<p style="color:rgb(32,31,30); text-align:start; background-color:rgb(255,255,255); font-size:11pt; font-family:Calibri,sans-serif; margin:0px">
<span style="margin:0px; font-size:10pt; font-family:Arial,sans-serif; color:black">HPC Systems Administrator / Information Technology</span><span style="margin:0px; font-family:"Times New Roman",serif; color:black"></span></p>
<p style="color:rgb(32,31,30); text-align:start; background-color:rgb(255,255,255); font-size:11pt; font-family:Calibri,sans-serif; margin:0px">
<span style="margin:0px; font-size:10pt; font-family:Arial,sans-serif; color:black"> </span><span style="margin:0px; font-family:"Times New Roman",serif; color:black"></span></p>
<p style="color:rgb(32,31,30); text-align:start; background-color:rgb(255,255,255); font-size:11pt; font-family:Calibri,sans-serif; margin:0px">
<span style="margin:0px; font-size:10pt; font-family:Arial,sans-serif; color:black"><span style="font-family:Arial,sans-serif; background-color:rgb(255,255,255); display:inline!important">Research 2 Building</span> 220B / Fargo ND 58108-6050</span></p>
<p style="color:rgb(32,31,30); text-align:start; background-color:rgb(255,255,255); font-size:11pt; font-family:Calibri,sans-serif; margin:0px">
<span style="margin:0px; font-size:10pt; font-family:Arial,sans-serif; color:black">p: 701.231.7749 /<span> </span><a href="http://www.ndsu.edu/" target="_blank" rel="noopener noreferrer" style="margin:0px"><span style="margin:0px; color:purple">www.ndsu.edu</span></a></span><span style="margin:0px; font-family:"Times New Roman",serif; color:black"></span></p>
<p style="color:rgb(32,31,30); text-align:start; background-color:rgb(255,255,255); font-size:11pt; font-family:Calibri,sans-serif; margin:0px">
<span style="margin:0px; font-size:10pt; font-family:Arial,sans-serif; color:black"> </span><span style="margin:0px; font-family:"Times New Roman",serif; color:black"></span></p>
<p style="color:rgb(32,31,30); text-align:start; background-color:rgb(255,255,255); font-size:11pt; font-family:Calibri,sans-serif; margin:0px">
<span style="margin:0px; color:rgb(31,73,125)"><img class="EmojiInsert" style="margin:0px; width:140.99pt; height:26.24pt" data-outlook-trace="F:0|T:1" src="cid:image001.gif@01D57DE0.91C300C0"></span></p>
</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<br>
</div>
<br>
<p style="margin-top:0px; margin-bottom:0px"></p>
</div>
</div>
</div>
<div id="appendonsend"></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> gpfsug-discuss-bounces@spectrumscale.org <gpfsug-discuss-bounces@spectrumscale.org> on behalf of gpfsug-discuss-request@spectrumscale.org
<gpfsug-discuss-request@spectrumscale.org><br>
<b>Sent:</b> Wednesday, June 3, 2020 4:56 PM<br>
<b>To:</b> gpfsug-discuss@spectrumscale.org <gpfsug-discuss@spectrumscale.org><br>
<b>Subject:</b> gpfsug-discuss Digest, Vol 101, Issue 6</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt">
<div class="PlainText">Send gpfsug-discuss mailing list submissions to<br>
gpfsug-discuss@spectrumscale.org<br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
or, via email, send a message with subject or body 'help' to<br>
gpfsug-discuss-request@spectrumscale.org<br>
<br>
You can reach the person managing the list at<br>
gpfsug-discuss-owner@spectrumscale.org<br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of gpfsug-discuss digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Introducing SSUG::Digital<br>
(Simon Thompson (Spectrum Scale User Group Chair))<br>
2. Client Latency and High NSD Server Load Average<br>
(Saula, Oluwasijibomi)<br>
3. Re: Client Latency and High NSD Server Load Average<br>
(Frederick Stock)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Wed, 03 Jun 2020 20:11:17 +0100<br>
From: "Simon Thompson (Spectrum Scale User Group Chair)"<br>
<chair@spectrumscale.org><br>
To: "gpfsug-discuss@spectrumscale.org"<br>
<gpfsug-discuss@spectrumscale.org><br>
Subject: [gpfsug-discuss] Introducing SSUG::Digital<br>
Message-ID: <AB923605-E4FE-45EC-A1EA-B61A4A147B06@spectrumscale.org><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Hi All.,<br>
<br>
<br>
<br>
I happy that we can finally announce SSUG:Digital, which will be a series of online session based on the types of topic we present at our in-person events.<br>
<br>
<br>
<br>
I know it?s taken use a while to get this up and running, but we?ve been working on trying to get the format right. So save the date for the first SSUG:Digital event which will take place on Thursday 18th June 2020 at 4pm BST. That?s:<br>
San Francisco, USA at 08:00 PDT<br>
New York, USA at 11:00 EDT<br>
London, United Kingdom at 16:00 BST<br>
Frankfurt, Germany at 17:00 CEST<br>
Pune, India at 20:30 IST<br>
We estimate about 90 minutes for the first session, and please forgive any teething troubles as we get this going!<br>
<br>
<br>
<br>
(I know the times don?t work for everyone in the global community!)<br>
<br>
<br>
<br>
Each of the sessions we run over the next few months will be a different Spectrum Scale Experts or Deep Dive session.<br>
<br>
More details at:<br>
<br>
<a href="https://www.spectrumscaleug.org/introducing-ssugdigital/">https://www.spectrumscaleug.org/introducing-ssugdigital/</a><br>
<br>
<br>
<br>
(We?ll announce the speakers and topic of the first session in the next few days ?)<br>
<br>
<br>
<br>
Thanks to Ulf, Kristy, Bill, Bob and Ted for their help and guidance in getting this going.<br>
<br>
<br>
<br>
We?re keen to include some user talks and site updates later in the series, so please let me know if you might be interested in presenting in this format.<br>
<br>
<br>
<br>
Simon Thompson<br>
<br>
SSUG Group Chair<br>
<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200603/e839fc73/attachment-0001.html">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200603/e839fc73/attachment-0001.html</a>><br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Wed, 3 Jun 2020 21:45:05 +0000<br>
From: "Saula, Oluwasijibomi" <oluwasijibomi.saula@ndsu.edu><br>
To: "gpfsug-discuss@spectrumscale.org"<br>
<gpfsug-discuss@spectrumscale.org><br>
Subject: [gpfsug-discuss] Client Latency and High NSD Server Load<br>
Average<br>
Message-ID:<br>
<DM6PR08MB5324B014BC4AA03CCF25557598880@DM6PR08MB5324.namprd08.prod.outlook.com><br>
<br>
Content-Type: text/plain; charset="iso-8859-1"<br>
<br>
<br>
Hello,<br>
<br>
Anyone faced a situation where a majority of NSDs have a high load average and a minority don't?<br>
<br>
Also, is 10x NSD server latency for write operations than for read operations expected in any circumstance?<br>
<br>
We are seeing client latency between 6 and 9 seconds and are wondering if some GPFS configuration or NSD server condition may be triggering this poor performance.<br>
<br>
<br>
<br>
Thanks,<br>
<br>
<br>
Oluwasijibomi (Siji) Saula<br>
<br>
HPC Systems Administrator / Information Technology<br>
<br>
<br>
<br>
Research 2 Building 220B / Fargo ND 58108-6050<br>
<br>
p: 701.231.7749 / www.ndsu.edu<<a href="http://www.ndsu.edu/">http://www.ndsu.edu/</a>><br>
<br>
<br>
<br>
[cid:image001.gif@01D57DE0.91C300C0]<br>
<br>
<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200603/2ac14173/attachment-0001.html">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200603/2ac14173/attachment-0001.html</a>><br>
<br>
------------------------------<br>
<br>
Message: 3<br>
Date: Wed, 3 Jun 2020 21:56:04 +0000<br>
From: "Frederick Stock" <stockf@us.ibm.com><br>
To: gpfsug-discuss@spectrumscale.org<br>
Cc: gpfsug-discuss@spectrumscale.org<br>
Subject: Re: [gpfsug-discuss] Client Latency and High NSD Server Load<br>
Average<br>
Message-ID:<br>
<OF4256061C.B3CA8966-ON0025857C.00786C34-0025857C.00787D7D@notes.na.collabserv.com><br>
<br>
Content-Type: text/plain; charset="us-ascii"<br>
<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200603/c252f3b9/attachment.html">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200603/c252f3b9/attachment.html</a>><br>
<br>
------------------------------<br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at spectrumscale.org<br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
<br>
<br>
End of gpfsug-discuss Digest, Vol 101, Issue 6<br>
**********************************************<br>
</div>
</span></font></div>
</body>
</html>