<html><body><p><font size="2" face="Arial">Hi,</font><br><br><font size="2" face="Arial">>> I do notice nsd03/nsd04 have long waiters, but nsd01 doesn't (nsd02-ib is offline for now): </font><br><br><font size="2" face="Arial">Please issue "mmlsdisk <fs> -m" in NSD client to ascertain the active NSD server serving a NSD. Since nsd02-ib is offlined, it is possible that some servers would be serving higher NSDs than the rest. </font><br><br><a href="https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.5/com.ibm.spectrum.scale.v5r05.doc/bl1pdg_PoorPerformanceDuetoDiskFailure.htm"><font size="2" face="Arial">https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.5/com.ibm.spectrum.scale.v5r05.doc/bl1pdg_PoorPerformanceDuetoDiskFailure.htm</font></a><br><a href="https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.5/com.ibm.spectrum.scale.v5r05.doc/bl1pdg_HealthStateOfNSDserver.htm"><font size="2" face="Arial">https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.5/com.ibm.spectrum.scale.v5r05.doc/bl1pdg_HealthStateOfNSDserver.htm</font></a><br><br><font size="2" face="Arial">>> From the waiters you provided I would guess there is something amiss with some of your storage systems.</font><br><br><font size="2" face="Arial">Please ensure there are no "disk rebuild" pertaining to certain NSDs/storage volumes in progress (in the storage subsystem) as this can sometimes impact block-level performance and thus impact latency, especially for write operations. Please ensure that the hardware components constituting the Spectrum Scale stack are healthy and performing optimally.</font><br><br><a href="https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.5/com.ibm.spectrum.scale.v5r05.doc/bl1pdg_pspduetosyslevelcompissue.htm"><font size="2" face="Arial">https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.5/com.ibm.spectrum.scale.v5r05.doc/bl1pdg_pspduetosyslevelcompissue.htm</font></a><br><br><font size="2" face="Arial">Please refer to the Spectrum Scale documentation (link below) for potential causes (e.g. Scale maintenance operation such as mmapplypolicy/mmestripefs in progress, slow disks) that can be contributing to this issue:</font><br><br><a href="https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.5/com.ibm.spectrum.scale.v5r05.doc/bl1pdg_performanceissues.htm"><font size="2" face="Arial">https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.5/com.ibm.spectrum.scale.v5r05.doc/bl1pdg_performanceissues.htm</font></a><br><br><font size="2" face="Arial">Thanks and Regards,</font><br><font size="2" face="Arial">-Kums</font><br><br><font color="#0000FF">Kumaran Rajaram</font><br><font size="2">Spectrum Scale Development, IBM Systems</font><br><u><font size="2" color="#0000FF">kums@us.ibm.com</font></u><br><br><br><img width="16" height="16" src="cid:1__=8FBB0FEEDFC13FA78f9e8a93df938690918c8FB@" border="0" alt="Inactive hide details for "Frederick Stock" ---06/04/2020 07:08:45 AM---From the waiters you provided I would guess there is so"><font size="2" color="#424282">"Frederick Stock" ---06/04/2020 07:08:45 AM---From the waiters you provided I would guess there is something amiss with some of your storage syste</font><br><br><font size="2" color="#5F5F5F">From: </font><font size="2">"Frederick Stock" <stockf@us.ibm.com></font><br><font size="2" color="#5F5F5F">To: </font><font size="2">gpfsug-discuss@spectrumscale.org</font><br><font size="2" color="#5F5F5F">Cc: </font><font size="2">gpfsug-discuss@spectrumscale.org</font><br><font size="2" color="#5F5F5F">Date: </font><font size="2">06/04/2020 07:08 AM</font><br><font size="2" color="#5F5F5F">Subject: </font><font size="2">[EXTERNAL] Re: [gpfsug-discuss] Client Latency and High NSD Server Load Average</font><br><font size="2" color="#5F5F5F">Sent by: </font><font size="2">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br><br><br><font face="Arial">From the waiters you provided I would guess there is something amiss with some of your storage systems. Since those waiters are on NSD servers they are waiting for IO requests to the kernel to complete. Generally IOs are expected to complete in milliseconds, not seconds. You could look at the output of "mmfsadm dump nsd" to see how the GPFS IO queues are working but that would be secondary to checking your storage systems.</font><br><br>Fred<br>__________________________________________________<br>Fred Stock | IBM Pittsburgh Lab | 720-430-8821<br>stockf@us.ibm.com<br><font face="Arial"> </font><br><font face="Arial"> </font><br><font face="Arial">----- Original message -----<br>From: "Saula, Oluwasijibomi" <oluwasijibomi.saula@ndsu.edu><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: "gpfsug-discuss@spectrumscale.org" <gpfsug-discuss@spectrumscale.org><br>Cc:<br>Subject: [EXTERNAL] Re: [gpfsug-discuss] Client Latency and High NSD Server Load Average<br>Date: Wed, Jun 3, 2020 6:24 PM<br> </font><br><font color="#323130" face="Helvetica Neue">Frederick,</font><br><font face="Calibri"> </font><br><font color="#323130" face="Helvetica Neue">Yes on both counts! - mmdf is showing pretty uniform (ie 5 NSDs out of 30 report 65% free; All others are uniform at 58% free)...</font><br><font face="Calibri"> </font><br><font color="#323130" face="Helvetica Neue">NSD servers per disks are called in round-robin fashion as well, for example:</font><br><font face="Arial"> </font><br><font face="Calibri"> gpfs1 tier2_001 nsd02-ib,nsd03-ib,nsd04-ib,tsm01-ib,nsd01-ib </font><br><font face="Calibri"> gpfs1 tier2_002 nsd03-ib,nsd04-ib,tsm01-ib,nsd01-ib,nsd02-ib </font><br><font face="Calibri"> gpfs1 tier2_003 nsd04-ib,tsm01-ib,nsd01-ib,nsd02-ib,nsd03-ib </font><br><font face="Calibri"> gpfs1 tier2_004 tsm01-ib,nsd01-ib,nsd02-ib,nsd03-ib,nsd04-ib </font><p><font face="Arial">Any other potential culprits to investigate?</font><br><font face="Arial"> </font><br><font face="Arial">I do notice nsd03/nsd04 have long waiters, but nsd01 doesn't (nsd02-ib is offline for now): </font><br><font face="Calibri">[nsd03-ib ~]# mmdiag --waiters</font><br><font face="Calibri">=== mmdiag: waiters ===</font><br><font face="Calibri">Waiting 6.5113 sec since 17:17:33, monitored, thread 4175 NSDThread: for I/O completion</font><br><font face="Calibri">Waiting 6.3810 sec since 17:17:33, monitored, thread 4127 NSDThread: for I/O completion</font><br><font face="Calibri">Waiting 6.1959 sec since 17:17:34, monitored, thread 4144 NSDThread: for I/O completion</font><br><font face="Arial"> </font><br><font face="Calibri">nsd04-ib:</font><br><font face="Arial"> </font><br><font face="Calibri">Waiting 13.1386 sec since 17:19:09, monitored, thread 9971 NSDThread: for I/O completion</font><br><font face="Calibri">Waiting 10.3562 sec since 17:19:12, monitored, thread 9958 NSDThread: for I/O completion</font><br><font face="Calibri">Waiting 10.0338 sec since 17:19:12, monitored, thread 9951 NSDThread: for I/O completion</font><p><font face="Arial"> </font><br><font face="Calibri">tsm01-ib:</font><br><font face="Arial"> </font><br><font face="Calibri">Waiting 8.1211 sec since 17:20:24, monitored, thread 3644 NSDThread: for I/O completion</font><br><font face="Calibri">Waiting 7.6690 sec since 17:20:24, monitored, thread 3641 NSDThread: for I/O completion</font><br><font face="Calibri">Waiting 7.4969 sec since 17:20:24, monitored, thread 3658 NSDThread: for I/O completion</font><br><font face="Calibri">Waiting 7.3573 sec since 17:20:24, monitored, thread 3642 NSDThread: for I/O completion</font><p><font face="Arial"> </font><br><font face="Calibri">nsd01-ib:</font><br><font face="Arial"> </font><br><font face="Calibri">Waiting 0.2548 sec since 17:21:47, monitored, thread 30513 NSDThread: for I/O completion</font><br><font face="Calibri">Waiting 0.1502 sec since 17:21:47, monitored, thread 30529 NSDThread: for I/O completion</font><p><font face="Arial"> </font><p><font face="Arial"> </font><br><font face="Arial"> </font><br><font face="Calibri"> </font><br><font face="Arial">Thanks,</font><br><font face="Calibri"> </font><br><b><font size="2" color="#201F1E" face="Arial">Oluwasijibomi (Siji) Saula</font></b><p><font size="2" color="#201F1E" face="Arial">HPC Systems Administrator / Information Technology</font><p><font size="2" color="#201F1E" face="Arial"> </font><p><font size="2" color="#201F1E" face="Arial">Research 2 Building 220B / Fargo ND 58108-6050</font><p><font size="2" color="#201F1E" face="Arial">p: 701.231.7749 / </font><a href="http://www.ndsu.edu/" target="_blank"><u><font size="2" color="#800080" face="Arial">www.ndsu.edu</font></u></a><p><font size="2" color="#201F1E" face="Arial"> </font><p><br><font face="Calibri"> </font><br><font face="Calibri"> </font><br><font face="Calibri"> </font><p><font face="Arial"> </font><br><font face="Calibri"> </font><br><hr width="100%" size="2" align="left"><br><b><font face="Calibri">From:</font></b><font face="Calibri"> gpfsug-discuss-bounces@spectrumscale.org <gpfsug-discuss-bounces@spectrumscale.org> on behalf of gpfsug-discuss-request@spectrumscale.org <gpfsug-discuss-request@spectrumscale.org></font><b><font face="Calibri"><br>Sent:</font></b><font face="Calibri"> Wednesday, June 3, 2020 4:56 PM</font><b><font face="Calibri"><br>To:</font></b><font face="Calibri"> gpfsug-discuss@spectrumscale.org <gpfsug-discuss@spectrumscale.org></font><b><font face="Calibri"><br>Subject:</font></b><font face="Calibri"> gpfsug-discuss Digest, Vol 101, Issue 6</font><font face="Arial"> </font><br><font face="Arial"> </font><br><font face="Arial">Send gpfsug-discuss mailing list submissions to<br> gpfsug-discuss@spectrumscale.org<br><br>To subscribe or unsubscribe via the World Wide Web, visit<br> </font><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><u><font color="#0000FF" face="Arial">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></u></a><font face="Arial"><br>or, via email, send a message with subject or body 'help' to<br> gpfsug-discuss-request@spectrumscale.org<br><br>You can reach the person managing the list at<br> gpfsug-discuss-owner@spectrumscale.org<br><br>When replying, please edit your Subject line so it is more specific<br>than "Re: Contents of gpfsug-discuss digest..."<br><br><br>Today's Topics:<br><br> 1. Introducing SSUG::Digital<br> (Simon Thompson (Spectrum Scale User Group Chair))<br> 2. Client Latency and High NSD Server Load Average<br> (Saula, Oluwasijibomi)<br> 3. Re: Client Latency and High NSD Server Load Average<br> (Frederick Stock)<br><br><br>----------------------------------------------------------------------<br><br>Message: 1<br>Date: Wed, 03 Jun 2020 20:11:17 +0100<br>From: "Simon Thompson (Spectrum Scale User Group Chair)"<br> <chair@spectrumscale.org><br>To: "gpfsug-discuss@spectrumscale.org"<br> <gpfsug-discuss@spectrumscale.org><br>Subject: [gpfsug-discuss] Introducing SSUG::Digital<br>Message-ID: <AB923605-E4FE-45EC-A1EA-B61A4A147B06@spectrumscale.org><br>Content-Type: text/plain; charset="utf-8"<br><br>Hi All.,<br><br> <br><br>I happy that we can finally announce SSUG:Digital, which will be a series of online session based on the types of topic we present at our in-person events.<br><br> <br><br>I know it?s taken use a while to get this up and running, but we?ve been working on trying to get the format right. So save the date for the first SSUG:Digital event which will take place on Thursday 18th June 2020 at 4pm BST. That?s:<br>San Francisco, USA at 08:00 PDT<br>New York, USA at 11:00 EDT<br>London, United Kingdom at 16:00 BST<br>Frankfurt, Germany at 17:00 CEST<br>Pune, India at 20:30 IST<br>We estimate about 90 minutes for the first session, and please forgive any teething troubles as we get this going!<br><br> <br><br>(I know the times don?t work for everyone in the global community!)<br><br> <br><br>Each of the sessions we run over the next few months will be a different Spectrum Scale Experts or Deep Dive session.<br><br>More details at:<br></font><u><font color="#0000FF" face="Arial"><br></font></u><a href="https://www.spectrumscaleug.org/introducing-ssugdigital/" target="_blank"><u><font color="#0000FF" face="Arial">https://www.spectrumscaleug.org/introducing-ssugdigital/</font></u></a><font face="Arial"><br><br> <br><br>(We?ll announce the speakers and topic of the first session in the next few days ?)<br><br> <br><br>Thanks to Ulf, Kristy, Bill, Bob and Ted for their help and guidance in getting this going.<br><br> <br><br>We?re keen to include some user talks and site updates later in the series, so please let me know if you might be interested in presenting in this format.<br><br> <br><br>Simon Thompson<br><br>SSUG Group Chair<br><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</font><a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200603/e839fc73/attachment-0001.html" target="_blank"><u><font color="#0000FF" face="Arial">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200603/e839fc73/attachment-0001.html</font></u></a><font face="Arial">><br><br>------------------------------<br><br>Message: 2<br>Date: Wed, 3 Jun 2020 21:45:05 +0000<br>From: "Saula, Oluwasijibomi" <oluwasijibomi.saula@ndsu.edu><br>To: "gpfsug-discuss@spectrumscale.org"<br> <gpfsug-discuss@spectrumscale.org><br>Subject: [gpfsug-discuss] Client Latency and High NSD Server Load<br> Average<br>Message-ID:<br> <DM6PR08MB5324B014BC4AA03CCF25557598880@DM6PR08MB5324.namprd08.prod.outlook.com><br> <br>Content-Type: text/plain; charset="iso-8859-1"<br><br><br>Hello,<br><br>Anyone faced a situation where a majority of NSDs have a high load average and a minority don't?<br><br>Also, is 10x NSD server latency for write operations than for read operations expected in any circumstance?<br><br>We are seeing client latency between 6 and 9 seconds and are wondering if some GPFS configuration or NSD server condition may be triggering this poor performance.<br><br><br><br>Thanks,<br><br><br>Oluwasijibomi (Siji) Saula<br><br>HPC Systems Administrator / Information Technology<br><br><br><br>Research 2 Building 220B / Fargo ND 58108-6050<br><br>p: 701.231.7749 / </font><font face="Arial">www.ndsu.edu</font><font face="Arial"><</font><a href="http://www.ndsu.edu/" target="_blank"><u><font color="#0000FF" face="Arial">http://www.ndsu.edu/</font></u></a><font face="Arial">><br><br><br><br>[</font><font face="Arial"><a href="cid:image001.gif@01D57DE0.91C300C0">cid:image001.gif@01D57DE0.91C300C0</a></font><font face="Arial">]<br><br><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</font><a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200603/2ac14173/attachment-0001.html" target="_blank"><u><font color="#0000FF" face="Arial">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200603/2ac14173/attachment-0001.html</font></u></a><font face="Arial">><br><br>------------------------------<br><br>Message: 3<br>Date: Wed, 3 Jun 2020 21:56:04 +0000<br>From: "Frederick Stock" <stockf@us.ibm.com><br>To: gpfsug-discuss@spectrumscale.org<br>Cc: gpfsug-discuss@spectrumscale.org<br>Subject: Re: [gpfsug-discuss] Client Latency and High NSD Server Load<br> Average<br>Message-ID:<br> <OF4256061C.B3CA8966-ON0025857C.00786C34-0025857C.00787D7D@notes.na.collabserv.com><br> <br>Content-Type: text/plain; charset="us-ascii"<br><br>An HTML attachment was scrubbed...<br>URL: <</font><a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200603/c252f3b9/attachment.html" target="_blank"><u><font color="#0000FF" face="Arial">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200603/c252f3b9/attachment.html</font></u></a><font face="Arial">><br><br>------------------------------<br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org</font><u><font color="#0000FF" face="Arial"><br></font></u><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><u><font color="#0000FF" face="Arial">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></u></a><font face="Arial"><br><br><br>End of gpfsug-discuss Digest, Vol 101, Issue 6<br>**********************************************</font><br><tt><font size="2">_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org</font></tt><tt><u><font size="2" color="#0000FF"><br></font></u></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><tt><u><font size="2" color="#0000FF">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></u></tt></a><tt><font size="2"> </font></tt><br><font face="Arial"> </font><br><tt><font size="2">_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><tt><font size="2"><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></tt><tt><font size="2"> <br></font></tt><br><br><BR>
</body></html>