<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi Fred,
<div class=""><br class="">
</div>
<div class="">Thanks for the response. I have been looking at the “mmfsadm dump nsd” data from the two NSD servers that serve up the two NSDs that most commonly experience high wait times (although, again, this varies from time to time). In addition, I have
been reading:</div>
<div class=""><br class="">
</div>
<div class=""><a href="https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General Parallel File System (GPFS)/page/NSD Server Design and Tuning" class="">https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/NSD%20Server%20Design%20and%20Tuning</a></div>
<div class=""><br class="">
</div>
<div class="">And:</div>
<div class=""><br class="">
</div>
<div class=""><a href="https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/NSD%20Server%20Tuning" class="">https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/NSD%20Server%20Tuning</a><br class="">
<div><br class="">
</div>
<div>Which seem to be the most relevant documents on the Wiki.</div>
<div><br class="">
</div>
<div>I would like to do a more detailed analysis of the “mmfsadm dump nsd” output, but my preliminary looks at it seems to indicate that I see I/O’s queueing in the 50 - 100 range for the small queues and the 60 - 200 range on the large queues.</div>
<div><br class="">
</div>
<div>In addition, I am regularly seeing all 12 threads on the LARGE queues active, while it is much more rare that I see all - or even close to all - the threads on the SMALL queues active.</div>
<div><br class="">
</div>
<div>As far as the parameters Scott and Yuri mention, on our cluster they are set thusly:</div>
<div><br class="">
</div>
<div>
<div>[common]</div>
<div>nsdMaxWorkerThreads 640</div>
<div>[<all the GPFS servers listed here>]</div>
<div>nsdMaxWorkerThreads 1024</div>
<div>[common]</div>
<div>nsdThreadsPerQueue 4</div>
<div>[<all the GPFS servers listed here>]</div>
<div>nsdThreadsPerQueue 12</div>
<div>[common]</div>
<div>nsdSmallThreadRatio 3</div>
<div>[<all the GPFS servers listed here>]</div>
<div>nsdSmallThreadRatio 1</div>
<div><br class="">
</div>
<div>So to me it sounds like I need more resources on the LARGE queue side of things … i.e. it sure doesn’t sound like I want to change my small thread ratio. If I increase the amount of threads it sounds like that might help, but that also takes more pagepool,
and I’ve got limited RAM in these (old) NSD servers. I do have nsdbufspace set to 70, but I’ve only got 16-24 GB RAM each in these NSD servers. And a while back I did try increase the page pool on them (very slightly) and ended up causing problems because
then they ran out of physical RAM.</div>
<div><br class="">
</div>
<div>Thoughts? Followup questions? Thanks!</div>
<div><br class="">
</div>
<div>Kevin</div>
</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Jul 3, 2018, at 3:11 PM, Frederick Stock <stockf@us.ibm.com> wrote:</div>
<br class="Apple-interchange-newline">
<div class=""><font size="3" face="Arial" class="">Are you seeing similar values for all the nodes or just some of them? One possible issue is how the NSD queues are configured on the NSD servers. You can see this with the output of "mmfsadm dump nsd". There
are queues for LARGE IOs (greater than 64K) and queues for SMALL IOs (64K or less). Check the highest pending values to see if many IOs are queueing. There are a couple of options to fix this but rather than explain them I suggest you look for information
about NSD queueing on the developerWorks site. There has been information posted there that should prove helpful.</font><br class="">
<br class="">
<font size="3" face="sans-serif" class="">Fred<br class="">
__________________________________________________<br class="">
Fred Stock | IBM Pittsburgh Lab | 720-430-8821<br class="">
stockf@us.ibm.com</font><br class="">
<br class="">
<br class="">
<br class="">
<font size="1" color="#5f5f5f" face="sans-serif" class="">From: </font><font size="1" face="sans-serif" class="">"Buterbaugh, Kevin L" <Kevin.Buterbaugh@Vanderbilt.Edu></font><br class="">
<font size="1" color="#5f5f5f" face="sans-serif" class="">To: </font><font size="1" face="sans-serif" class="">gpfsug main discussion list <gpfsug-discuss@spectrumscale.org></font><br class="">
<font size="1" color="#5f5f5f" face="sans-serif" class="">Date: </font><font size="1" face="sans-serif" class="">07/03/2018 03:49 PM</font><br class="">
<font size="1" color="#5f5f5f" face="sans-serif" class="">Subject: </font><font size="1" face="sans-serif" class="">[gpfsug-discuss] High I/O wait times</font><br class="">
<font size="1" color="#5f5f5f" face="sans-serif" class="">Sent by: </font><font size="1" face="sans-serif" class="">gpfsug-discuss-bounces@spectrumscale.org</font><br class="">
<hr noshade="" class="">
<br class="">
<br class="">
<br class="">
<font size="3" class="">Hi all, </font><br class="">
<br class="">
<font size="3" class="">We are experiencing some high I/O wait times (5 - 20 seconds!) on some of our NSDs as reported by “mmdiag —iohist" and are struggling to understand why. One of the confusing things is that, while certain NSDs tend to show the problem
more than others, the problem is not consistent … i.e. the problem tends to move around from NSD to NSD (and storage array to storage array) whenever we check … which is sometimes just a few minutes apart.</font><br class="">
<br class="">
<font size="3" class="">In the past when I have seen “mmdiag —iohist” report high wait times like this it has *always* been hardware related. In our environment, the most common cause has been a battery backup unit on a storage array controller going bad and
the storage array switching to write straight to disk. But that’s *not* happening this time.
</font><br class="">
<br class="">
<font size="3" class="">Is there anything within GPFS / outside of a hardware issue that I should be looking for?? Thanks!</font><br class="">
<br class="">
<font size="3" class="">—</font><br class="">
<font size="3" class="">Kevin Buterbaugh - Senior System Administrator</font><br class="">
<font size="3" class="">Vanderbilt University - Advanced Computing Center for Research and Education</font><br class="">
<a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" class=""><font size="3" color="blue" class=""><u class="">Kevin.Buterbaugh@vanderbilt.edu</u></font></a><font size="3" class="">- (615)875-9633</font><br class="">
<br class="">
<br class="">
<tt class=""><font size="2" class="">_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at spectrumscale.org<br class="">
</font></tt><a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cd3d7ff675bb440286cb908d5e1212b66%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636662454938066010&sdata=jL0pB5MEaWtJZjMbS8JzhsKGvwmYB6qV%2FVyosdUKcSU%3D&reserved=0" originalsrc="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" shash="DKMQ3PTjOsYxK6js2bHIxgNIKXxKOBBFm2D9HrzCSzbABpWwZ74IGWffRVKiTjZR7xQCVD+wS7UfBQLb+XXnV3S2ztXr3pKVaKL1K53aVlgWGOyyXCF77E7RGZaln/3CSwXN0m9fupUSH6xgib+8OOO10S2L+dQ7mGkQ3HeZkQ0=" class=""><tt class=""><font size="2" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt class=""><font size="2" class=""><br class="">
</font></tt><br class="">
<br class="">
<br class="">
_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at spectrumscale.org<br class="">
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cd3d7ff675bb440286cb908d5e1212b66%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636662454938076014&sdata=wIyB66HoqvL13I3LX0Ott%2Btr7HQQdInZ028QUp0QMhE%3D&reserved=0<br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>