<font size=3 face="Arial">How many NSDs are served by the NSD servers
and what is your maximum file system block size? Have you confirmed
that you have sufficient NSD worker threads to handle the maximum number
of IOs you are configured to have active? That would be the number
of NSDs served times 12 (you have 12 threads per queue).</font><br><br><font size=3 face="sans-serif">Fred<br>__________________________________________________<br>Fred Stock | IBM Pittsburgh Lab | 720-430-8821<br>stockf@us.ibm.com</font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">"Buterbaugh, Kevin
L" <Kevin.Buterbaugh@Vanderbilt.Edu></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug main discussion
list <gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">07/03/2018 05:41 PM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
High I/O wait times</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><font size=3>Hi Fred, </font><br><br><font size=3>Thanks for the response. I have been looking at
the “mmfsadm dump nsd” data from the two NSD servers that serve up the
two NSDs that most commonly experience high wait times (although, again,
this varies from time to time). In addition, I have been reading:</font><br><br><a href="https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General Parallel File System (GPFS)/page/NSD Server Design and Tuning"><font size=3 color=blue><u>https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/NSD%20Server%20Design%20and%20Tuning</u></font></a><br><br><font size=3>And:</font><br><br><a href="https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/NSD%20Server%20Tuning"><font size=3 color=blue><u>https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/NSD%20Server%20Tuning</u></font></a><br><br><font size=3>Which seem to be the most relevant documents on the Wiki.</font><br><br><font size=3>I would like to do a more detailed analysis of the “mmfsadm
dump nsd” output, but my preliminary looks at it seems to indicate that
I see I/O’s queueing in the 50 - 100 range for the small queues and the
60 - 200 range on the large queues.</font><br><br><font size=3>In addition, I am regularly seeing all 12 threads on the
LARGE queues active, while it is much more rare that I see all - or even
close to all - the threads on the SMALL queues active.</font><br><br><font size=3>As far as the parameters Scott and Yuri mention, on our
cluster they are set thusly:</font><br><br><font size=3>[common]</font><br><font size=3>nsdMaxWorkerThreads 640</font><br><font size=3>[<all the GPFS servers listed here>]</font><br><font size=3>nsdMaxWorkerThreads 1024</font><br><font size=3>[common]</font><br><font size=3>nsdThreadsPerQueue 4</font><br><font size=3>[<all the GPFS servers listed here>]</font><br><font size=3>nsdThreadsPerQueue 12</font><br><font size=3>[common]</font><br><font size=3>nsdSmallThreadRatio 3</font><br><font size=3>[<all the GPFS servers listed here>]</font><br><font size=3>nsdSmallThreadRatio 1</font><br><br><font size=3>So to me it sounds like I need more resources on the LARGE
queue side of things … i.e. it sure doesn’t sound like I want to change
my small thread ratio. If I increase the amount of threads it sounds
like that might help, but that also takes more pagepool, and I’ve got
limited RAM in these (old) NSD servers. I do have nsdbufspace set
to 70, but I’ve only got 16-24 GB RAM each in these NSD servers. And
a while back I did try increase the page pool on them (very slightly) and
ended up causing problems because then they ran out of physical RAM.</font><br><br><font size=3>Thoughts? Followup questions? Thanks!</font><br><br><font size=3>Kevin</font><br><br><font size=3>On Jul 3, 2018, at 3:11 PM, Frederick Stock <stockf@us.ibm.com>
wrote:</font><br><br><font size=3 face="Arial">Are you seeing similar values for all the
nodes or just some of them? One possible issue is how the NSD queues
are configured on the NSD servers. You can see this with the output
of "mmfsadm dump nsd". There are queues for LARGE IOs (greater
than 64K) and queues for SMALL IOs (64K or less). Check the highest
pending values to see if many IOs are queueing. There are a couple
of options to fix this but rather than explain them I suggest you look
for information about NSD queueing on the developerWorks site. There
has been information posted there that should prove helpful.</font><font size=3><br></font><font size=3 face="sans-serif"><br>Fred<br>__________________________________________________<br>Fred Stock | IBM Pittsburgh Lab | 720-430-8821<br>stockf@us.ibm.com</font><font size=3><br><br><br></font><font size=1 color=#5f5f5f face="sans-serif"><br>From: </font><font size=1 face="sans-serif">"Buterbaugh,
Kevin L" <Kevin.Buterbaugh@Vanderbilt.Edu></font><font size=1 color=#5f5f5f face="sans-serif"><br>To: </font><font size=1 face="sans-serif">gpfsug
main discussion list <gpfsug-discuss@spectrumscale.org></font><font size=1 color=#5f5f5f face="sans-serif"><br>Date: </font><font size=1 face="sans-serif">07/03/2018
03:49 PM</font><font size=1 color=#5f5f5f face="sans-serif"><br>Subject: </font><font size=1 face="sans-serif">[gpfsug-discuss]
High I/O wait times</font><font size=1 color=#5f5f5f face="sans-serif"><br>Sent by: </font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><font size=3><br></font><hr noshade><font size=3><br><br><br>Hi all, <br><br>We are experiencing some high I/O wait times (5 - 20 seconds!) on some
of our NSDs as reported by “mmdiag —iohist" and are struggling to
understand why. One of the confusing things is that, while certain
NSDs tend to show the problem more than others, the problem is not consistent
… i.e. the problem tends to move around from NSD to NSD (and storage array
to storage array) whenever we check … which is sometimes just a few minutes
apart.<br><br>In the past when I have seen “mmdiag —iohist” report high wait times
like this it has *always* been hardware related. In our environment,
the most common cause has been a battery backup unit on a storage array
controller going bad and the storage array switching to write straight
to disk. But that’s *not* happening this time. <br><br>Is there anything within GPFS / outside of a hardware issue that I should
be looking for?? Thanks!<br><br>—<br>Kevin Buterbaugh - Senior System Administrator<br>Vanderbilt University - Advanced Computing Center for Research and Education</font><font size=3 color=blue><u><br></u></font><a href="mailto:Kevin.Buterbaugh@vanderbilt.edu"><font size=3 color=blue><u>Kevin.Buterbaugh@vanderbilt.edu</u></font></a><font size=3>-
(615)875-9633<br><br></font><tt><font size=2><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org</font></tt><font size=3 color=blue><u><br></u></font><a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cd3d7ff675bb440286cb908d5e1212b66%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636662454938066010&sdata=jL0pB5MEaWtJZjMbS8JzhsKGvwmYB6qV%2FVyosdUKcSU%3D&reserved=0"><tt><font size=2 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></tt></a><font size=3><br><br><br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font><a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cd3d7ff675bb440286cb908d5e1212b66%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636662454938076014&sdata=wIyB66HoqvL13I3LX0Ott%2Btr7HQQdInZ028QUp0QMhE%3D&reserved=0"><font size=3>https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Cd3d7ff675bb440286cb908d5e1212b66%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636662454938076014&sdata=wIyB66HoqvL13I3LX0Ott%2Btr7HQQdInZ028QUp0QMhE%3D&reserved=0</font></a><br><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br></font></tt><br><br><BR>