<div dir="ltr">Hi,<div><br></div><div>using nsdSmallThreadRatio 1 is not necessarily correct, as it 'significant depends' (most used word combination of performance engineers) on your workload. to give some more background - on reads you need much more threads for small i/os than for large i/os to get maximum performance, the reason is a small i/o usually only reads one strip of data (sitting on one physical device) while a large i/o reads an entire stripe (which typically spans multiple devices). as a more concrete example, in a 8+2p raid setup a single full stripe read will trigger internal reads in parallel to 8 different targets at the same time, so for small i/os you would need 8 times as many small read requests (and therefore threads) to keep the drives busy at the same level. on writes its even more complex, a large full stripe write usually just writes to all target disks, while a tiny small write in the middle might force a read / modify / write which can have a huge write amplification and cause more work than a large full track i/o. raid controller caches also play a significant role here and make this especially hard to optimize as you need to know exactly what and where to measure when you tune to get improvements for real world workload and not just improve your synthetic test but actually hurt your real application performance. </div><div>i should write a book about this some day ;-)</div><div><br></div><div>hope that helps. Sven</div><div><br></div><div><br></div><div> <br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Feb 21, 2019 at 4:23 AM Frederick Stock <<a href="mailto:stockf@us.ibm.com">stockf@us.ibm.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="m_-8146459914603134490socmaildefaultfont" dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:12pt"><div dir="ltr">Kevin I'm assuming you have seen the article on IBM developerWorks about the GPFS NSD queues. It provides useful background for analyzing the dump nsd information. Here I'll list some thoughts for items that you can investigate/consider.</div>
<div dir="ltr"> </div>
<div dir="ltr">If your NSD servers are doing both large (greater than 64K) and small (64K or less) IOs then you want to have the nsdSmallThreadRatio set to 1 as it seems you do for the NSD servers. This provides an equal number of SMALL and LARGE NSD queues. You can also increase the total number of queues (currently 256) but I cannot determine if that is necessary from the data you provided. Only on rare occasions have I seen a need to increase the number of queues.</div>
<div dir="ltr"> </div>
<div dir="ltr">The fact that you have 71 highest pending on your LARGE queues and 73 highest pending on your SMALL queues would imply your IOs are queueing for a good while either waiting for resources in GPFS or waiting for IOs to complete. Your maximum buffer size is 16M which is defined to be the largest IO that can be requested by GPFS. This is the buffer size that GPFS will use for LARGE IOs. You indicated you had sufficient memory on the NSD servers but what is the value for the pagepool on those servers, and what is the value of the nsdBufSpace parameter? If the NSD server is just that then usually nsdBufSpace is set to 70. The IO buffers used by the NSD server come from the pagepool so you need sufficient space there for the maximum number of LARGE IO buffers that would be used concurrently by GPFS or threads will need to wait for those buffers to become available. Essentially you want to ensure you have sufficient memory for the maximum number of IOs all doing a large IO and that value being less than 70% of the pagepool size.</div>
<div dir="ltr"> </div>
<div dir="ltr">You could look at the settings for the FC cards to ensure they are configured to do the largest IOs possible. I forget the actual values (have not done this for awhile) but there are settings for the adapters that control the maximum IO size that will be sent. I think you want this to be as large as the adapter can handle to reduce the number of messages needed to complete the large IOs done by GPFS.</div>
<div dir="ltr"> </div>
<div dir="ltr"> </div>
<div dir="ltr"><div class="m_-8146459914603134490socmaildefaultfont" dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10.5pt"><div dir="ltr"><span style="font-size:1.143em">Fred<br>__________________________________________________<br>Fred Stock | IBM Pittsburgh Lab | <a href="tel:(720)%20430-8821" value="+17204308821" target="_blank">720-430-8821</a><br><a href="mailto:stockf@us.ibm.com" target="_blank">stockf@us.ibm.com</a></span></div></div></div>
<div dir="ltr"> </div>
<div dir="ltr"> </div>
<blockquote dir="ltr" style="border-left:solid #aaaaaa 2px;margin-left:5px;padding-left:5px;direction:ltr;margin-right:0px"></blockquote></div><div class="m_-8146459914603134490socmaildefaultfont" dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:12pt"><blockquote dir="ltr" style="border-left:solid #aaaaaa 2px;margin-left:5px;padding-left:5px;direction:ltr;margin-right:0px">----- Original message -----<br>From: "Buterbaugh, Kevin L" <Kevin.Buterbaugh@Vanderbilt.Edu><br>Sent by: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a><br>To: gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a>><br></blockquote></div><div class="m_-8146459914603134490socmaildefaultfont" dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:12pt"><blockquote dir="ltr" style="border-left:solid #aaaaaa 2px;margin-left:5px;padding-left:5px;direction:ltr;margin-right:0px">Cc:<br>Subject: Re: [gpfsug-discuss] Clarification of mmdiag --iohist output<br>Date: Thu, Feb 21, 2019 6:39 AM<br> <br> Hi All,
<div> </div>
<div>My thanks to Aaron, Sven, Steve, and whoever responded for the GPFS team. You confirmed what I suspected … my example 10 second I/O was _from an NSD server_ … and since we’re in a 8 Gb FC SAN environment, it therefore means - correct me if I’m wrong about this someone - that I’ve got a problem somewhere in one (or more) of the following 3 components:</div>
<div> </div>
<div>1) the NSD servers</div>
<div>2) the SAN fabric</div>
<div>3) the storage arrays</div>
<div> </div>
<div>I’ve been looking at all of the above and none of them are showing any obvious problems. I’ve actually got a techie from the storage array vendor stopping by on Thursday, so I’ll see if he can spot anything there. Our FC switches are QLogic’s, so I’m kinda screwed there in terms of getting any help. But I don’t see any errors in the switch logs and “show perf” on the switches is showing I/O rates of 50-100 MB/sec on the in use ports, so I don’t _think_ that’s the issue.</div>
<div> </div>
<div>And this is the GPFS mailing list, after all … so let’s talk about the NSD servers. Neither memory (64 GB) nor CPU (2 x quad-core Intel Xeon E5620’s) appear to be an issue. But I have been looking at the output of “mmfsadm saferdump nsd” based on what Aaron and then Steve said. Here’s some fairly typical output from one of the SMALL queues (I’ve checked several of my 8 NSD servers and they’re all showing similar output):</div>
<div> </div>
<div><div> Queue NSD type NsdQueueTraditional [244]: SMALL, threads started 12, active 3, highest 12, deferred 0, chgSize 0, draining 0, is_chg 0</div>
<div> requests pending 0, highest pending 73, total processed 4859732</div>
<div> mutex 0x7F3E449B8F10, reqCond 0x7F3E449B8F58, thCond 0x7F3E449B8F98, queue 0x7F3E449B8EF0, nFreeNsdRequests 29</div></div>
<div> </div>
<div>And for a LARGE queue:</div>
<div> </div>
<div> Queue NSD type NsdQueueTraditional [8]: LARGE, threads started 12, active 1, highest 12, deferred 0, chgSize 0, draining 0, is_chg 0</div>
<div> requests pending 0, highest pending 71, total processed 2332966</div>
<div> mutex 0x7F3E441F3890, reqCond 0x7F3E441F38D8, thCond 0x7F3E441F3918, queue 0x7F3E441F3870, nFreeNsdRequests 31</div>
<div> </div>
<div>So my large queues seem to be slightly less utilized than my small queues overall … i.e. I see more inactive large queues and they generally have a smaller “highest pending” value.</div>
<div> </div>
<div>Question: are those non-zero “highest pending” values something to be concerned about?</div>
<div> </div>
<div>I have the following thread-related parameters set:</div>
<div> </div>
<div>[common]</div>
<div><div>maxReceiverThreads 12</div>
<div>nsdMaxWorkerThreads 640</div>
<div>nsdThreadsPerQueue 4</div>
<div>nsdSmallThreadRatio 3</div>
<div>workerThreads 128</div>
<div> </div>
<div>[serverLicense]</div>
<div>nsdMaxWorkerThreads 1024</div>
<div>nsdThreadsPerQueue 12</div>
<div>nsdSmallThreadRatio 1</div>
<div>pitWorkerThreadsPerNode 3</div>
<div>workerThreads 1024</div></div>
<div> </div>
<div>Also, at the top of the “mmfsadm saferdump nsd” output I see:</div>
<div> </div>
<div><div>Total server worker threads: running 1008, desired 147, forNSD 147, forGNR 0, nsdBigBufferSize 16777216</div>
<div>nsdMultiQueue: 256, nsdMultiQueueType: 1, nsdMinWorkerThreads: 16, nsdMaxWorkerThreads: 1024</div></div>
<div> </div>
<div>Question: is the fact that 1008 is pretty close to 1024 a concern?</div>
<div> </div>
<div>Anything jump out at anybody? I don’t mind sharing full output, but it is rather lengthy. Is this worthy of a PMR?</div>
<div> </div>
<div>Thanks!</div>
<div> </div>
<div>--</div>
<div><div><div><div>Kevin Buterbaugh - Senior System Administrator</div>
<div>Vanderbilt University - Advanced Computing Center for Research and Education</div>
<div><a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" target="_blank">Kevin.Buterbaugh@vanderbilt.edu</a> - <a href="tel:(615)%20875-9633" value="+16158759633" target="_blank">(615)875-9633</a></div></div></div>
<div>
<blockquote type="cite"><div>On Feb 17, 2019, at 1:01 PM, IBM Spectrum Scale <<a href="mailto:scale@us.ibm.com" target="_blank">scale@us.ibm.com</a>> wrote:</div>
<div><span style="font-size:10pt;font-family:sans-serif">Hi Kevin,</span><br><br><span style="font-size:10pt;font-family:sans-serif">The I/O hist shown by the command mmdiag --iohist actually depends on the node on which you are running this command from.</span><br><span style="font-size:10pt;font-family:sans-serif">If you are running this on a NSD server node then it will show the time taken to complete/serve the read or write I/O operation sent from the client node. </span><br><span style="font-size:10pt;font-family:sans-serif">And if you are running this on a client (or non NSD server) node then it will show the complete time taken by the read or write I/O operation requested by the client node to complete.</span><br><span style="font-size:10pt;font-family:sans-serif">So in a nut shell for the NSD server case it is just the latency of the I/O done on disk by the server whereas for the NSD client case it also the latency of send and receive of I/O request to the NSD server along with the latency of I/O done on disk by the NSD server.</span><br><span style="font-size:10pt;font-family:sans-serif">I hope this answers your query.</span><br><br><br><span style="font-size:10pt;font-family:sans-serif">Regards, The Spectrum Scale (GPFS) team<br><br>------------------------------------------------------------------------------------------------------------------<br>If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at </span><a href="https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ibm.com%2Fdeveloperworks%2Fcommunity%2Fforums%2Fhtml%2Fforum%3Fid%3D11111111-0000-0000-0000-000000000479&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2bfb2e8e30e64fa06c0f08d6959b2d38%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636860891056267091&sdata=%2FWFsVfr73xZcfH25vIFYC4ts7LlWDFUIoh9fLheAEwE%3D&reserved=0" target="_blank"><span style="font-size:10pt;font-family:sans-serif">https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479</span></a><span style="font-size:10pt;font-family:sans-serif">.<br><br>If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact <a href="tel:(800)%20237-5511" value="+18002375511" target="_blank">1-800-237-5511</a> in the United States or your local IBM Service Center in other countries.<br><br>The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team.</span><br><br><br><br><span style="font-size:9pt;color:#5f5f5f;font-family:sans-serif">From: </span><span style="font-size:9pt;font-family:sans-serif">"Buterbaugh, Kevin L" <<a href="mailto:Kevin.Buterbaugh@Vanderbilt.Edu" target="_blank">Kevin.Buterbaugh@Vanderbilt.Edu</a>></span><br><span style="font-size:9pt;color:#5f5f5f;font-family:sans-serif">To: </span><span style="font-size:9pt;font-family:sans-serif">gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a>></span><br><span style="font-size:9pt;color:#5f5f5f;font-family:sans-serif">Date: </span><span style="font-size:9pt;font-family:sans-serif">02/16/2019 08:18 PM</span><br><span style="font-size:9pt;color:#5f5f5f;font-family:sans-serif">Subject: </span><span style="font-size:9pt;font-family:sans-serif">[gpfsug-discuss] Clarification of mmdiag --iohist output</span><br><span style="font-size:9pt;color:#5f5f5f;font-family:sans-serif">Sent by: </span><span style="font-size:9pt;font-family:sans-serif"><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a></span>
<hr noshade><br><br><br><span style="font-size:12pt">Hi All, </span><br><br><span style="font-size:12pt">Been reading man pages, docs, and Googling, and haven’t found a definitive answer to this question, so I knew exactly where to turn… ;-)</span><br><br><span style="font-size:12pt">I’m dealing with some slow I/O’s to certain storage arrays in our environments … like really, really slow I/O’s … here’s just one example from one of my NSD servers of a 10 second I/O:</span><br><br><span style="font-size:12pt">08:49:34.943186 W data 30:41615622144 2048 10115.192 srv dm-92 <client IP redacted></span><br><br><span style="font-size:12pt">So here’s my question … when mmdiag —iohist tells me that that I/O took slightly over 10 seconds, is that:</span><br><br><span style="font-size:12pt">1. The time from when the NSD server received the I/O request from the client until it shipped the data back onto the wire towards the client?</span><br><span style="font-size:12pt">2. The time from when the client issued the I/O request until it received the data back from the NSD server?</span><br><span style="font-size:12pt">3. Something else?</span><br><br><span style="font-size:12pt">I’m thinking it’s #1, but want to confirm. Which one it is has very obvious implications for our troubleshooting steps. Thanks in advance…</span><br><br><span style="font-size:12pt">Kevin</span><br><span style="font-size:12pt">—</span><br><span style="font-size:12pt">Kevin Buterbaugh - Senior System Administrator</span><br><span style="font-size:12pt">Vanderbilt University - Advanced Computing Center for Research and Education</span><br><a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" target="_blank"><span style="font-size:12pt;color:blue"><u>Kevin.Buterbaugh@vanderbilt.edu</u></span></a><span style="font-size:12pt">- <a href="tel:(615)%20875-9633" value="+16158759633" target="_blank">(615)875-9633</a></span><br><tt><span style="font-size:10pt">_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a></span></tt><br><tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></tt><br><br><br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br><a href="https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2bfb2e8e30e64fa06c0f08d6959b2d38%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636860891056297114&sdata=5pL67mhVyScJovkRHRqZog9bM5BZG8F2q972czIYAbA%3D&reserved=0" target="_blank">https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C2bfb2e8e30e64fa06c0f08d6959b2d38%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636860891056297114&sdata=5pL67mhVyScJovkRHRqZog9bM5BZG8F2q972czIYAbA%3D&reserved=0</a></div></blockquote></div></div>
<div><font size="2" face="Default Monospace,Courier New,Courier,monospace">_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></div></blockquote></div><br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
</blockquote></div>