<html><body><p>If you want to have more NSD worker threads running, the recommended way to do this is increase nsdThreadsPerDisk, and also increase nsdMaxWorkerThreads. Make sure you have enough pagepool for all of the new threads though. You can also achieve a similar result by increasing nsdMinWorkerThreads, which would render threads-per-disk logic irrelevant, although that's not how this parameter is normally used. The basic idea is that you want to achieve a certain degree of IO parallelism on the block device level, and that calls for some number of worker threads <i>per disk</i>. The number of NSD queues is not really relevant here, multiple queues is just a technique to achieve better SMP scalability, and with the default 256 queues there should be no contention for any individual queue on typical hardware (tens of CPU cores). Controlling the number of worker threads through nsdThreadsPerDisk gives you a way to adjust the thread count dynamically as disks are added or removed, without you having to micro-manage this by adjusting nsdMinWorkerThreads every time. <br><br>It's not clear whether having more IO parallelism necessarily improves the overall throughput, vs shifting bottlenecks elsewhere. Given the basic nature of flash storage, the former intuitively seems plausible, but some hard numbers from a real performance experiment would be helpful. <br><br>yuri<br><br><img width="16" height="16" src="cid:1__=07BBF5ABDFE42C218f9e8a93df938690918c07B@" border="0" alt="Inactive hide details for "Oesterlin, Robert" ---01/12/2016 11:31:44 AM---I'm experimenting with moving some of our more heavil"><font color="#424282">"Oesterlin, Robert" ---01/12/2016 11:31:44 AM---I'm experimenting with moving some of our more heavily used files (about 5% of the file system) from</font><br><br><font size="2" color="#5F5F5F">From: </font><font size="2">"Oesterlin, Robert" <Robert.Oesterlin@nuance.com></font><br><font size="2" color="#5F5F5F">To: </font><font size="2">gpfsug main discussion list <gpfsug-discuss@spectrumscale.org>, </font><br><font size="2" color="#5F5F5F">Date: </font><font size="2">01/12/2016 11:31 AM</font><br><font size="2" color="#5F5F5F">Subject: </font><font size="2">[gpfsug-discuss] Advice on NSD Thread tuning, moving from Disk to Flash</font><br><font size="2" color="#5F5F5F">Sent by: </font><font size="2">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br><br><br><font size="4" color="#222222" face="Arial">I'm experimenting with moving some of our more heavily used files (about 5% of the file system) from disk to a flash tier. The disk latency is around 10ms, while the flash is less than 1ms. What I'm seeing that when I move the files to flash, my NSD server queues (thread queues) have a much larger number of pending RPCs. (on the Small queues) Not surprising, since the storage subsystem is faster, the clients push more requests and the NSD server backs up. I don't see an evidence of the NSD server itself running out of steam (CPU below 20%) so I'm thinking I just need to adjust the number of threads and the threads per queue. Right now I have this:</font><br><br><font size="4" color="#222222" face="Courier New">nsdMaxWorkerThreads=256, nsdMinWorkerThreads=256, nsdThreadsPerQueue=8, nsdSmallThreadRatio=1</font><br><br><font size="4" color="#222222" face="Arial">What I'm trying to decide is whether I should increase just the MaxThreads or both the MaxThreads and the ThredsPerQueue.</font><br><font size="4" color="#222222" face="Arial">Can anyone offer advice? I did read through Yuri’s document.</font><p><br><br><font face="Arial">Bob Oesterlin<br>Sr Storage Engineer, Nuance HPC Grid</font><tt>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></tt><tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></tt><tt><br></tt><br><BR>
</body></html>