[gpfsug-discuss] Advice on NSD Thread tuning, moving from Disk to Flash

Yuri L Volobuev volobuev at us.ibm.com
Tue Jan 12 21:58:25 GMT 2016


If you want to have more NSD worker threads running, the recommended way to
do this is increase nsdThreadsPerDisk, and also increase
nsdMaxWorkerThreads.  Make sure you have enough pagepool for all of the new
threads though.  You can also achieve a similar result by increasing
nsdMinWorkerThreads, which would render threads-per-disk logic irrelevant,
although that's not how this parameter is normally used.  The basic idea is
that you want to achieve a certain degree of IO parallelism on the block
device level, and that calls for some number of worker threads per disk.
The number of NSD queues is not really relevant here, multiple queues is
just a technique to achieve better SMP scalability, and with the default
256 queues there should be no contention for any individual queue on
typical hardware (tens of CPU cores).  Controlling the number of worker
threads through nsdThreadsPerDisk gives you a way to adjust the thread
count dynamically as disks are added or removed, without you having to
micro-manage this by adjusting nsdMinWorkerThreads every time.

It's not clear whether having more IO parallelism necessarily improves the
overall throughput, vs shifting bottlenecks elsewhere.  Given the basic
nature of flash storage, the former intuitively seems plausible, but some
hard numbers from a real performance experiment would be helpful.

yuri



From:	"Oesterlin, Robert" <Robert.Oesterlin at nuance.com>
To:	gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>,
Date:	01/12/2016 11:31 AM
Subject:	[gpfsug-discuss] Advice on NSD Thread tuning,	moving from
            Disk to Flash
Sent by:	gpfsug-discuss-bounces at spectrumscale.org



I'm experimenting with moving some of our more heavily used files (about 5%
of the file system) from disk to a flash tier. The disk latency is around
10ms, while the flash is less than 1ms. What I'm seeing that when I move
the files to flash, my NSD server queues (thread queues) have a much larger
number of pending RPCs. (on the Small queues) Not surprising, since the
storage subsystem is faster, the clients push more requests and the NSD
server backs up. I don't see an evidence of the NSD server itself running
out of steam (CPU below 20%) so I'm thinking I just need to adjust the
number of threads and the threads per queue. Right now I have this:

nsdMaxWorkerThreads=256, nsdMinWorkerThreads=256, nsdThreadsPerQueue=8,
nsdSmallThreadRatio=1

What I'm trying to decide is whether I should increase just the MaxThreads
or both the MaxThreads and the ThredsPerQueue.
Can anyone offer advice? I did read through Yuri’s document.




Bob Oesterlin
Sr Storage Engineer, Nuance HPC Grid
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160112/7869833d/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160112/7869833d/attachment-0002.gif>


More information about the gpfsug-discuss mailing list