[gpfsug-discuss] Advice on NSD Thread tuning, moving from Disk to Flash

[Oesterlin, Robert]

Thanks Yuri. As you can see I was already bumping the Min so I  was effectively increasing the threads per disk as you suggest. I don't see nsdThreadsPerDisk in the config output, nor is it documented. I know it was before.

Interestingly, increasing the threads didn't really help, and in some cases made things worse. I settled on 256 threads and 32 threads per queue, with a ratio of 1. This gives me 4 large and 4 shall queues.

Regardless of the number of queues and threads I toss at it, I still see a number of queues with pending request in the 100's.

Waiters shows I/O waiters in the 200-300ms range, but the flash system (not IBM btw) still reports latency below 1ms.

Not sure where the bottleneck is. Perhaps it's the disk subsystem in Linux. Not sure how much can tell.

Back to playing whack-a-mole.

Bob Oesterlin
Sr Storage Engineer, Nuance HPC Grid

_____________________________
From: Yuri L Volobuev <volobuev at us.ibm.com<mailto:volobuev at us.ibm.com>>
Sent: Tuesday, January 12, 2016 3:58 PM
Subject: Re: [gpfsug-discuss] Advice on NSD Thread tuning, moving from Disk to Flash
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>



If you want to have more NSD worker threads running, the recommended way to do this is increase nsdThreadsPerDisk, and also increase nsdMaxWorkerThreads. Make sure you have enough pagepool for all of the new threads though. You can also achieve a similar result by increasing nsdMinWorkerThreads, which would render threads-per-disk logic irrelevant, although that's not how this parameter is normally used.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160112/30ce69d0/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: graycol.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160112/30ce69d0/attachment-0002.gif>