<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body>
<div id="compose" contenteditable="true" style="padding-left: 20px; padding-right: 20px; padding-bottom: 8px;">
<div>Thanks Yuri. As you can see I was already bumping the Min so I was effectively increasing the threads per disk as you suggest. I don't see nsdThreadsPerDisk in the config output, nor is it documented. I know it was before. </div>
<div><br>
</div>
<div>Interestingly, increasing the threads didn't really help, and in some cases made things worse. I settled on 256 threads and 32 threads per queue, with a ratio of 1. This gives me 4 large and 4 shall queues. </div>
<div><br>
</div>
<div>Regardless of the number of queues and threads I toss at it, I still see a number of queues with pending request in the 100's. </div>
<div><br>
</div>
<div>Waiters shows I/O waiters in the 200-300ms range, but the flash system (not IBM btw) still reports latency below 1ms. </div>
<div><br>
</div>
<div>Not sure where the bottleneck is. Perhaps it's the disk subsystem in Linux. Not sure how much can tell. </div>
<div><br>
</div>
<div>Back to playing whack-a-mole. <br>
<br>
<div class="acompli_signature">Bob Oesterlin<br>
Sr Storage Engineer, Nuance HPC Grid<br>
</div>
<br>
</div>
</div>
<div class="gmail_quote">_____________________________<br>
From: Yuri L Volobuev <<a dir="ltr" href="mailto:volobuev@us.ibm.com" x-apple-data-detectors="true" x-apple-data-detectors-type="link" x-apple-data-detectors-result="2">volobuev@us.ibm.com</a>><br>
Sent: Tuesday, January 12, 2016 3:58 PM<br>
Subject: Re: [gpfsug-discuss] Advice on NSD Thread tuning, moving from Disk to Flash<br>
To: gpfsug main discussion list <<a dir="ltr" href="mailto:gpfsug-discuss@spectrumscale.org" x-apple-data-detectors="true" x-apple-data-detectors-type="link" x-apple-data-detectors-result="4">gpfsug-discuss@spectrumscale.org</a>><br>
<br>
<br>
<meta content="text/html; charset=utf-8">
<p>If you want to have more NSD worker threads running, the recommended way to do this is increase nsdThreadsPerDisk, and also increase nsdMaxWorkerThreads. Make sure you have enough pagepool for all of the new threads though. You can also achieve a similar
result by increasing nsdMinWorkerThreads, which would render threads-per-disk logic irrelevant, although that's not how this parameter is normally used. </p>
<br>
<br>
</div>
</body>
</html>