[gpfsug-discuss] tuning parameters question

Aaron Knister aaron.s.knister at nasa.gov
Mon Sep 11 23:15:10 BST 2017


Thanks, Olaf. I ended up un-setting a bunch of settings that are now 
auto-tuned (worker1threads, worker3threads, etc.) and just set 
workerthreads as you suggest. That combined with increasing 
maxfilestocache to above the max concurrent open file threshold of the 
workload got me consistently with in 1%-3% of the performance of the 
same storage hardware running btrfs instead of GPFS. I think that's 
pretty darned good considering the additional complexity GPFS has over 
btrfs of being a clustered filesystem. Plus I now get NFS server 
failover for very little effort and without having to deal with corosync 
or pacemaker.

-Aaron

On 9/11/17 4:11 AM, Olaf Weiser wrote:
> Hi Aaron ,
> 
> 0,0009 s response time for your meta data IO ... seems to be a very 
> good/fast storage BE.. which is hard to improve..
> you can raise the parallelism a bit for accessing metadata , but if this 
> will help to improve your "workload" is not assured
> 
> The worker3threads parameter specifies the number of threads to use for 
> inode prefetch. Usually , I would suggest, that you should not touch 
> single parameters any longer. By the great improvements of the last few 
> releases.. GPFS can calculate / retrieve the right settings 
> semi-automatically...
> You only need to set simpler "workerThreads" ..
> 
> But in your case , you can see, if this more specific value will help 
> you out .
> 
> depending on your blocksize and average filesize .. you may see 
> additional improvements when tuning nfsPrefetchStrategy , which tells 
> GPFS to consider all IOs wihtin */N/* blockboundaries as sequential  and 
> starts prefetch
> 
> l.b.n.t. set ignoreprefetchLunCount to yes .. (if not already done) . 
> this helps GPFS to use all available workerThreads
> 
> cheers
> olaf
> 
> 
> 
> From: Aaron Knister <aaron.s.knister at nasa.gov>
> To: <gpfsug-discuss at spectrumscale.org>
> Date: 09/11/2017 02:50 AM
> Subject: Re: [gpfsug-discuss] tuning parameters question
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> ------------------------------------------------------------------------
> 
> 
> 
> As an aside, my initial attempt was to use Ganesha via CES but the
> performance was significantly worse than CNFS for this workload. The
> docs seem to suggest that CNFS performs better for metadata intensive
> workloads which certainly seems to fit the bill here.
> 
> -Aaron
> 
> On 9/10/17 8:43 PM, Aaron Knister wrote:
>  > Hi All (but mostly Sven),
>  >
>  > I stumbled across this great gem:
>  >
>  > files.gpfsug.org/presentations/2014/UG10_GPFS_Performance_Session_v10.pdf
>  >
>  > and I'm wondering which, if any, of those tuning parameters are still
>  > relevant with the 4.2.3 code. Specifically for a CNFS cluster. I'm
>  > exporting a gpfs fs as an NFS root to 1k nodes. The boot storm is
>  > particularly ugly and the storage doesn't appear to be bottlenecked.
>  >
>  > I see a lot of waiters like these:
>  >
>  > Waiting 0.0009 sec since 20:41:31, monitored, thread 2881
>  > InodePrefetchWorkerThread: on ThCond 0x1800635A120 (LkObjCondvar),
>  > reason 'waiting for LX lock'
>  > Waiting 0.0009 sec since 20:41:31, monitored, thread 26231
>  > InodePrefetchWorkerThread: on ThCond 0x1800635A120 (LkObjCondvar),
>  > reason 'waiting for LX lock'
>  > Waiting 0.0009 sec since 20:41:31, monitored, thread 26146
>  > InodePrefetchWorkerThread: on ThCond 0x1800635A120 (LkObjCondvar),
>  > reason 'waiting for LX lock'
>  > Waiting 0.0009 sec since 20:41:31, monitored, thread 18637
>  > InodePrefetchWorkerThread: on ThCond 0x1800635A120 (LkObjCondvar),
>  > reason 'waiting for LX lock'
>  > Waiting 0.0009 sec since 20:41:31, monitored, thread 25013
>  > InodePrefetchWorkerThread: on ThCond 0x1800635A120 (LkObjCondvar),
>  > reason 'waiting for LX lock'
>  > Waiting 0.0009 sec since 20:41:31, monitored, thread 27879
>  > InodePrefetchWorkerThread: on ThCond 0x1800635A120 (LkObjCondvar),
>  > reason 'waiting for LX lock'
>  > Waiting 0.0009 sec since 20:41:31, monitored, thread 26553
>  > InodePrefetchWorkerThread: on ThCond 0x1800635A120 (LkObjCondvar),
>  > reason 'waiting for LX lock'
>  > Waiting 0.0009 sec since 20:41:31, monitored, thread 25334
>  > InodePrefetchWorkerThread: on ThCond 0x1800635A120 (LkObjCondvar),
>  > reason 'waiting for LX lock'
>  > Waiting 0.0009 sec since 20:41:31, monitored, thread 25337
>  > InodePrefetchWorkerThread: on ThCond 0x1800635A120 (LkObjCondvar),
>  > reason 'waiting for LX lock'
>  >
>  > and I'm wondering if there's anything immediate one would suggest to
>  > help with that.
>  >
>  > -Aaron
>  >
> 
> -- 
> Aaron Knister
> NASA Center for Climate Simulation (Code 606.2)
> Goddard Space Flight Center
> (301) 286-2776
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 
> 
> 
> 
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 

-- 
Aaron Knister
NASA Center for Climate Simulation (Code 606.2)
Goddard Space Flight Center
(301) 286-2776



More information about the gpfsug-discuss mailing list