[gpfsug-discuss] maxStatCache and maxFilesToCache: Tip"gpfs_maxstatcache_low".

[Philipp Grau]

Hello all,

thank you all a lot for the feedback to my question. I think that I now
understand the situation better. I will talk with my coworkers and we
will find a better setting for the values...

Many thanks,

Philipp

* Felipe Knop <knop at us.ibm.com> [13.03.20 16:22]:
> All,
>  
> Looks to me that the demands of the workload will dictate how many files we
> should be cache, that is: maxStatCache + maxFilesToCache .
>  
> The "mix" between maxStatCache and maxFilesToCache depends on how much memory
> can be made available. Accessing files from maxFilesToCache is more efficient,
> but stat cache entries use much less space.
>  
> With the
>  
>  ! maxFilesToCache 3000000
>     maxStatCache 10000
>  
> combination, the stat cache is not providing any significant help, since only
> 0.3% of the files that are cached can fit in the stat cache. If enough memory
> is available then maxStatCache could be increased to (say) 3000000, at a cost
> of 1.4GB.  But maxFilesToCache = 3000000 uses up to 27GB. The next questions
> are then
>  
> 1) Can such memory become available on the node, given the pagepool size ?
>  
> 2) Does the workload require caching that many files?
>  
>  
>   Felipe
>  
> ----
> Felipe Knop knop at us.ibm.com
> GPFS Development and Security
> IBM Systems
> IBM Building 008
> 2455 South Rd, Poughkeepsie, NY 12601
> (845) 433-9314 T/L 293-9314
>  
>  
>  
> 
>     ----- Original message -----
>     From: "Frederick Stock" <stockf at us.ibm.com>
>     Sent by: gpfsug-discuss-bounces at spectrumscale.org
>     To: gpfsug-discuss at spectrumscale.org
>     Cc: gpfsug-discuss at spectrumscale.org
>     Subject: [EXTERNAL] Re: [gpfsug-discuss] maxStatCache and maxFilesToCache:
>     Tip"gpfs_maxstatcache_low".
>     Date: Fri, Mar 13, 2020 10:01 AM
>      
>     As you have learned there is no simple formula for setting the
>     maxStatToCache, or for that matter the maxFilesToCache, configuration
>     values.  Memory is certainly one consideration but another is directory
>     listing operations.  The information kept in the stat cache is sufficient
>     for fulfilling directory listings.  If your users are doing directory
>     listings regularly then a larger stat cache could be helpful. 
> 
>     Fred
>     __________________________________________________
>     Fred Stock | IBM Pittsburgh Lab | 720-430-8821
>     stockf at us.ibm.com
>      
>      
> 
>         ----- Original message -----
>         From: Philipp Grau <phgrau at zedat.fu-berlin.de>
>         Sent by: gpfsug-discuss-bounces at spectrumscale.org
>         To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
>         Cc:
>         Subject: [EXTERNAL] [gpfsug-discuss] maxStatCache and maxFilesToCache:
>         Tip "gpfs_maxstatcache_low".
>         Date: Fri, Mar 13, 2020 8:49 AM
>          
>         Hello,
> 
>         we have a two node NSD cluster based on a DDN system.  Currently we
>         run Spectrum Scale 5.0.4.1 in an HPC environment.
> 
>         Mmhealth shows a tip stating "gpfs_maxstatcache_low". Our current
>         settings are:
> 
>         # mmdiag --config | grep -i cache
>          ! maxFilesToCache 3000000
>             maxStatCache 10000
> 
>         maxFilesToCache was tuned during installion and maxStatCache is the
>         according default value.
> 
>         After discussing this issue on the german spectumscale meeting, I
>         understand that it is difficult to give a formula on howto calulate
>         this values.
> 
>         But I learnt that a FilesToCache entry costs about 10 kbytes of memory
>         and a StatCache entry about 500 bytes. And typically maxStatCache
>         should (obviously) be greater than maxFilesToCache. There is a average
>         100 GB memory usage on our systems (with a total of 265 GB RAM).
> 
>         So setting maxStatCache to at least 3000000 should be no problem. But
>         is that correct or to high/low?
> 
>         Has anyone some hints or thoughts on this topic? Help is welcome.
> 
>         Regards,
> 
>         Philipp
>