[gpfsug-discuss] maxStatCache and maxFilesToCache: Tip"gpfs_maxstatcache_low".

[Tomer Perry]

Hi,

Just remember to take into account the token memory aspect of the chosen 
values, especially for large clusters ( check 
https://www.spectrumscaleug.org/wp-content/uploads/2019/05/SSSD19DE-Day-2-B04-Spectrum-Scale-Memory-Usage.pdf 
 for more details).


Regards,

Tomer Perry





From:   Philipp Grau <phgrau at zedat.fu-berlin.de>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:   14/03/2020 15:12
Subject:        [EXTERNAL] Re: [gpfsug-discuss] maxStatCache and 
maxFilesToCache: Tip"gpfs_maxstatcache_low".
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Hello all,

thank you all a lot for the feedback to my question. I think that I now
understand the situation better. I will talk with my coworkers and we
will find a better setting for the values...

Many thanks,

Philipp

* Felipe Knop <knop at us.ibm.com> [13.03.20 16:22]:
> All,
> 
> Looks to me that the demands of the workload will dictate how many files 
we
> should be cache, that is: maxStatCache + maxFilesToCache .
> 
> The "mix" between maxStatCache and maxFilesToCache depends on how much 
memory
> can be made available. Accessing files from maxFilesToCache is more 
efficient,
> but stat cache entries use much less space.
> 
> With the
> 
>  ! maxFilesToCache 3000000
>     maxStatCache 10000
> 
> combination, the stat cache is not providing any significant help, since 
only
> 0.3% of the files that are cached can fit in the stat cache. If enough 
memory
> is available then maxStatCache could be increased to (say) 3000000, at a 
cost
> of 1.4GB.  But maxFilesToCache = 3000000 uses up to 27GB. The next 
questions
> are then
> 
> 1) Can such memory become available on the node, given the pagepool size 
?
> 
> 2) Does the workload require caching that many files?
> 
> 
>   Felipe
> 
> ----
> Felipe Knop knop at us.ibm.com
> GPFS Development and Security
> IBM Systems
> IBM Building 008
> 2455 South Rd, Poughkeepsie, NY 12601
> (845) 433-9314 T/L 293-9314
> 
> 
> 
> 
>     ----- Original message -----
>     From: "Frederick Stock" <stockf at us.ibm.com>
>     Sent by: gpfsug-discuss-bounces at spectrumscale.org
>     To: gpfsug-discuss at spectrumscale.org
>     Cc: gpfsug-discuss at spectrumscale.org
>     Subject: [EXTERNAL] Re: [gpfsug-discuss] maxStatCache and 
maxFilesToCache:
>     Tip"gpfs_maxstatcache_low".
>     Date: Fri, Mar 13, 2020 10:01 AM
> 
>     As you have learned there is no simple formula for setting the
>     maxStatToCache, or for that matter the maxFilesToCache, 
configuration
>     values.  Memory is certainly one consideration but another is 
directory
>     listing operations.  The information kept in the stat cache is 
sufficient
>     for fulfilling directory listings.  If your users are doing 
directory
>     listings regularly then a larger stat cache could be helpful. 
> 
>     Fred
>     __________________________________________________
>     Fred Stock | IBM Pittsburgh Lab | 720-430-8821
>     stockf at us.ibm.com
> 
> 
> 
>         ----- Original message -----
>         From: Philipp Grau <phgrau at zedat.fu-berlin.de>
>         Sent by: gpfsug-discuss-bounces at spectrumscale.org
>         To: gpfsug main discussion list 
<gpfsug-discuss at spectrumscale.org>
>         Cc:
>         Subject: [EXTERNAL] [gpfsug-discuss] maxStatCache and 
maxFilesToCache:
>         Tip "gpfs_maxstatcache_low".
>         Date: Fri, Mar 13, 2020 8:49 AM
> 
>         Hello,
> 
>         we have a two node NSD cluster based on a DDN system.  Currently 
we
>         run Spectrum Scale 5.0.4.1 in an HPC environment.
> 
>         Mmhealth shows a tip stating "gpfs_maxstatcache_low". Our 
current
>         settings are:
> 
>         # mmdiag --config | grep -i cache
>          ! maxFilesToCache 3000000
>             maxStatCache 10000
> 
>         maxFilesToCache was tuned during installion and maxStatCache is 
the
>         according default value.
> 
>         After discussing this issue on the german spectumscale meeting, 
I
>         understand that it is difficult to give a formula on howto 
calulate
>         this values.
> 
>         But I learnt that a FilesToCache entry costs about 10 kbytes of 
memory
>         and a StatCache entry about 500 bytes. And typically 
maxStatCache
>         should (obviously) be greater than maxFilesToCache. There is a 
average
>         100 GB memory usage on our systems (with a total of 265 GB RAM).
> 
>         So setting maxStatCache to at least 3000000 should be no 
problem. But
>         is that correct or to high/low?
> 
>         Has anyone some hints or thoughts on this topic? Help is 
welcome.
> 
>         Regards,
> 
>         Philipp
> 
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=mLPyKeOa1gNDrORvEXBgMw&m=OfAPrgGhpUsG419cks6FynnKaWgb84-BkzQXkaxraJw&s=HFKr-6vELN-qgWlEQsLH74gFoLShJnfLN89xNRNGocQ&e= 






-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20200316/8220d3ec/attachment.htm>