[gpfsug-discuss] Ganesha daemon has 400'000 open files - is this unusual?
Billich Heinrich Rainer (ID SD)
heinrich.billich at id.ethz.ch
Tue Sep 24 09:52:34 BST 2019
Just some addition, maybe its of interest to someone: The number of max open files for Ganesha is based on maxFilesToCache. Its. 80%of maxFilesToCache up to an upper and lower limits of 2000/1M. The active setting is visible in /etc/sysconfig/ganesha.
On 19.09.19, 16:37, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Frederik Ferner" <gpfsug-discuss-bounces at spectrumscale.org on behalf of frederik.ferner at diamond.ac.uk> wrote:
we are seeing similar issues with CES/ganesha NFS, in our case it
exclusively with NFSv3 clients.
What is maxFilesToCache set to on your ganesha node(s)? In our case
ganesha was running into the limit of open file descriptors because
maxFilesToCache was set at a low default and for now we've increased it
It seemed that ganesha was never releasing files even after clients
unmounted the file system.
We've only recently made the change, so we'll see how much that improved
I thought we had a reproducer but after our recent change, I can now no
longer successfully reproduce the increase in open files not being released.
On 19/09/2019 15:20, Billich Heinrich Rainer (ID SD) wrote:
> Is it usual to see 200’000-400’000 open files for a single ganesha
> process? Or does this indicate that something ist wrong?
> We have some issues with ganesha (on spectrum scale protocol nodes)
> reporting NFS3ERR_IO in the log. I noticed that the affected nodes
> have a large number of open files, 200’000-400’000 open files per daemon
> (and 500 threads and about 250 client connections). Other nodes have
> 1’000 – 10’000 open files by ganesha only and don’t show the issue.
> If someone could explain how ganesha decides which files to keep open
> and which to close that would help, too. As NFSv3 is stateless the
> client doesn’t open/close a file, it’s the server to decide when to
> close it? We do have a few NFSv4 clients, too.
> Are there certain access patterns that can trigger such a large number
> of open file? Maybe traversing and reading a large number of small files?
> Thank you,
> I did count the open files by counting the entries in /proc/<pid of
> ganesha>/fd/ . With several 100k entries I failed to do a ‘ls -ls’ to
> list all the symbolic links, hence I can’t relate the open files to
> different exports easily.
> I did post this to the ganesha mailing list, too.
> Heinrich Billich
> ETH Zürich
> Tel.: +41 44 632 72 56
> heinrich.billich at id.ethz.ch
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
More information about the gpfsug-discuss