[gpfsug-discuss] Ganesha daemon has 400'000 open files - is this unusual?

[Billich Heinrich Rainer (ID SD)]

Hello Frederik,

Just some addition, maybe its of interest to someone:  The number of max open files for Ganesha is based on maxFilesToCache. Its. 80%of maxFilesToCache up to an upper and lower limits of 2000/1M. The active setting is visible in /etc/sysconfig/ganesha.

Cheers,

Heiner

On 19.09.19, 16:37, "gpfsug-discuss-bounces at spectrumscale.org on behalf of Frederik Ferner" <gpfsug-discuss-bounces at spectrumscale.org on behalf of frederik.ferner at diamond.ac.uk> wrote:

    Heiner,
    
    we are seeing similar issues with CES/ganesha NFS, in our case it 
    exclusively with NFSv3 clients.
    
    What is maxFilesToCache set to on your ganesha node(s)? In our case 
    ganesha was running into the limit of open file descriptors because 
    maxFilesToCache was set at a low default and for now we've increased it 
    to 1M.
    
    It seemed that ganesha was never releasing files even after clients 
    unmounted the file system.
    
    We've only recently made the change, so we'll see how much that improved 
    the situation.
    
    I thought we had a reproducer but after our recent change, I can now no 
    longer successfully reproduce the increase in open files not being released.
    
    Kind regards,
    Frederik
    
    On 19/09/2019 15:20, Billich  Heinrich Rainer (ID SD) wrote:
    > Hello,
    > 
    > Is it usual to see 200’000-400’000 open files for a single ganesha 
    > process? Or does this indicate that something ist wrong?
    > 
    > We have some issues with ganesha (on spectrum scale protocol nodes) 
    >   reporting NFS3ERR_IO in the log. I noticed that the affected nodes 
    > have a large number of open files, 200’000-400’000 open files per daemon 
    > (and 500 threads and about 250 client connections). Other nodes have 
    > 1’000 – 10’000 open files by ganesha only and don’t show the issue.
    > 
    > If someone could explain how ganesha decides which files to keep open 
    > and which to close that would help, too. As NFSv3 is stateless the 
    > client doesn’t open/close a file, it’s the server to decide when to 
    > close it? We do have a few NFSv4 clients, too.
    > 
    > Are there certain access patterns that can trigger such a large number 
    > of open file? Maybe traversing and reading a large number of small files?
    > 
    > Thank you,
    > 
    > Heiner
    > 
    > I did count the open files  by counting the entries in /proc/<pid of 
    > ganesha>/fd/ . With several 100k entries I failed to do a ‘ls -ls’ to 
    > list all the symbolic links, hence I can’t relate the open files to 
    > different exports easily.
    > 
    > I did post this to the ganesha mailing list, too.
    > 
    > -- 
    > 
    > =======================
    > 
    > Heinrich Billich
    > 
    > ETH Zürich
    > 
    > Informatikdienste
    > 
    > Tel.: +41 44 632 72 56
    > 
    > heinrich.billich at id.ethz.ch
    > 
    > ========================
    > 
    > 
    > _______________________________________________
    > gpfsug-discuss mailing list
    > gpfsug-discuss at spectrumscale.org
    > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
    > 
    
    
    
    -- 
    This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
    Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. 
    Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
    Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
    _______________________________________________
    gpfsug-discuss mailing list
    gpfsug-discuss at spectrumscale.org
    http://gpfsug.org/mailman/listinfo/gpfsug-discuss