<font size=2 face="sans-serif">Forwarding Malahal's reply.</font><br><br><font size=2 face="Arial">Max open files of the ganesha process is
set internally (and written to /etc/sysconfig/ganesha file as NOFILE parameter)
based on MFTC (max files to cache) gpfs parameter.</font><br><br><font size=2 face="sans-serif">Regards, The Spectrum Scale (GPFS) team<br><br>------------------------------------------------------------------------------------------------------------------<br>If you feel that your question can benefit other users of Spectrum
Scale (GPFS), then please post it to the public IBM developerWroks Forum
at </font><a href="https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479"><font size=2 face="sans-serif">https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479</font></a><font size=2 face="sans-serif">.
<br><br>If your query concerns a potential software error in Spectrum Scale (GPFS)
and you have an IBM software maintenance contract please contact 1-800-237-5511
in the United States or your local IBM Service Center in other countries.
<br><br>The forum is informally monitored as time permits and should not be used
for priority messages to the Spectrum Scale (GPFS) team.</font><br><br><br><font size=2 face="sans-serif">>></font><br><br><font size=2 color=#5f5f5f face="Arial">From: </font><font size=2 face="Arial">Matt
Weil <mweil@wustl.edu></font><font size=2 color=#5f5f5f face="Arial"><br>To: </font><font size=2 face="Arial"><gpfsug-discuss@spectrumscale.org></font><font size=2 color=#5f5f5f face="Arial"><br>Date: </font><font size=2 face="Arial">03/24/2017 01:20 PM</font><font size=2 color=#5f5f5f face="Arial"><br>Subject: </font><font size=2 face="Arial">Re: [gpfsug-discuss] CES node
slow to respond</font><font size=2 color=#5f5f5f face="Arial"><br>Sent by: </font><font size=2 face="Arial">gpfsug-discuss-bounces@spectrumscale.org</font><font size=2 face="Arial"></font><hr><font size=3 face="Arial"><br>On 3/24/17 12:17 PM, IBM Spectrum Scale wrote:</font><br><font size=2 face="Arial">Hi Bryan,</font><font size=2 face="Arial"><br></font><font size=2 face="Arial"><br>Making sure Malahal's reply was received by the user group.</font><font size=2 face="Arial"><br></font><font size=2 face="Arial"><br>>> Then we noticed that the CES host had 5.4 million files open<br><br>This is technically not possible with ganesha alone. A process can only
open 1 million files on RHEL distro. Either we have leaks in kernel or
some other processes contributing to this.<br><br>Ganesha does keep NFSv3 files open and keep open for performance. It doesn't
have a good logic to close them after some inactivity. It does close them
if the number is close to max open files which is configurable.</font><br><font size=2 face="Arial">how do you set the max open files for Ganesha?</font><font size=3 face="Arial"></font><font size=2 face="Arial"> </font><br><font size=2 face="Arial"><br>PS: kNFS does open, read/write, and then close. No caching in older
versions. They did have a feature in a recent code to cache open files
in NFSv3 as well.</font><font size=2 face="Arial"><br></font><font size=2 face="Arial"><br>Regards, The Spectrum Scale (GPFS) team<br><br>------------------------------------------------------------------------------------------------------------------<br>If you feel that your question can benefit other users of Spectrum
Scale (GPFS), then please post it to the public IBM developerWroks Forum
at </font><a href="https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479" target=_blank><font size=2 color=blue face="Arial"><u>https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479</u></font></a><font size=2 face="Arial">.<br><br>If your query concerns a potential software error in Spectrum Scale (GPFS)
and you have an IBM software maintenance contract please contact 1-800-237-5511
in the United States or your local IBM Service Center in other countries.<br><br>The forum is informally monitored as time permits and should not be used
for priority messages to the Spectrum Scale (GPFS) team.</font><font size=2 face="Arial"><br><br><br></font><font size=1 color=#5f5f5f face="Arial"><br>From: </font><font size=1 face="Arial">Bryan
Banister </font><a href=mailto:bbanister@jumptrading.com target=_blank><font size=1 color=blue face="Arial"><u><bbanister@jumptrading.com></u></font></a><font size=1 color=#5f5f5f face="Arial"><br>To: </font><font size=1 face="Arial">gpfsug
main discussion list </font><a href="mailto:gpfsug-discuss@spectrumscale.org" target=_blank><font size=1 color=blue face="Arial"><u><gpfsug-discuss@spectrumscale.org></u></font></a><font size=1 color=#5f5f5f face="Arial"><br>Date: </font><font size=1 face="Arial">03/23/2017
08:27 AM</font><font size=1 color=#5f5f5f face="Arial"><br>Subject: </font><font size=1 face="Arial">Re:
[gpfsug-discuss] CES node slow to respond</font><font size=1 color=#5f5f5f face="Arial"><br>Sent by: </font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target=_blank><font size=1 color=blue face="Arial"><u>gpfsug-discuss-bounces@spectrumscale.org</u></font></a><font size=2 face="Arial"></font><hr noshade><font size=2 face="Arial"><br><br></font><tt><font size=3><br>Anybody from IBM willing/able to give us some explanation of why ganesha
is holding open so many files? Is this expected/needed/etc?<br><br>Or do we have to open a PMR to get some kind of explanation?<br>-B<br><br>-----Original Message-----<br>From: </font></tt><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target=_blank><tt><font size=3 color=blue><u>gpfsug-discuss-bounces@spectrumscale.org</u></font></tt></a><tt><font size=3>[</font></tt><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target=_blank><tt><font size=3 color=blue><u>mailto:gpfsug-discuss-bounces@spectrumscale.org</u></font></tt></a><tt><font size=3>]
On Behalf Of Matt Weil<br>Sent: Thursday, March 23, 2017 10:24 AM<br>To: </font></tt><a href="mailto:gpfsug-discuss@spectrumscale.org" target=_blank><tt><font size=3 color=blue><u>gpfsug-discuss@spectrumscale.org</u></font></tt></a><tt><font size=3><br>Subject: Re: [gpfsug-discuss] CES node slow to respond<br><br>FYI all<br><br>We also ran into this after bumping maxFilesToCache.<br><br>Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348:<br>unexpected error:<br>Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many<br>open files in system<br><br>fix<br><br>sysctl -w fs.file-max=1000000<br><br><br>On 3/22/17 12:01 PM, Bryan Banister wrote:<br>> We had a similar issue and also were instructed by IBM Support to
increase the maxFilesToCache to an insane value... Basically when the file
cache gets full then the host will spend all of its cycles looking for
a file to evict every time a new file is opened... baaaah.<br>><br>> Not sure why Ganesha has to keep so many files open... I can't believe
our NFS clients actually keep that many open. cNFS never needed this.<br>> -Bryan<br>><br>> -----Original Message-----<br>> From: </font></tt><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target=_blank><tt><font size=3 color=blue><u>gpfsug-discuss-bounces@spectrumscale.org</u></font></tt></a><tt><font size=3>[</font></tt><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target=_blank><tt><font size=3 color=blue><u>mailto:gpfsug-discuss-bounces@spectrumscale.org</u></font></tt></a><tt><font size=3>]
On Behalf Of Matt Weil<br>> Sent: Wednesday, March 22, 2017 11:43 AM<br>> To: gpfsug main discussion list </font></tt><a href="mailto:gpfsug-discuss@spectrumscale.org" target=_blank><tt><font size=3 color=blue><u><gpfsug-discuss@spectrumscale.org></u></font></tt></a><tt><font size=3><br>> Subject: [gpfsug-discuss] CES node slow to respond<br>><br>> All,<br>><br>> We had an indecent yesterday where one of our CES nodes slowed to
a<br>> crawl. GPFS waiters showed pre fetch threads going after inodes.<br>> iohist also showed lots of inode fetching. Then we noticed that
the CES<br>> host had 5.4 million files open.<br>><br>> The change I made was to set maxStatCache=DEFAULT because it is linux.<br>> And set maxFilesToCache=10000000 it was set to 500000. Then
restarted GPFS.<br>><br>> Is there something else we should change as well.<br>><br>> Thanks<br>><br>> Matt<br>><br>><br>> ________________________________<br>> The materials in this message are private and may contain Protected
Healthcare Information or other information of a sensitive nature. If you
are not the intended recipient, be advised that any unauthorized use, disclosure,
copying or the taking of any action in reliance on the contents of this
information is strictly prohibited. If you have received this email in
error, please immediately notify the sender via telephone or return mail.<br>> _______________________________________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at spectrumscale.org<br>> </font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target=_blank><tt><font size=3 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></tt></a><tt><font size=3><br>><br>> ________________________________<br>><br>> Note: This email is for the confidential use of the named addressee(s)
only and may contain proprietary, confidential or privileged information.
If you are not the intended recipient, you are hereby notified that any
review, dissemination or copying of this email is strictly prohibited,
and to please notify the sender immediately and destroy this email and
any attachments. Email transmission cannot be guaranteed to be secure or
error-free. The Company, therefore, does not make any guarantees as to
the completeness or accuracy of this email or any attachments. This email
is for informational purposes only and does not constitute a recommendation,
offer, request or solicitation of any kind to buy, sell, subscribe, redeem
or perform any type of transaction of a financial product.<br>> _______________________________________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at spectrumscale.org<br>> </font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target=_blank><tt><font size=3 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></tt></a><font size=2 face="Arial"><br><br></font><tt><font size=3><br>________________________________<br>The materials in this message are private and may contain Protected Healthcare
Information or other information of a sensitive nature. If you are not
the intended recipient, be advised that any unauthorized use, disclosure,
copying or the taking of any action in reliance on the contents of this
information is strictly prohibited. If you have received this email in
error, please immediately notify the sender via telephone or return mail.<br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org</font></tt><tt><font size=3 color=blue><u><br></u></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target=_blank><tt><font size=3 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></tt></a><font size=2 face="Arial"><br></font><tt><font size=3><br>________________________________<br><br>Note: This email is for the confidential use of the named addressee(s)
only and may contain proprietary, confidential or privileged information.
If you are not the intended recipient, you are hereby notified that any
review, dissemination or copying of this email is strictly prohibited,
and to please notify the sender immediately and destroy this email and
any attachments. Email transmission cannot be guaranteed to be secure or
error-free. The Company, therefore, does not make any guarantees as to
the completeness or accuracy of this email or any attachments. This email
is for informational purposes only and does not constitute a recommendation,
offer, request or solicitation of any kind to buy, sell, subscribe, redeem
or perform any type of transaction of a financial product.<br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org</font></tt><tt><font size=3 color=blue><u><br></u></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target=_blank><tt><font size=3 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></tt></a><font size=2 face="Arial"><br><br><br><br><br></font><tt><font size=3><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org</font></tt><tt><font size=3 color=blue><u><br></u></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target=_blank><tt><font size=3 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></tt></a><p><font size=3 face="Arial"> </font><div align=center><font size=2 face="Arial"> </font><hr></div><br><font size=1 color=#808080 face="Arial"><br>The materials in this message are private and may contain Protected Healthcare
Information or other information of a sensitive nature. If you are not
the intended recipient, be advised that any unauthorized use, disclosure,
copying or the taking of any action in reliance on the contents of this
information is strictly prohibited. If you have received this email in
error, please immediately notify the sender via telephone or return mail.</font><tt><font size=3>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org</font></tt><tt><font size=3 color=blue><u><br></u></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target=_blank><tt><font size=3 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></tt></a><br><font size=2 face="Arial"> </font><BR>