<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div dir="ltr" >I don't have an immediate answer, but I can look into it.<br> </div>
<div dir="ltr" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div dir="ltr" ><div style="font-size: 10pt; font-weight: bold; font-family: sans-serif;" ><div>Kevin D. Johnson, MBA, MAFM<br><span style="font-size: 0.786em;" >Spectrum Computing, Senior Managing Consultant<br><br><span style="color:#696969;" >IBM Certified Deployment Professional - Spectrum Scale V4.1.1<br>IBM Certified Deployment Professional - Cloud Object Storage V3.8</span></span></div>
<div><span style="color:#696969;" ><span style="font-size: 0.786em;" >IBM Certified Solution Advisor - Spectrum Computing V1</span></span></div>
<div> </div>
<div><span style="color:#696969;" ><span style="font-size: 0.786em;" >720.349.6199 - kevindjo@us.ibm.com</span></span></div></div>
<div style="font-size: 8pt; font-family: sans-serif; margin-top: 10px;" ><div> </div></div></div></div></div></div></div></div></div></div></div></div></div></div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Original message -----<br>From: Bryan Banister <bbanister@jumptrading.com><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Cc:<br>Subject: Re: [gpfsug-discuss] CES node slow to respond<br>Date: Thu, Mar 23, 2017 11:27 AM<br>
<div><font face="Default Monospace,Courier New,Courier,monospace" size="2" >Anybody from IBM willing/able to give us some explanation of why ganesha is holding open so many files? Is this expected/needed/etc?<br><br>Or do we have to open a PMR to get some kind of explanation?<br>-B<br><br>-----Original Message-----<br>From: gpfsug-discuss-bounces@spectrumscale.org [<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank" >mailto:gpfsug-discuss-bounces@spectrumscale.org</a>] On Behalf Of Matt Weil<br>Sent: Thursday, March 23, 2017 10:24 AM<br>To: gpfsug-discuss@spectrumscale.org<br>Subject: Re: [gpfsug-discuss] CES node slow to respond<br><br>FYI all<br><br>We also ran into this after bumping maxFilesToCache.<br><br>Mar 22 13:02:37 ces1 ntpd[1191]: ./../lib/isc/unix/ifiter_ioctl.c:348:<br>unexpected error:<br>Mar 22 13:02:37 ces1 ntpd[1191]: making interface scan socket: Too many<br>open files in system<br><br>fix<br><br>sysctl -w fs.file-max=1000000<br><br><br>On 3/22/17 12:01 PM, Bryan Banister wrote:<br>> We had a similar issue and also were instructed by IBM Support to increase the maxFilesToCache to an insane value... Basically when the file cache gets full then the host will spend all of its cycles looking for a file to evict every time a new file is opened... baaaah.<br>><br>> Not sure why Ganesha has to keep so many files open... I can't believe our NFS clients actually keep that many open. cNFS never needed this.<br>> -Bryan<br>><br>> -----Original Message-----<br>> From: gpfsug-discuss-bounces@spectrumscale.org [<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank" >mailto:gpfsug-discuss-bounces@spectrumscale.org</a>] On Behalf Of Matt Weil<br>> Sent: Wednesday, March 22, 2017 11:43 AM<br>> To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>> Subject: [gpfsug-discuss] CES node slow to respond<br>><br>> All,<br>><br>> We had an indecent yesterday where one of our CES nodes slowed to a<br>> crawl. GPFS waiters showed pre fetch threads going after inodes.<br>> iohist also showed lots of inode fetching. Then we noticed that the CES<br>> host had 5.4 million files open.<br>><br>> The change I made was to set maxStatCache=DEFAULT because it is linux.<br>> And set maxFilesToCache=10000000 it was set to 500000. Then restarted GPFS.<br>><br>> Is there something else we should change as well.<br>><br>> Thanks<br>><br>> Matt<br>><br>><br>> ________________________________<br>> The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.<br>> _______________________________________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at spectrumscale.org<br>> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank" >http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>><br>> ________________________________<br>><br>> Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.<br>> _______________________________________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at spectrumscale.org<br>> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank" >http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br><br><br>________________________________<br>The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.<br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank" >http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br><br>________________________________<br><br>Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.<br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank" >http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font><br> </div></blockquote>
<div dir="ltr" > </div></div><BR>