<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div dir="ltr" >Hi Iban,</div>
<div dir="ltr" > </div>
<div dir="ltr" >I think it's necessary more information to be able to help you. Here they are:</div>
<div dir="ltr" >- Redhat version: Which is 7.2, 7.3 or 7.4?</div>
<div dir="ltr" >- Redhat kernel version: In the FAQ of GPFS has the recommended kernel levels</div>
<div dir="ltr" >- Platform: Is it x86_64?</div>
<div dir="ltr" >- Is there a reason for you stay in 4.2.3-6? Could you update to 4.2.3-9 or 5.0.1?</div>
<div dir="ltr" >- How is the name resolution? Can you do test ping from one node to another and it's reverse?</div>
<div dir="ltr" >- TCP/IP tuning: What is the TCP/IP parameters you are using? I have used for 7.4 the following:</div>
<div dir="ltr" >[root@XXXX sysctl.d]# cat 99-ibmscale.conf<br>net.core.somaxconn = 10000<br>net.core.netdev_max_backlog = 250000<br>net.ipv4.ip_local_port_range = 2000 65535<br>net.ipv4.tcp_rfc1337 = 1<br>net.ipv4.tcp_max_tw_buckets = 1440000<br>net.ipv4.tcp_mtu_probing = 1<br>net.ipv4.tcp_window_scaling = 1<br>net.ipv4.tcp_low_latency = 1<br>net.ipv4.tcp_max_syn_backlog = 4096<br>net.ipv4.tcp_fin_timeout = 10<br>net.core.rmem_default = 4194304<br>net.core.rmem_max = 4194304<br>net.core.wmem_default = 4194304<br>net.core.wmem_max = 4194304<br>net.core.optmem_max = 4194304<br>net.ipv4.tcp_rmem=4096 87380 16777216<br>net.ipv4.tcp_wmem=4096 65536 16777216<br>vm.min_free_kbytes = 512000<br>kernel.panic_on_oops = 0<br>net.ipv6.conf.all.disable_ipv6 = 1<br>net.ipv6.conf.default.disable_ipv6 = 1<br>vm.swappiness = 0<br>vm.dirty_ratio = 10<br> </div>
<div dir="ltr" >Since we disabled ipv6, we had to rebuild the kernel image with the following command:</div>
<div dir="ltr" >[root@XXXX ~]# dracut -f -v</div>
<div dir="ltr" > </div>
<div dir="ltr" >- GPFS tuning parameters: Can you list them?</div>
<div dir="ltr" >- Spectrum Scale status: Can you send the following outputs:</div>
<div dir="ltr" > mmgetstate -a -L</div>
<div dir="ltr" > mmlscluster</div>
<div dir="ltr" > mmhealth cluster show</div>
<div dir="ltr" > mmhealth cluster show --verbose</div>
<div dir="ltr" > mmhealth node eventlog</div>
<div dir="ltr" > mmlsnode -L -N waiters</div>
<div dir="ltr" > </div>
<div dir="ltr" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div dir="ltr" ><div> </div>
<table cellpadding="0" cellspacing="0" style="border:0;table-layout:auto;white-space:normal;color:#5f5f5f;width:650px;font-size:7.2pt" width="650px" > <tbody> <tr> <td style="vertical-align: bottom;" > <div style="font-size:9pt" ><span style="font-size:1.000em;" >Abraços / Regards / Saludos,</span></div>
<p> </p>
<div class="vcard" style="margin:0 0 8px 0" ><span style="font-size:1.000em;" ><b class="fn n" style="color:#888888;font-size:12pt" ><span class="given-name" >Anderson</span> <span class="family-name" >Nobre</span></b><br> AIX & Power Consultant<br> Master Certified IT Specialist<br> IBM Systems Hardware Client Technical Team – IBM Systems Lab Services<br> <br> <img alt="community_general_lab_services" class="org" height="120" src="http://esign.webmaster.ibm.com/img/signature_lab_services.jpg" width="333" ></span></div> </td> <td> </td> </tr> </tbody></table>
<table cellpadding="0" cellspacing="0" style="border:0;table-layout:auto;white-space:normal;color:#5f5f5f;width:650px;font-size:7.2pt" width="650px" > <tbody> <tr> <td colspan="3" > <hr style="border:0;color:#696969;background-color:#696969;height:1px;" ></td> </tr> <tr> <td style="vertical-align: top;font-family:arial,helvetica,sans-serif" width="400px" ><span style="font-size:1.000em;" ><span class="tel" ><span style="color:#466BB0;font-weight:bold" ><abbr class="type" title="work" >Phone:</abbr></span> <span class="value" >55-19-2132-4317</span></span><br> <span style="color:#466BB0;font-weight:bold;font-style:7.2pt" >E-mail:</span> <a class="email" href="mailto:anobre@br.ibm.com" target="_blank"><span class="value" style="color:#5f5f5f" >anobre@br.ibm.com</span></a></span></td> <td align="right" style="vertical-align: top;font-family:arial,helvetica,sans-serif" width="150px" ><span style="font-size:1.000em;" ><img alt="IBM" class="org" src="http://esign.webmaster.ibm.com/img/ibm3.gif" ></span></td> </tr> </tbody></table></div></div></div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Original message -----<br>From: Iban Cabrillo <cabrillo@ifca.unican.es><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: gpfsug-discuss@spectrumscale.org<br>Cc:<br>Subject: [gpfsug-discuss] Thousands of CLOSE_WAIT connections<br>Date: Fri, Jun 15, 2018 9:12 AM<br>
<div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; color: #000000" ><div>Dear,</div>
<div> </div>
<div> We have reinstall recently from gpfs 3.5 to SpectrumScale 4.2.3-6 version redhat 7.</div>
<div> </div>
<div> We are running two nsd servers and a a gui, there is no firewall on gpfs network, and selinux is disable, I have checked changing the manager and cluster manager node between server with the same result, server 01 always increase the CLOSE_WAIT :</div>
<div> </div>
<div> Node Daemon node name IP address Admin node name Designation<br>--------------------------------------------------------------------------------<br> 1 gpfs01.ifca.es 10.10.0.111 gpfs01.ifca.es quorum-manager-perfmon<br> 2 gpfs02.ifca.es 10.10.0.112 gpfs02.ifca.es quorum-manager-perfmon<br> 3 gpfsgui.ifca.es 10.10.0.60 gpfsgui.ifca.es quorum-perfmon<br>.......</div>
<div> </div>
<div>Installation and configuration works fine, but now we see that one of the servers do not close the mmfsd connections and this growing for ever while the othe nsd servers is always in the same range:</div>
<div> </div>
<div>[root@gpfs01 ~]# netstat -putana | grep 1191 | wc -l<br>19701</div>
<div>[root@gpfs01 ~]# netstat -putana | grep 1191 | grep CLOSE_WAIT| wc -l<br>19528</div>
<div>....</div>
<div>[root@gpfs02 ~]# netstat -putana | grep 1191 | wc -l<br>215<br>[root@gpfs02 ~]# netstat -putana | grep 1191 | grep CLOSE_WAIT| wc -l<br>0</div>
<div> </div>
<div>this is causing that gpfs01 do not answer to cluster commands</div>
<div> </div>
<div>NSD are balance between server (same size):</div>
<div>[root@gpfs02 ~]# mmlsnsd<br><br> File system Disk name NSD servers <br>---------------------------------------------------------------------------<br> gpfs nsd1 gpfs01,gpfs02<br> gpfs nsd2 gpfs01,gpfs02<br> gpfs nsd3 gpfs02,gpfs01<br> gpfs nsd4 gpfs02,gpfs01<br>.....</div>
<div> </div>
<div>proccess seems to be similar in both servers, only mmccr is running on server 1 and not in 2</div>
<div> </div>
<div>gpfs01</div>
<div>#######</div>
<div>root 9169 1 0 feb07 ? 22:27:54 python /usr/lpp/mmfs/bin/mmsysmon.py<br>root 11533 6154 0 13:41 ? 00:00:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmsdrquery sdrq_fs_info all<br>root 11713 1 0 13:41 ? 00:00:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15<br>root 12367 11533 0 13:43 ? 00:00:00 /usr/lpp/mmfs/bin/mmccr vget mmRunningCommand<br>root 12641 6162 0 13:44 ? 00:00:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmsdrquery sdrq_nsd_info sdrq_nsd_name:sdrq_fs_name:sdrq_storage_pool<br>root 12668 12641 0 13:44 ? 00:00:00 /usr/lpp/mmfs/bin/mmccr fget -c 835 mmsdrfs /var/mmfs/gen/mmsdrfs.12641<br>root 12950 11713 0 13:44 ? 00:00:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15<br>root 12959 9169 13 13:44 ? 00:00:00 /usr/lpp/mmfs/bin/mmccr check -Y -e<br>root 12968 3150 0 13:45 pts/3 00:00:00 grep --color=auto mm<br>root 19620 26468 38 jun14 ? 11:28:36 /usr/lpp/mmfs/bin/mmfsd<br>root 19701 2 0 jun14 ? 00:00:00 [mmkproc]<br>root 19702 2 0 jun14 ? 00:00:00 [mmkproc]<br>root 19703 2 0 jun14 ? 00:00:00 [mmkproc]<br>root 26468 1 0 jun05 ? 00:00:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/runmmfs</div>
<div> </div>
<div> </div>
<div>[root@gpfs02 ~]# ps -feA | grep mm<br>root 5074 1 0 feb07 ? 01:00:34 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15<br>root 5128 31456 28 jun14 ? 06:18:07 /usr/lpp/mmfs/bin/mmfsd<br>root 5255 2 0 jun14 ? 00:00:00 [mmkproc]<br>root 5256 2 0 jun14 ? 00:00:00 [mmkproc]<br>root 5257 2 0 jun14 ? 00:00:00 [mmkproc]<br>root 15196 5074 0 13:47 ? 00:00:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15<br>root 15265 13117 0 13:47 pts/0 00:00:00 grep --color=auto mm<br>root 31456 1 0 jun05 ? 00:00:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/runmmfs</div>
<div> </div>
<div>Any idea will be appreciated.</div>
<div>Regards, I</div>
<div> </div></div>
<div><font face="Default Monospace,Courier New,Courier,monospace" size="2" >_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></div></blockquote>
<div dir="ltr" > </div></div><BR>