<html><body><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; color: #000000"><div>Hi Anderson,</div><div><br data-mce-bogus="1"></div><div>Comments are in line</div><div><br></div><hr id="zwchr" data-marker="__DIVIDER__"><div data-marker="__HEADERS__"><b>From: </b>"Anderson Ferreira Nobre" <anobre@br.ibm.com><br><b>To: </b>"gpfsug-discuss" <gpfsug-discuss@spectrumscale.org><br><b>Cc: </b>"gpfsug-discuss" <gpfsug-discuss@spectrumscale.org><br><b>Sent: </b>Friday, 15 June, 2018 16:49:14<br><b>Subject: </b>Re: [gpfsug-discuss] Thousands of CLOSE_WAIT connections<br></div><br><div data-marker="__QUOTED_TEXT__"><div class="socmaildefaultfont" dir="ltr" style="font-family: Arial, Helvetica, sans-serif; font-size: 10.5pt;" data-mce-style="font-family: Arial, Helvetica, sans-serif; font-size: 10.5pt;"><div dir="ltr">Hi Iban,</div>
<div dir="ltr"> </div>
<div dir="ltr">I think it's necessary more information to be able to help you. Here they are:</div>
<div dir="ltr">- Redhat version: Which is 7.2, 7.3 or 7.4?</div><div dir="ltr"> CentOS Linux release 7.5.1804 (Core) <br></div><div dir="ltr"><br data-mce-bogus="1"></div>
<div dir="ltr">- Redhat kernel version: In the FAQ of GPFS has the recommended kernel levels</div>
<div dir="ltr">- Platform: Is it x86_64?</div><div dir="ltr"> Yes it is</div>
<div dir="ltr">- Is there a reason for you stay in 4.2.3-6? Could you update to 4.2.3-9 or 5.0.1?</div><div dir="ltr"> No, that wasthe default version we get from our costumer we could upgrade to 4.2.3-9 with time...</div><div dir="ltr"><br data-mce-bogus="1"></div>
<div dir="ltr">- How is the name resolution? Can you do test ping from one node to another and it's reverse?</div><div dir="ltr"><br data-mce-bogus="1"></div><div dir="ltr"> yes resolution works fine in both directions (there is no firewall or icmp filter) using ethernet private network (not IB)</div><div dir="ltr"><br data-mce-bogus="1"></div>
<div dir="ltr">- TCP/IP tuning: What is the TCP/IP parameters you are using? I have used for 7.4 the following:</div>
<div dir="ltr">[root@XXXX sysctl.d]# cat 99-ibmscale.conf<br>net.core.somaxconn = 10000<br>net.core.netdev_max_backlog = 250000<br>net.ipv4.ip_local_port_range = 2000 65535<br>net.ipv4.tcp_rfc1337 = 1<br>net.ipv4.tcp_max_tw_buckets = 1440000<br>net.ipv4.tcp_mtu_probing = 1<br>net.ipv4.tcp_window_scaling = 1<br>net.ipv4.tcp_low_latency = 1<br>net.ipv4.tcp_max_syn_backlog = 4096<br>net.ipv4.tcp_fin_timeout = 10<br>net.core.rmem_default = 4194304<br>net.core.rmem_max = 4194304<br>net.core.wmem_default = 4194304<br>net.core.wmem_max = 4194304<br>net.core.optmem_max = 4194304<br>net.ipv4.tcp_rmem=4096 87380 16777216<br>net.ipv4.tcp_wmem=4096 65536 16777216<br>vm.min_free_kbytes = 512000<br>kernel.panic_on_oops = 0<br>net.ipv6.conf.all.disable_ipv6 = 1<br>net.ipv6.conf.default.disable_ipv6 = 1<br>vm.swappiness = 0<br>vm.dirty_ratio = 10<br> </div><div dir="ltr"><br data-mce-bogus="1"></div><div dir="ltr">That is mine:</div><div dir="ltr">net.ipv4.conf.default.accept_source_route = 0<br>net.core.somaxconn = 8192<br>net.ipv4.tcp_fin_timeout = 30<br>kernel.sysrq = 1<br>kernel.core_uses_pid = 1<br>net.ipv4.tcp_syncookies = 1<br>kernel.msgmnb = 65536<br>kernel.msgmax = 65536<br>kernel.shmmax = 13491064832<br>kernel.shmall = 4294967296<br>net.ipv4.neigh.default.gc_stale_time = 120<br>net.ipv4.tcp_synack_retries = 10<br>net.ipv4.tcp_sack = 0<br>net.ipv4.icmp_echo_ignore_broadcasts = 1<br>net.ipv6.conf.all.disable_ipv6 = 1<br>net.ipv6.conf.default.disable_ipv6 = 1<br>net.ipv6.conf.lo.disable_ipv6 = 1<br>net.core.netdev_max_backlog = 250000<br>net.core.rmem_default = 16777216<br>net.core.wmem_default = 16777216<br>net.core.rmem_max = 16777216<br>net.core.wmem_max = 16777216<br>net.ipv4.tcp_mem = 16777216 16777216 16777216<br>net.ipv4.tcp_rmem = 4096 87380 16777216<br>net.ipv4.tcp_wmem = 4096 87380 16777216<br>net.ipv4.tcp_adv_win_scale = 2<br>net.ipv4.tcp_low_latency = 1<br>net.ipv4.tcp_reordering = 3<br>net.ipv4.tcp_timestamps = 0<br>net.ipv4.tcp_window_scaling = 1<br>net.ipv4.tcp_max_syn_backlog = 8192<br>net.ipv4.neigh.default.gc_thresh1 = 30000<br>net.ipv4.neigh.default.gc_thresh2 = 32000<br>net.ipv4.neigh.default.gc_thresh3 = 32768<br>net.ipv4.conf.all.arp_filter = 1<br>net.ipv4.conf.all.arp_ignore = 1<br>net.ipv4.neigh.enp3s0.mcast_solicit = 9<br>net.ipv4.neigh.enp3s0.ucast_solicit = 9<br>net.ipv6.neigh.enp3s0.ucast_solicit = 9<br>net.ipv6.neigh.enp3s0.mcast_solicit = 9<br>net.ipv4.neigh.ib0.mcast_solicit = 18<br>vm.oom_dump_tasks = 1<br>vm.min_free_kbytes = 524288<br></div><div dir="ltr"><br data-mce-bogus="1"></div><div dir="ltr">Since we disabled ipv6, we had to rebuild the kernel image with the following command:</div>
<div dir="ltr">[root@XXXX ~]# dracut -f -v</div>
<div dir="ltr"> </div><div dir="ltr"> I did that on Wns but no on GPFS servers...</div>
<div dir="ltr">- GPFS tuning parameters: Can you list them?</div>
<div dir="ltr">- Spectrum Scale status: Can you send the following outputs:</div>
<div dir="ltr"> mmgetstate -a -L</div>
<div dir="ltr"> mmlscluster</div><div dir="ltr"><br data-mce-bogus="1"></div><div dir="ltr">[root@gpfs01 ~]# mmlscluster <br><br>GPFS cluster information<br>========================<br> GPFS cluster name: gpfsgui.ifca.es<br> GPFS cluster id: 8574383285738337182<br> GPFS UID domain: gpfsgui.ifca.es<br> Remote shell command: /usr/bin/ssh<br> Remote file copy command: /usr/bin/scp<br> Repository type: CCR<br><br> Node Daemon node name IP address Admin node name Designation<br>--------------------------------------------------------------------------------<br> 1 gpfs01.ifca.es 10.10.0.111 gpfs01.ifca.es quorum-manager-perfmon<br> 2 gpfs02.ifca.es 10.10.0.112 gpfs02.ifca.es quorum-manager-perfmon<br> 3 gpfsgui.ifca.es 10.10.0.60 gpfsgui.ifca.es quorum-perfmon<br> 9 cloudprv-02-9.ifca.es 10.10.140.26 cloudprv-02-9.ifca.es <br> 10 cloudprv-02-8.ifca.es 10.10.140.25 cloudprv-02-8.ifca.es <br> 13 node1.ifca.es 10.10.151.3 node3.ifca.es <br>......<br>44 node24.ifca.es 10.10.151.24 node24.ifca.es </div><div dir="ltr">.....</div>
<div dir="ltr"> mmhealth cluster show (It was shoutdown by hand)</div><div dir="ltr"><br data-mce-bogus="1"></div><div dir="ltr">[root@gpfs01 ~]# mmhealth cluster show --verbose<br><br>Error: The monitoring service is down and does not respond, please restart it.<br></div><div dir="ltr"><br data-mce-bogus="1"></div>
<div dir="ltr"> mmhealth cluster show --verbose</div><div dir="ltr"><br data-mce-bogus="1"></div>
<div dir="ltr"> mmhealth node eventlog</div><div dir="ltr">2018-06-12 23:31:31.487471 CET quorum_down ERROR The node is not able to form a quorum with the other available nodes.<br>2018-06-12 23:31:52.856082 CET ccr_local_server_ok INFO The local GPFS CCR server is reachable PC_LOCAL_SERVER<br>2018-06-12 23:33:06.397125 CET fs_remount_mount INFO The filesystem gpfs was mounted internal<br>2018-06-12 23:33:06.400622 CET fs_remount_mount INFO The filesystem gpfs was mounted remount<br>2018-06-12 23:33:06.787556 CET mounted_fs_check INFO The filesystem gpfs is mounted<br>2018-06-12 23:33:22.670023 CET quorum_up INFO Quorum achieved<br>2018-06-13 14:01:51.376885 CET service_removed INFO On the node gpfs01.ifca.es the threshold monitor was removed<br>2018-06-13 14:01:51.385115 CET service_removed INFO On the node gpfs01.ifca.es the perfmon monitor was removed<br>2018-06-13 18:41:55.846893 CET quorum_down ERROR The node is not able to form a quorum with the other available nodes.<br>2018-06-13 18:42:39.217545 CET fs_remount_mount INFO The filesystem gpfs was mounted internal<br>2018-06-13 18:42:39.221455 CET fs_remount_mount INFO The filesystem gpfs was mounted remount<br>2018-06-13 18:42:39.653778 CET mounted_fs_check INFO The filesystem gpfs is mounted<br>2018-06-13 18:42:55.956125 CET quorum_up INFO Quorum achieved<br>2018-06-13 18:43:17.448980 CET service_running INFO The service perfmon is running on node gpfs01.ifca.es<br>2018-06-13 18:51:14.157351 CET service_running INFO The service threshold is running on node gpfs01.ifca.es<br>2018-06-14 08:04:06.341564 CET ib_rdma_nic_unrecognized ERROR IB RDMA NIC mlx5_0/1 was not recognized<br>2018-06-14 08:04:30.216689 CET quorum_down ERROR The node is not able to form a quorum with the other available nodes.<br>2018-06-14 08:05:10.836900 CET fs_remount_mount INFO The filesystem gpfs was mounted internal<br>2018-06-14 08:05:27.135275 CET quorum_up INFO Quorum achieved<br>2018-06-14 08:05:40.446601 CET fs_remount_mount INFO The filesystem gpfs was mounted remount<br>2018-06-14 08:05:40.881064 CET mounted_fs_check INFO The filesystem gpfs is mounted<br>2018-06-14 08:08:56.455851 CET ib_rdma_nic_recognized INFO IB RDMA NIC mlx5_0/1 was recognized<br>2018-06-14 12:29:58.772033 CET ccr_quorum_nodes_warn WARNING At least one quorum node is not reachable Item=PC_QUORUM_NODES,ErrMsg='Ping CCR quorum nodes failed',Failed='10.10.0.112'<br>2018-06-14 15:41:57.860925 CET ccr_quorum_nodes_ok INFO All quorum nodes are reachable PC_QUORUM_NODES<br>2018-06-15 13:04:41.403505 CET pmcollector_down ERROR pmcollector service should be started and is stopped<br>2018-06-15 15:23:00.121760 CET quorum_down ERROR The node is not able to form a quorum with the other available nodes.<br>2018-06-15 15:23:43.616075 CET fs_remount_mount INFO The filesystem gpfs was mounted internal<br>2018-06-15 15:23:43.619593 CET fs_remount_mount INFO The filesystem gpfs was mounted remount<br>2018-06-15 15:23:44.053493 CET mounted_fs_check INFO The filesystem gpfs is mounted<br>2018-06-15 15:24:00.219003 CET quorum_up INFO Quorum achieved<br> </div><div dir="ltr"><br></div><div dir="ltr"> [root@gpfs02 ~]# mmhealth node eventlog <br>Error: The monitoring service is down and does not respond, please restart it.<br></div><div dir="ltr"><br data-mce-bogus="1"></div>
<div dir="ltr"> mmlsnode -L -N waiters</div>
<div dir="ltr"> </div>
<div dir="ltr"><div class="socmaildefaultfont" dir="ltr" style="font-family: Arial, Helvetica, sans-serif; font-size: 10.5pt;" data-mce-style="font-family: Arial, Helvetica, sans-serif; font-size: 10.5pt;"><div dir="ltr"><div> </div><div>non default parameters:</div><div><br data-mce-bogus="1"></div><div>[root@gpfs01 ~]# mmdiag --config | grep !<br> ! ccrEnabled 1<br> ! cipherList AUTHONLY<br> ! clusterId 8574383285738337182<br> ! clusterName gpfsgui.ifca.es<br> ! dmapiFileHandleSize 32<br> ! idleSocketTimeout 0<br> ! ignorePrefetchLUNCount 1<br> ! maxblocksize 16777216<br> ! maxFilesToCache 4000<br> ! maxInodeDeallocPrefetch 64<br> ! maxMBpS 6000<br> ! maxStatCache 512<br> ! minReleaseLevel 1700<br> ! myNodeConfigNumber 1<br> ! pagepool 17179869184<br> ! socketMaxListenConnections 512<br> ! socketRcvBufferSize 131072<br> ! socketSndBufferSize 65536<br> ! verbsPorts mlx5_0/1<br> ! verbsRdma enable<br> ! worker1Threads 256<br></div><div><br data-mce-bogus="1"></div><div>Regards, I</div>
<table cellpadding="0" cellspacing="0" style="border: 0; table-layout: auto; white-space: normal; color: #5f5f5f; width: 650px; font-size: 7.2pt;" width="650px" data-mce-style="border: 0; table-layout: auto; white-space: normal; color: #5f5f5f; width: 650px; font-size: 7.2pt;"> <tbody> <tr> <td style="vertical-align: bottom;" data-mce-style="vertical-align: bottom;"> <div style="font-size: 9pt;" data-mce-style="font-size: 9pt;"><span style="font-size: 1.000em;" data-mce-style="font-size: 1.000em;">Abraços / Regards / Saludos,</span></div>
<p> </p>
<div class="vcard" style="margin: 0 0 8px 0;" data-mce-style="margin: 0 0 8px 0;"><span style="font-size: 1.000em;" data-mce-style="font-size: 1.000em;"><b class="fn n" style="color: #888888; font-size: 12pt;" data-mce-style="color: #888888; font-size: 12pt;"><span class="given-name">Anderson</span> <span class="family-name">Nobre</span></b><br> AIX & Power Consultant<br> Master Certified IT Specialist<br> IBM Systems Hardware Client Technical Team – IBM Systems Lab Services<br> <br> <img alt="community_general_lab_services" class="org" height="120" width="333" src="http://esign.webmaster.ibm.com/img/signature_lab_services.jpg"></span></div> </td> <td> </td> </tr> </tbody></table>
<table cellpadding="0" cellspacing="0" style="border: 0; table-layout: auto; white-space: normal; color: #5f5f5f; width: 650px; font-size: 7.2pt;" width="650px" data-mce-style="border: 0; table-layout: auto; white-space: normal; color: #5f5f5f; width: 650px; font-size: 7.2pt;" data-mce-selected="1"> <tbody> <tr> <td colspan="3"> <hr style="border: 0; color: #696969; background-color: #696969; height: 1px;" data-mce-style="border: 0; color: #696969; background-color: #696969; height: 1px;"></td> </tr> <tr> <td style="vertical-align: top; font-family: arial,helvetica,sans-serif;" width="400px" data-mce-style="vertical-align: top; font-family: arial,helvetica,sans-serif;"><span style="font-size: 1.000em;" data-mce-style="font-size: 1.000em;"><span class="tel"><span style="color: #466bb0; font-weight: bold;" data-mce-style="color: #466bb0; font-weight: bold;"><abbr class="type" title="work">Phone:</abbr></span> <span class="value">55-19-2132-4317</span></span><br> <span style="color: #466bb0; font-weight: bold; font-style: 7.2pt;" data-mce-style="color: #466bb0; font-weight: bold; font-style: 7.2pt;">E-mail:</span> <a class="email" href="mailto:anobre@br.ibm.com" target="_blank"><span class="value" style="color: #5f5f5f;" data-mce-style="color: #5f5f5f;">anobre@br.ibm.com</span></a></span></td> <td align="right" style="vertical-align: top; font-family: arial,helvetica,sans-serif;" width="150px" data-mce-style="vertical-align: top; font-family: arial,helvetica,sans-serif;"><span style="font-size: 1.000em;" data-mce-style="font-size: 1.000em;"><img alt="IBM" class="org" src="http://esign.webmaster.ibm.com/img/ibm3.gif"></span></td> </tr> </tbody></table></div></div></div>
<div dir="ltr"> </div>
<div dir="ltr"> </div>
<blockquote dir="ltr" style="border-left: solid #aaaaaa 2px; margin-left: 5px; padding-left: 5px; direction: ltr; margin-right: 0px;" data-mce-style="border-left: solid #aaaaaa 2px; margin-left: 5px; padding-left: 5px; direction: ltr; margin-right: 0px;">----- Original message -----<br>From: Iban Cabrillo <cabrillo@ifca.unican.es><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: gpfsug-discuss@spectrumscale.org<br>Cc:<br>Subject: [gpfsug-discuss] Thousands of CLOSE_WAIT connections<br>Date: Fri, Jun 15, 2018 9:12 AM<br>
<div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; color: #000000;" data-mce-style="font-family: arial, helvetica, sans-serif; font-size: 12pt; color: #000000;"><div>Dear,</div>
<div> </div>
<div> We have reinstall recently from gpfs 3.5 to SpectrumScale 4.2.3-6 version redhat 7.</div>
<div> </div>
<div> We are running two nsd servers and a a gui, there is no firewall on gpfs network, and selinux is disable, I have checked changing the manager and cluster manager node between server with the same result, server 01 always increase the CLOSE_WAIT :</div>
<div> </div>
<div> Node Daemon node name IP address Admin node name Designation<br>--------------------------------------------------------------------------------<br> 1 gpfs01.ifca.es 10.10.0.111 gpfs01.ifca.es quorum-manager-perfmon<br> 2 gpfs02.ifca.es 10.10.0.112 gpfs02.ifca.es quorum-manager-perfmon<br> 3 gpfsgui.ifca.es 10.10.0.60 gpfsgui.ifca.es quorum-perfmon<br>.......</div>
<div> </div>
<div>Installation and configuration works fine, but now we see that one of the servers do not close the mmfsd connections and this growing for ever while the othe nsd servers is always in the same range:</div>
<div> </div>
<div>[root@gpfs01 ~]# netstat -putana | grep 1191 | wc -l<br>19701</div>
<div>[root@gpfs01 ~]# netstat -putana | grep 1191 | grep CLOSE_WAIT| wc -l<br>19528</div>
<div>....</div>
<div>[root@gpfs02 ~]# netstat -putana | grep 1191 | wc -l<br>215<br>[root@gpfs02 ~]# netstat -putana | grep 1191 | grep CLOSE_WAIT| wc -l<br>0</div>
<div> </div>
<div>this is causing that gpfs01 do not answer to cluster commands</div>
<div> </div>
<div>NSD are balance between server (same size):</div>
<div>[root@gpfs02 ~]# mmlsnsd<br><br> File system Disk name NSD servers <br>---------------------------------------------------------------------------<br> gpfs nsd1 gpfs01,gpfs02<br> gpfs nsd2 gpfs01,gpfs02<br> gpfs nsd3 gpfs02,gpfs01<br> gpfs nsd4 gpfs02,gpfs01<br>.....</div>
<div> </div>
<div>proccess seems to be similar in both servers, only mmccr is running on server 1 and not in 2</div>
<div> </div>
<div>gpfs01</div>
<div>#######</div>
<div>root 9169 1 0 feb07 ? 22:27:54 python /usr/lpp/mmfs/bin/mmsysmon.py<br>root 11533 6154 0 13:41 ? 00:00:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmsdrquery sdrq_fs_info all<br>root 11713 1 0 13:41 ? 00:00:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15<br>root 12367 11533 0 13:43 ? 00:00:00 /usr/lpp/mmfs/bin/mmccr vget mmRunningCommand<br>root 12641 6162 0 13:44 ? 00:00:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmsdrquery sdrq_nsd_info sdrq_nsd_name:sdrq_fs_name:sdrq_storage_pool<br>root 12668 12641 0 13:44 ? 00:00:00 /usr/lpp/mmfs/bin/mmccr fget -c 835 mmsdrfs /var/mmfs/gen/mmsdrfs.12641<br>root 12950 11713 0 13:44 ? 00:00:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15<br>root 12959 9169 13 13:44 ? 00:00:00 /usr/lpp/mmfs/bin/mmccr check -Y -e<br>root 12968 3150 0 13:45 pts/3 00:00:00 grep --color=auto mm<br>root 19620 26468 38 jun14 ? 11:28:36 /usr/lpp/mmfs/bin/mmfsd<br>root 19701 2 0 jun14 ? 00:00:00 [mmkproc]<br>root 19702 2 0 jun14 ? 00:00:00 [mmkproc]<br>root 19703 2 0 jun14 ? 00:00:00 [mmkproc]<br>root 26468 1 0 jun05 ? 00:00:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/runmmfs</div>
<div> </div>
<div> </div>
<div>[root@gpfs02 ~]# ps -feA | grep mm<br>root 5074 1 0 feb07 ? 01:00:34 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15<br>root 5128 31456 28 jun14 ? 06:18:07 /usr/lpp/mmfs/bin/mmfsd<br>root 5255 2 0 jun14 ? 00:00:00 [mmkproc]<br>root 5256 2 0 jun14 ? 00:00:00 [mmkproc]<br>root 5257 2 0 jun14 ? 00:00:00 [mmkproc]<br>root 15196 5074 0 13:47 ? 00:00:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15<br>root 15265 13117 0 13:47 pts/0 00:00:00 grep --color=auto mm<br>root 31456 1 0 jun05 ? 00:00:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/runmmfs</div>
<div> </div>
<div>Any idea will be appreciated.</div>
<div>Regards, I</div>
<div> </div></div>
<div><span face="Default Monospace,Courier New,Courier,monospace" size="2" data-mce-style="font-family: 'Default Monospace', 'Courier New', Courier, monospace; font-size: small;" style="font-family: 'Default Monospace', 'Courier New', Courier, monospace; font-size: small;">_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></span></div></blockquote>
<div dir="ltr"> </div></div><br>
<br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br>http://gpfsug.org/mailman/listinfo/gpfsug-discuss<br></div></div></body></html>