<font size=3 face="Arial">Assuming CentOS 7.5 parallels RHEL 7.5 then
you would need Spectrum Scale 4.2.3.9 because that is the release version
(along with 5.0.1 PTF1) that supports RHEL 7.5.</font><br><br><font size=3 face="sans-serif">Fred<br>__________________________________________________<br>Fred Stock | IBM Pittsburgh Lab | 720-430-8821<br>stockf@us.ibm.com</font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Iban Cabrillo <cabrillo@ifca.unican.es></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug-discuss <gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">06/15/2018 11:16 AM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
Thousands of CLOSE_WAIT connections</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><font size=3 face="Arial">Hi Anderson,</font><br><br><font size=3 face="Arial">Comments are in line</font><br><br><hr><br><font size=3 face="Arial"><b>From: </b>"Anderson Ferreira Nobre"
<anobre@br.ibm.com><b><br>To: </b>"gpfsug-discuss" <gpfsug-discuss@spectrumscale.org><b><br>Cc: </b>"gpfsug-discuss" <gpfsug-discuss@spectrumscale.org><b><br>Sent: </b>Friday, 15 June, 2018 16:49:14<b><br>Subject: </b>Re: [gpfsug-discuss] Thousands of CLOSE_WAIT connections</font><br><br><font size=2 face="Arial">Hi Iban,</font><br><font size=2 face="Arial"> </font><br><font size=2 face="Arial">I think it's necessary more information to
be able to help you. Here they are:</font><br><font size=2 face="Arial">- Redhat version: Which is 7.2, 7.3 or 7.4?</font><br><font size=2 face="Arial"> CentOS Linux release 7.5.1804
(Core) </font><br><br><font size=2 face="Arial">- Redhat kernel version: In the FAQ of GPFS
has the recommended kernel levels</font><br><font size=2 face="Arial">- Platform: Is it x86_64?</font><br><font size=2 face="Arial"> Yes it is</font><br><font size=2 face="Arial">- Is there a reason for you stay in 4.2.3-6?
Could you update to 4.2.3-9 or 5.0.1?</font><br><font size=2 face="Arial"> No, that wasthe default
version we get from our costumer we could upgrade to 4.2.3-9 with time...</font><br><br><font size=2 face="Arial">- How is the name resolution? Can you do
test ping from one node to another and it's reverse?</font><br><br><font size=2 face="Arial"> yes resolution works fine in
both directions (there is no firewall or icmp filter) using ethernet private
network (not IB)</font><br><br><font size=2 face="Arial">- TCP/IP tuning: What is the TCP/IP parameters
you are using? I have used for 7.4 the following:</font><br><font size=2 face="Arial">[root@XXXX sysctl.d]# cat 99-ibmscale.conf<br>net.core.somaxconn = 10000<br>net.core.netdev_max_backlog = 250000<br>net.ipv4.ip_local_port_range = 2000 65535<br>net.ipv4.tcp_rfc1337 = 1<br>net.ipv4.tcp_max_tw_buckets = 1440000<br>net.ipv4.tcp_mtu_probing = 1<br>net.ipv4.tcp_window_scaling = 1<br>net.ipv4.tcp_low_latency = 1<br>net.ipv4.tcp_max_syn_backlog = 4096<br>net.ipv4.tcp_fin_timeout = 10<br>net.core.rmem_default = 4194304<br>net.core.rmem_max = 4194304<br>net.core.wmem_default = 4194304<br>net.core.wmem_max = 4194304<br>net.core.optmem_max = 4194304<br>net.ipv4.tcp_rmem=4096 87380 16777216<br>net.ipv4.tcp_wmem=4096 65536 16777216<br>vm.min_free_kbytes = 512000<br>kernel.panic_on_oops = 0<br>net.ipv6.conf.all.disable_ipv6 = 1<br>net.ipv6.conf.default.disable_ipv6 = 1<br>vm.swappiness = 0<br>vm.dirty_ratio = 10<br> </font><br><br><font size=2 face="Arial">That is mine:</font><br><font size=2 face="Arial">net.ipv4.conf.default.accept_source_route
= 0<br>net.core.somaxconn = 8192<br>net.ipv4.tcp_fin_timeout = 30<br>kernel.sysrq = 1<br>kernel.core_uses_pid = 1<br>net.ipv4.tcp_syncookies = 1<br>kernel.msgmnb = 65536<br>kernel.msgmax = 65536<br>kernel.shmmax = 13491064832<br>kernel.shmall = 4294967296<br>net.ipv4.neigh.default.gc_stale_time = 120<br>net.ipv4.tcp_synack_retries = 10<br>net.ipv4.tcp_sack = 0<br>net.ipv4.icmp_echo_ignore_broadcasts = 1<br>net.ipv6.conf.all.disable_ipv6 = 1<br>net.ipv6.conf.default.disable_ipv6 = 1<br>net.ipv6.conf.lo.disable_ipv6 = 1<br>net.core.netdev_max_backlog = 250000<br>net.core.rmem_default = 16777216<br>net.core.wmem_default = 16777216<br>net.core.rmem_max = 16777216<br>net.core.wmem_max = 16777216<br>net.ipv4.tcp_mem = 16777216 16777216 16777216<br>net.ipv4.tcp_rmem = 4096 87380 16777216<br>net.ipv4.tcp_wmem = 4096 87380 16777216<br>net.ipv4.tcp_adv_win_scale = 2<br>net.ipv4.tcp_low_latency = 1<br>net.ipv4.tcp_reordering = 3<br>net.ipv4.tcp_timestamps = 0<br>net.ipv4.tcp_window_scaling = 1<br>net.ipv4.tcp_max_syn_backlog = 8192<br>net.ipv4.neigh.default.gc_thresh1 = 30000<br>net.ipv4.neigh.default.gc_thresh2 = 32000<br>net.ipv4.neigh.default.gc_thresh3 = 32768<br>net.ipv4.conf.all.arp_filter = 1<br>net.ipv4.conf.all.arp_ignore = 1<br>net.ipv4.neigh.enp3s0.mcast_solicit = 9<br>net.ipv4.neigh.enp3s0.ucast_solicit = 9<br>net.ipv6.neigh.enp3s0.ucast_solicit = 9<br>net.ipv6.neigh.enp3s0.mcast_solicit = 9<br>net.ipv4.neigh.ib0.mcast_solicit = 18<br>vm.oom_dump_tasks = 1<br>vm.min_free_kbytes = 524288</font><br><br><font size=2 face="Arial">Since we disabled ipv6, we had to rebuild
the kernel image with the following command:</font><br><font size=2 face="Arial">[root@XXXX ~]# dracut -f -v</font><br><font size=2 face="Arial"> </font><br><font size=2 face="Arial"> I did that on Wns but no on GPFS servers...</font><br><font size=2 face="Arial">- GPFS tuning parameters: Can you list them?</font><br><font size=2 face="Arial">- Spectrum Scale status: Can you send the
following outputs:</font><br><font size=2 face="Arial"> mmgetstate -a -L</font><br><font size=2 face="Arial"> mmlscluster</font><br><br><font size=2 face="Arial">[root@gpfs01 ~]# mmlscluster <br><br>GPFS cluster information<br>========================<br>GPFS cluster name: gpfsgui.ifca.es<br>GPFS cluster id: 8574383285738337182<br>GPFS UID domain: gpfsgui.ifca.es<br>Remote shell command: /usr/bin/ssh<br>Remote file copy command: /usr/bin/scp<br>Repository type: CCR<br><br>Node Daemon node name IP address Admin node name Designation<br>--------------------------------------------------------------------------------<br>1 gpfs01.ifca.es 10.10.0.111 gpfs01.ifca.es quorum-manager-perfmon<br>2 gpfs02.ifca.es 10.10.0.112 gpfs02.ifca.es quorum-manager-perfmon<br>3 gpfsgui.ifca.es 10.10.0.60 gpfsgui.ifca.es quorum-perfmon<br>9 cloudprv-02-9.ifca.es 10.10.140.26 cloudprv-02-9.ifca.es <br>10 cloudprv-02-8.ifca.es 10.10.140.25 cloudprv-02-8.ifca.es <br>13 node1.ifca.es 10.10.151.3 node3.ifca.es <br>......<br>44 node24.ifca.es 10.10.151.24 node24.ifca.es </font><br><font size=2 face="Arial">.....</font><br><font size=2 face="Arial"> mmhealth cluster show (It was shoutdown
by hand)</font><br><br><font size=2 face="Arial">[root@gpfs01 ~]# mmhealth cluster show --verbose<br><br>Error: The monitoring service is down and does not respond, please restart
it.</font><br><br><font size=2 face="Arial"> mmhealth cluster show --verbose</font><br><br><font size=2 face="Arial"> mmhealth node eventlog</font><br><font size=2 face="Arial">2018-06-12 23:31:31.487471 CET quorum_down
ERROR The node is not able to form a quorum with the other available nodes.<br>2018-06-12 23:31:52.856082 CET ccr_local_server_ok INFO The local GPFS
CCR server is reachable PC_LOCAL_SERVER<br>2018-06-12 23:33:06.397125 CET fs_remount_mount INFO The filesystem gpfs
was mounted internal<br>2018-06-12 23:33:06.400622 CET fs_remount_mount INFO The filesystem gpfs
was mounted remount<br>2018-06-12 23:33:06.787556 CET mounted_fs_check INFO The filesystem gpfs
is mounted<br>2018-06-12 23:33:22.670023 CET quorum_up INFO Quorum achieved<br>2018-06-13 14:01:51.376885 CET service_removed INFO On the node gpfs01.ifca.es
the threshold monitor was removed<br>2018-06-13 14:01:51.385115 CET service_removed INFO On the node gpfs01.ifca.es
the perfmon monitor was removed<br>2018-06-13 18:41:55.846893 CET quorum_down ERROR The node is not able to
form a quorum with the other available nodes.<br>2018-06-13 18:42:39.217545 CET fs_remount_mount INFO The filesystem gpfs
was mounted internal<br>2018-06-13 18:42:39.221455 CET fs_remount_mount INFO The filesystem gpfs
was mounted remount<br>2018-06-13 18:42:39.653778 CET mounted_fs_check INFO The filesystem gpfs
is mounted<br>2018-06-13 18:42:55.956125 CET quorum_up INFO Quorum achieved<br>2018-06-13 18:43:17.448980 CET service_running INFO The service perfmon
is running on node gpfs01.ifca.es<br>2018-06-13 18:51:14.157351 CET service_running INFO The service threshold
is running on node gpfs01.ifca.es<br>2018-06-14 08:04:06.341564 CET ib_rdma_nic_unrecognized ERROR IB RDMA NIC
mlx5_0/1 was not recognized<br>2018-06-14 08:04:30.216689 CET quorum_down ERROR The node is not able to
form a quorum with the other available nodes.<br>2018-06-14 08:05:10.836900 CET fs_remount_mount INFO The filesystem gpfs
was mounted internal<br>2018-06-14 08:05:27.135275 CET quorum_up INFO Quorum achieved<br>2018-06-14 08:05:40.446601 CET fs_remount_mount INFO The filesystem gpfs
was mounted remount<br>2018-06-14 08:05:40.881064 CET mounted_fs_check INFO The filesystem gpfs
is mounted<br>2018-06-14 08:08:56.455851 CET ib_rdma_nic_recognized INFO IB RDMA NIC
mlx5_0/1 was recognized<br>2018-06-14 12:29:58.772033 CET ccr_quorum_nodes_warn WARNING At least one
quorum node is not reachable Item=PC_QUORUM_NODES,ErrMsg='Ping CCR quorum
nodes failed',Failed='10.10.0.112'<br>2018-06-14 15:41:57.860925 CET ccr_quorum_nodes_ok INFO All quorum nodes
are reachable PC_QUORUM_NODES<br>2018-06-15 13:04:41.403505 CET pmcollector_down ERROR pmcollector service
should be started and is stopped<br>2018-06-15 15:23:00.121760 CET quorum_down ERROR The node is not able to
form a quorum with the other available nodes.<br>2018-06-15 15:23:43.616075 CET fs_remount_mount INFO The filesystem gpfs
was mounted internal<br>2018-06-15 15:23:43.619593 CET fs_remount_mount INFO The filesystem gpfs
was mounted remount<br>2018-06-15 15:23:44.053493 CET mounted_fs_check INFO The filesystem gpfs
is mounted<br>2018-06-15 15:24:00.219003 CET quorum_up INFO Quorum achieved<br> </font><br><br><font size=2 face="Arial"> [root@gpfs02 ~]# mmhealth node eventlog
<br>Error: The monitoring service is down and does not respond, please restart
it.</font><br><br><font size=2 face="Arial"> mmlsnode -L -N waiters</font><br><font size=2 face="Arial"> </font><br><font size=2 face="Arial"> </font><br><font size=2 face="Arial">non default parameters:</font><br><br><font size=2 face="Arial">[root@gpfs01 ~]# mmdiag --config | grep !<br>! ccrEnabled 1<br>! cipherList AUTHONLY<br>! clusterId 8574383285738337182<br>! clusterName gpfsgui.ifca.es<br>! dmapiFileHandleSize 32<br>! idleSocketTimeout 0<br>! ignorePrefetchLUNCount 1<br>! maxblocksize 16777216<br>! maxFilesToCache 4000<br>! maxInodeDeallocPrefetch 64<br>! maxMBpS 6000<br>! maxStatCache 512<br>! minReleaseLevel 1700<br>! myNodeConfigNumber 1<br>! pagepool 17179869184<br>! socketMaxListenConnections 512<br>! socketRcvBufferSize 131072<br>! socketSndBufferSize 65536<br>! verbsPorts mlx5_0/1<br>! verbsRdma enable<br>! worker1Threads 256</font><br><br><font size=2 face="Arial">Regards, I</font><table width=650 style="border-collapse:collapse;"><tr height=8><td width=644 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#5f5f5f>Abraços
/ Regards / Saludos,</font><p><font size=1 color=#5f5f5f> </font><br><font size=3 color=#8f8f8f><b>Anderson Nobre</b></font><font size=1 color=#5f5f5f><br>AIX & Power Consultant<br>Master Certified IT Specialist<br>IBM Systems Hardware Client Technical Team – IBM Systems Lab Services<br><br></font><img src=cid:_2_A4D26570A4D261980054C21D852582AD width=333 height=120 alt=community_general_lab_services style="border:0px solid;"><td width=5 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#5f5f5f> </font></table><br><table width=650 style="border-collapse:collapse;"><tr height=8><td width=650 colspan=2 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><hr><tr valign=top height=8><td width=472 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#4181c0 face="Arial"><b>Phone:</b></font><font size=1 color=#5f5f5f face="Arial">55-19-2132-4317</font><font size=1 color=#4181c0 face="Arial"><b><br>E-mail:</b></font><font size=1 color=#5f5f5f face="Arial"> </font><a href="mailto:anobre@br.ibm.com" target="_blank"><font size=1 color=#5f5f5f face="Arial"><u>anobre@br.ibm.com</u></font></a><td width=177 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><div align=right><img align=bottom src=cid:_1_A4D29598A4D28E980054C21D852582AD width=83 height=30 alt=IBM style="border:0px solid;"></div></table><br><font size=2 face="Arial"> </font><br><font size=2 face="Arial"> </font><br><font size=2 face="Arial">----- Original message -----<br>From: Iban Cabrillo <cabrillo@ifca.unican.es><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: gpfsug-discuss@spectrumscale.org<br>Cc:<br>Subject: [gpfsug-discuss] Thousands of CLOSE_WAIT connections<br>Date: Fri, Jun 15, 2018 9:12 AM<br> </font><br><font size=3 face="Arial">Dear,</font><br><font size=3 face="Arial"> </font><br><font size=3 face="Arial"> We have reinstall recently from gpfs
3.5 to SpectrumScale 4.2.3-6 version redhat 7.</font><br><font size=3 face="Arial"> </font><br><font size=3 face="Arial"> We are running two nsd servers and
a a gui, there is no firewall on gpfs network, and selinux is disable,
I have checked changing the manager and cluster manager node between server
with the same result, server 01 always increase the CLOSE_WAIT :</font><br><font size=3 face="Arial"> </font><br><font size=3 face="Arial"> Node Daemon node name
IP address Admin node name
Designation<br>--------------------------------------------------------------------------------<br> 1 gpfs01.ifca.es 10.10.0.111
gpfs01.ifca.es quorum-manager-perfmon<br> 2 gpfs02.ifca.es 10.10.0.112
gpfs02.ifca.es quorum-manager-perfmon<br> 3 gpfsgui.ifca.es 10.10.0.60
gpfsgui.ifca.es quorum-perfmon<br>.......</font><br><font size=3 face="Arial"> </font><br><font size=3 face="Arial">Installation and configuration works fine,
but now we see that one of the servers do not close the mmfsd connections
and this growing for ever while the othe nsd servers is always in the same
range:</font><br><font size=3 face="Arial"> </font><br><font size=3 face="Arial">[root@gpfs01 ~]# netstat -putana | grep 1191
| wc -l<br>19701</font><br><font size=3 face="Arial">[root@gpfs01 ~]# netstat -putana | grep 1191
| grep CLOSE_WAIT| wc -l<br>19528</font><br><font size=3 face="Arial">....</font><br><font size=3 face="Arial">[root@gpfs02 ~]# netstat -putana | grep 1191
| wc -l<br>215<br>[root@gpfs02 ~]# netstat -putana | grep 1191 | grep CLOSE_WAIT| wc -l</font><br><font size=3 face="Arial"> </font><br><font size=3 face="Arial">this is causing that gpfs01 do not answer
to cluster commands</font><br><font size=3 face="Arial"> </font><br><font size=3 face="Arial">NSD are balance between server (same size):</font><br><font size=3 face="Arial">[root@gpfs02 ~]# mmlsnsd<br><br> File system Disk name NSD servers
<br>---------------------------------------------------------------------------<br> gpfs nsd1
gpfs01,gpfs02<br> gpfs nsd2
gpfs01,gpfs02<br> gpfs nsd3
gpfs02,gpfs01<br> gpfs nsd4
gpfs02,gpfs01<br>.....</font><br><font size=3 face="Arial"> </font><br><font size=3 face="Arial">proccess seems to be similar in both servers,
only mmccr is running on server 1 and not in 2</font><br><font size=3 face="Arial"> </font><br><font size=3 face="Arial">gpfs01</font><br><font size=3 face="Arial">#######</font><br><font size=3 face="Arial">root 9169
1 0 feb07 ? 22:27:54 python /usr/lpp/mmfs/bin/mmsysmon.py<br>root 11533 6154 0 13:41 ?
00:00:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmsdrquery sdrq_fs_info
all<br>root 11713 1 0 13:41 ?
00:00:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor
15<br>root 12367 11533 0 13:43 ? 00:00:00
/usr/lpp/mmfs/bin/mmccr vget mmRunningCommand<br>root 12641 6162 0 13:44 ?
00:00:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmsdrquery sdrq_nsd_info
sdrq_nsd_name:sdrq_fs_name:sdrq_storage_pool<br>root 12668 12641 0 13:44 ? 00:00:00
/usr/lpp/mmfs/bin/mmccr fget -c 835 mmsdrfs /var/mmfs/gen/mmsdrfs.12641<br>root 12950 11713 0 13:44 ? 00:00:00
/usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15<br>root 12959 9169 13 13:44 ? 00:00:00
/usr/lpp/mmfs/bin/mmccr check -Y -e<br>root 12968 3150 0 13:45 pts/3 00:00:00
grep --color=auto mm<br>root 19620 26468 38 jun14 ? 11:28:36
/usr/lpp/mmfs/bin/mmfsd<br>root 19701 2 0 jun14 ?
00:00:00 [mmkproc]<br>root 19702 2 0 jun14 ?
00:00:00 [mmkproc]<br>root 19703 2 0 jun14 ?
00:00:00 [mmkproc]<br>root 26468 1 0 jun05 ?
00:00:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/runmmfs</font><br><font size=3 face="Arial"> </font><br><font size=3 face="Arial"> </font><br><font size=3 face="Arial">[root@gpfs02 ~]# ps -feA | grep mm<br>root 5074 1 0 feb07 ?
01:00:34 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor
15<br>root 5128 31456 28 jun14 ? 06:18:07
/usr/lpp/mmfs/bin/mmfsd<br>root 5255 2 0 jun14 ?
00:00:00 [mmkproc]<br>root 5256 2 0 jun14 ?
00:00:00 [mmkproc]<br>root 5257 2 0 jun14 ?
00:00:00 [mmkproc]<br>root 15196 5074 0 13:47 ?
00:00:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/mmccrmonitor 15<br>root 15265 13117 0 13:47 pts/0 00:00:00
grep --color=auto mm<br>root 31456 1 0 jun05 ?
00:00:00 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/runmmfs</font><br><font size=3 face="Arial"> </font><br><font size=3 face="Arial">Any idea will be appreciated.</font><br><font size=3 face="Arial">Regards, I</font><br><font size=3 face="Arial"> </font><br><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org</font></tt><tt><font size=2 color=blue><u><br></u></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><tt><font size=2 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></tt></a><br><font size=2 face="Arial"> </font><br><font size=3 face="Arial"><br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><font size=3 face="Arial">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></a><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br></font></tt><br><br><BR>