<font size=2 face="sans-serif">Hi Simon, </font><br><br><font size=2 face="sans-serif">this seems like a very good catch, and
the different symptoms seen can be a good explanation for the seen behaviour.
</font><br><br><font size=2 face="sans-serif">For whatever reason the iptables restart
was issued, it seemed to have caused a big number of expels, which sounds
logical, since the lease renewal requests might have been blocked by the
firewall at these points in time. the expels can be seen in the clustermgrs
mmfs.log </font><br><br><font size=2 face="sans-serif">Now if a node gets expelled, its share
of the Block and Inode Allocation Map need to be recovered by the filesystem
managers of the filessystems which were mounted on that expelled node earlier.
</font><br><font size=2 face="sans-serif">In your case, due to the fact that the
-numnodes parameter for the filesystem in question is high (2k) and the
number of filesets in that filesystem is also high (>1200) the Inode
Allocation Map is growing quite big, as it needs to hold a region
for every fileset per possibly mounting node (in order to be able to assign
this to the node mounting the FS, so that node can do allocations on its
own). </font><br><br><font size=2 face="sans-serif">This Inode Allocation Map File needs
to be scanned by the Inode alloc manager thread, whenever a node leaves
the cluster or the fs mgr is being moved, in order to recover that nodes
share, and to recalculate the amount of used inodes. During this reintialization
phase some operations are blocked (like mmlsfileset for example) . this
can lead to a almost hanging cluster. </font><br><br><font size=2 face="sans-serif">There are 2 APARs open on this, One
is already fixed (IJ11105) (watch out for changelog information :<b> </b></font><tt><font size=2><b>Fixed
slow inode expansion on large cluster</b></font></tt><font size=2 face="sans-serif">),
the second one is still in testing phase but soon to come.</font><br><font size=2 face="sans-serif">(changelog should read something like
: </font><font size=2 face="Arial"><b>Speed up inode alloc manager initialization)</b></font><br><br><font size=2 face="sans-serif">Preventing the expels will naturally
prevent those phases ;) </font><br><font size=1 face="Arial"><br>Mit freundlichen Grüßen / Kind regards</font><p><font size=2 face="Arial"><b>Achim Rehor</b></font><p><table width=600 style="border-collapse:collapse;"><tr height=8><td width=600 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><div align=center><hr noshade></div><tr height=8><td width=600 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 face="Arial"> </font><tr height=8><td width=496 colspan=3 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 face="Arial">Software
Technical Support Specialist AIX/ Emea HPC Support</font><td width=103 rowspan=10 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><div align=right><img align=bottom src=cid:_1_DCF516D0DCF5115000648428C125838A style="border:0px solid;"></div><tr height=8><td width=496 colspan=3 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 face="Arial">IBM
Certified Advanced Technical Expert - Power Systems with AIX</font><tr height=8><td width=496 colspan=3 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 face="Arial">TSCC
Software Service, Dept. 7922</font><tr height=8><td width=496 colspan=3 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 face="Arial">Global
Technology Services </font><tr height=8><td width=496 colspan=3 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><div align=center><hr noshade></div><tr height=8><td width=44 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial">Phone:</font><td width=197 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial">+49-7034-274-7862</font><td width=255 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial"> IBM
Deutschland</font><tr height=8><td width=44 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial">E-Mail:</font><td width=197 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial">Achim.Rehor@de.ibm.com</font><td width=255 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial"> Am
Weiher 24</font><tr height=8><td width=44 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial"> </font><td width=197 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial"> </font><td width=255 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial"> 65451
Kelsterbach</font><tr height=8><td width=44 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial"> </font><td width=197 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial"> </font><td width=255 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial"> Germany</font><tr height=8><td width=44 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial"> </font><td width=197 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial"> </font><td width=255 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial"> </font><tr height=8><td width=600 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><div align=center><hr noshade></div><tr height=8><td width=600 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 face="Arial"> </font><tr height=8><td width=600 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#a2a2a2 face="Arial">IBM
Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter <br>Geschäftsführung: Martin Hartmann (Vorsitzender), Norbert Janzen, Stefan
Lutz, Nicole Reimer, Dr. Klaus Seifert, Wolfgang Wendt <br>Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart,
HRB 14562 WEEE-Reg.-Nr. DE 99369940 </font></table><br><p><font size=3> </font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Simon Thompson <S.J.Thompson@bham.ac.uk></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug main discussion
list <gpfsug-discuss@spectrumscale.org>, "Tomer
Perry" <TOMP@il.ibm.com></font><br><font size=1 color=#5f5f5f face="sans-serif">Cc:
</font><font size=1 face="sans-serif">Yong Ze Chen <yongzech@cn.ibm.com></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">22/01/2019 15:35</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
Node expels</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><font size=2 face="sans-serif">OK we think we might have a reason for
this.</font><br><font size=2 face="sans-serif"> </font><br><font size=2 face="sans-serif">We run iptables on some of our management
function nodes, and we found that in some cases, our config management
tool can cause a ‘systemctl restart iptables’ to occur (the rule ordering
generation was non deterministic meaning it could shuffle rules … we fixed
that and made it reload rather than restart). Which takes a fraction of
a second, but it appears that this is sufficient for GPFS to get into a
state. What I didn’t mention before was that we could get it into a state
where the only way to recover was to shutdown the storage cluster and restart
it.</font><br><font size=2 face="sans-serif"> </font><br><font size=2 face="sans-serif">I’m not sure why normal expel and recovery
doesn’t appear to work in this case, though we’re not 100% certain that
its iptables restart. (we just have a very smoky gun at present). (I have
a ticket with that question open).</font><br><font size=2 face="sans-serif"> </font><br><font size=2 face="sans-serif">Maybe it’s a combination of having
a default DROP policy on iptables as well - we have also switched to ACCEPT
and added a DROP rule at the end of the ruleset which gives the same result.</font><br><font size=2 face="sans-serif"> </font><br><font size=2 face="sans-serif">Simon</font><br><font size=2 face="sans-serif"> </font><br><font size=3 face="sans-serif"><b>From: </b><gpfsug-discuss-bounces@spectrumscale.org>
on behalf of "jlewars@us.ibm.com" <jlewars@us.ibm.com><b><br>Reply-To: </b>"gpfsug-discuss@spectrumscale.org" <gpfsug-discuss@spectrumscale.org><b><br>Date: </b>Thursday, 17 January 2019 at 14:31<b><br>To: </b>Tomer Perry <TOMP@il.ibm.com>, "gpfsug-discuss@spectrumscale.org"
<gpfsug-discuss@spectrumscale.org><b><br>Cc: </b>Yong Ze Chen <yongzech@cn.ibm.com><b><br>Subject: </b>Re: [gpfsug-discuss] Node expels</font><br><font size=2 face="sans-serif"> </font><br><font size=2 face="Arial">></font><font size=2 face="sans-serif">They
always appear to be to a specific type of hardware with the same Ethernet
controller, </font><font size=2 face="sans-serif"><br></font><font size=2 face="Arial"><br>That makes me think you might be seeing packet loss that could require
ring buffer tuning (the defaults and limits will differ with different
ethernet adapters). </font><font size=2 face="sans-serif"><br></font><font size=2 face="Arial"><br>The expel section in the slides on this page has been expanded to include
a 'debugging expels section' (slides 19-20, which also reference ring buffer
tuning):</font><font size=2 color=blue face="sans-serif"><u><br></u></font><a href="https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/DEBUG%20Expels/comment/7e4f9433-7ca3-430f-b40b-94777c507381"><font size=2 color=blue face="Arial"><u>https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/DEBUG%20Expels/comment/7e4f9433-7ca3-430f-b40b-94777c507381</u></font></a><font size=2 face="sans-serif"><br></font><font size=2 face="Arial"><br>Regards,<br>John Lewars <br>Spectrum Scale Performance, IBM Poughkeepsie</font><font size=2 face="sans-serif"><br><br><br><br></font><font size=1 color=#5f5f5f face="Arial"><br>From: </font><font size=1 face="Arial">Tomer
Perry/Israel/IBM</font><font size=1 color=#5f5f5f face="Arial"><br>To: </font><font size=1 face="Arial">gpfsug
main discussion list <gpfsug-discuss@spectrumscale.org></font><font size=1 color=#5f5f5f face="Arial"><br>Cc: </font><font size=1 face="Arial">John Lewars/Poughkeepsie/IBM@IBMUS,
Yong Ze Chen/China/IBM@IBMCN</font><font size=1 color=#5f5f5f face="Arial"><br>Date: </font><font size=1 face="Arial">01/17/2019
08:28 AM</font><font size=1 color=#5f5f5f face="Arial"><br>Subject: </font><font size=1 face="Arial">Re:
[gpfsug-discuss] Node expels</font><div align=center><hr noshade></div><br><font size=2 face="sans-serif"><br></font><font size=2 face="Arial"><br>Hi,</font><font size=2 face="sans-serif"><br></font><font size=2 face="Arial"><br>I was asked to elaborate a bit ( thus also adding John and Yong Ze Chen).</font><font size=2 face="sans-serif"><br></font><font size=2 face="Arial"><br>As written on the slide:</font><font size=3 face="sans-serif"><br>One of the best ways to determine if a network layer problem is root cause
for an expel is to look at the low-level socket details dumped in the ‘extra’
log data (mmfs dump all) saved as part of automatic data collection on
Linux GPFS nodes. </font><font size=2 face="sans-serif"><br></font><font size=2 face="Arial"><br>So, the idea is that in expel situation, we dump the socket state from
the OS ( you can see the same using 'ss -i' for example).<br>In your example, it shows that the ca_state is 4, there are retransmits,
high rto and all the point to a network problem.<br>You can find more details here: </font><a href="http://www.yonch.com/tech/linux-tcp-congestion-control-internals"><font size=2 color=blue face="Arial"><u>http://www.yonch.com/tech/linux-tcp-congestion-control-internals</u></font></a><font size=2 face="sans-serif"><br></font><font size=2 face="Arial"><br><br>Regards,<br><br>Tomer Perry<br>Scalable I/O Development (Spectrum Scale)<br>email: tomp@il.ibm.com<br>1 Azrieli Center, Tel Aviv 67021, Israel<br>Global Tel: +1 720 3422758<br>Israel Tel: +972 3 9188625<br>Mobile: +972 52 2554625</font><font size=2 face="sans-serif"><br><br><br><br><br></font><font size=1 color=#5f5f5f face="Arial"><br>From: </font><font size=1 face="Arial">"Tomer
Perry" <TOMP@il.ibm.com></font><font size=1 color=#5f5f5f face="Arial"><br>To: </font><font size=1 face="Arial">gpfsug
main discussion list <gpfsug-discuss@spectrumscale.org></font><font size=1 color=#5f5f5f face="Arial"><br>Date: </font><font size=1 face="Arial">17/01/2019
13:46</font><font size=1 color=#5f5f5f face="Arial"><br>Subject: </font><font size=1 face="Arial">Re:
[gpfsug-discuss] Node expels</font><font size=1 color=#5f5f5f face="Arial"><br>Sent by: </font><font size=1 face="Arial">gpfsug-discuss-bounces@spectrumscale.org</font><div align=center><hr noshade></div><br><font size=2 face="sans-serif"><br><br></font><font size=2 face="Arial"><br>Simon,<br><br>Take a look at </font><a href="http://files.gpfsug.org/presentations/2018/USA/Scale_Network_Flow-0.8.pdf"><font size=2 color=blue face="Arial"><u>http://files.gpfsug.org/presentations/2018/USA/Scale_Network_Flow-0.8.pdf</u></font></a><font size=2 face="Arial">slide
13.<br><br><br>Regards,<br><br>Tomer Perry<br>Scalable I/O Development (Spectrum Scale)<br>email: tomp@il.ibm.com<br>1 Azrieli Center, Tel Aviv 67021, Israel<br>Global Tel: +1 720 3422758<br>Israel Tel: +972 3 9188625<br>Mobile: +972 52 2554625</font><font size=3 face="sans-serif"><br><br><br></font><font size=1 color=#5f5f5f face="Arial"><br><br>From: </font><font size=1 face="Arial">Simon
Thompson <S.J.Thompson@bham.ac.uk></font><font size=1 color=#5f5f5f face="Arial"><br>To: </font><font size=1 face="Arial">"gpfsug-discuss@spectrumscale.org"
<gpfsug-discuss@spectrumscale.org></font><font size=1 color=#5f5f5f face="Arial"><br>Date: </font><font size=1 face="Arial">17/01/2019
13:35</font><font size=1 color=#5f5f5f face="Arial"><br>Subject: </font><font size=1 face="Arial">[gpfsug-discuss]
Node expels</font><font size=1 color=#5f5f5f face="Arial"><br>Sent by: </font><font size=1 face="Arial">gpfsug-discuss-bounces@spectrumscale.org</font><div align=center><hr noshade></div><br><font size=3 face="sans-serif"><br></font><font size=2 face="sans-serif"><br><br>We’ve recently been seeing quite a few node expels with messages of the
form:<br><br>2019-01-17_11:19:30.882+0000: [W] The TCP connection to IP address 10.20.0.58
proto-pg-pf01.bear.cluster <c0n236> (socket 153) state is unexpected:
state=1 ca_state=4 snd_cwnd=1 snd_ssthresh=5 unacked=5 probes=0 backoff=7
retransmits=7 rto=26496000 rcv_ssthresh=102828 rtt=6729 rttvar=12066 sacked=0
retrans=1 reordering=3 lost=5<br>2019-01-17_11:19:30.882+0000: [I] tscCheckTcpConn: Sending debug data collection
request to node 10.20.0.58 proto-pg-pf01.bear.cluster<br>2019-01-17_11:19:30.882+0000: Sending request to collect TCP debug data
to proto-pg-pf01.bear.cluster localNode<br>2019-01-17_11:19:30.882+0000: [I] Calling user exit script gpfsSendRequestToNodes:
event sendRequestToNodes, Async command /usr/lpp/mmfs/bin/mmcommon.<br>2019-01-17_11:24:52.611+0000: [E] Timed out in 300 seconds waiting for
a commMsgCheckMessages reply from node 10.20.0.58 proto-pg-pf01.bear.cluster.
Sending expel message.<br><br>On the client node, we see messages of the form:<br><br>2019-01-17_11:19:31.101+0000: [N] sdrServ: Received Tcp data collection
request from 10.10.0.33<br>2019-01-17_11:19:31.102+0000: [N] GPFS will attempt to collect Tcp debug
data on this node.<br>2019-01-17_11:24:52.838+0000: [N] sdrServ: Received expel data collection
request from 10.10.0.33<br>2019-01-17_11:24:52.838+0000: [N] GPFS will attempt to collect debug data
on this node.<br>2019-01-17_11:25:02.741+0000: [N] This node will be expelled from cluster
rds.gpfs.servers due to expel msg from 10.10.12.41 (b<br>ber-les-nsd01-data.bb2.cluster in rds.gpfs.server<br>2019-01-17_11:25:03.160+0000: [N] sdrServ: Received expel data collection
request from 10.20.0.56<br><br>They always appear to be to a specific type of hardware with the same Ethernet
controller, though the nodes are split across three data centres and we
aren’t seeing link congestion on the links between them.<br><br>On the node I listed above, it’s not actually doing anything either as
the software on it is still being installed (i.e. it’s not doing GPFS
or any other IO other than a couple of home directories).<br><br>Any suggestions on what “(socket 153) state is unexpected” means?<br><br>Thanks<br><br>Simon<br></font><font size=2 face="Courier New"><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org</font><font size=2 color=blue face="sans-serif"><u><br></u></font><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><font size=2 color=blue face="Courier New"><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></a><font size=3 face="sans-serif"><br><br></font><font size=2 face="Courier New"><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org</font><font size=2 color=blue face="sans-serif"><u><br></u></font><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><font size=2 color=blue face="Courier New"><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></a><font size=2 face="sans-serif"><br><br><br><br><br></font><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br></font></tt><br><br><BR>