<font size=2 face="sans-serif">Jake,</font><br><br><font size=2 face="sans-serif">If AFM is using NFS it is all about
NFS tuning. The copy from one side to the other is basically just a client
writing to an NFS mount. Thee are a few things you can look at:</font><br><font size=2 face="sans-serif">1. NFS Transfer size (Make is 1MiB,
I think that is the max)</font><br><font size=2 face="sans-serif">2. TCP Tuning for large window size.
This is discussed on </font><a href=http://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adv.doc/bl1adv_afmtuning.htm><font size=2 color=blue face="sans-serif">Tuning
active file management home communications</font></a><font size=2 face="sans-serif">in the docs. On this page you will find some discussion on increasing gateway
threads, and other things similar that may help as well.</font><br><font size=2 face="sans-serif"><br>We can discuss further as I understand we will be meeting at SC16.</font><br><br><font size=2 face="sans-serif">Scott Fadden<br>Spectrum Scale - Technical Marketing <br>Phone: (503) 880-5833 <br>sfadden@us.ibm.com<br></font><a href=http://www.ibm.com/systems/storage/spectrum/scale><font size=2 face="sans-serif">http://www.ibm.com/systems/storage/spectrum/scale</font></a><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Jake Carroll <jake.carroll@uq.edu.au></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">"gpfsug-discuss@spectrumscale.org"
<gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">11/09/2016 09:39 AM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">[gpfsug-discuss]
Tuning AFM for high throughput/high IO over _really_
long distances</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><font size=2 face="Calibri">Hi.</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">I’ve got an GPFS to GPFS AFM cache/home
(IW) relationship set up over a really long distance. About 180ms of latency
between the two clusters and around 13,000km of optical path. Fortunately
for me, I’ve actually got near theoretical maximum IO over the NIC’s
between the clusters and I’m iPerf’ing at around 8.90 to 9.2Gbit/sec
over a 10GbE circuit. All MTU9000 all the way through.</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">Anyway – I’m finding my AFM traffic to
be dragging its feet and I don’t really understand why that might be.
I’ve verified the links and transports ability as I said above with iPerf,
and CERN’s FDT to near 10Gbit/sec. </font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">I also verified the clusters on both sides
in terms of disk IO and they both seem easily capable in IOZone and IOR
tests of multiple GB/sec of throughput.</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">So – my questions:</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">1. Are there very
specific tunings AFM needs for high latency/long distance IO? </font><br><font size=2 face="Calibri">2. Are there very
specific NIC/TCP-stack tunings (beyond the type of thing we already have
in place) that benefits AFM over really long distances and high latency?</font><br><font size=2 face="Calibri">3. We are seeing on
the “cache” side really lazy/sticky “ls –als” in the home mount. It
sometimes takes 20 to 30 seconds before the command line will report back
with a long listing of files. Any ideas why it’d take that long to get
a response from “home”.</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">We’ve got our TCP stack setup fairly aggressively,
on all hosts that participate in these two clusters.</font><br><font size=2> </font><br><font size=2>ethtool -C enp2s0f0 adaptive-rx off</font><br><font size=2>ifconfig enp2s0f0 txqueuelen 10000</font><br><font size=2>sysctl -w net.core.rmem_max=536870912</font><br><font size=2>sysctl -w net.core.wmem_max=536870912</font><br><font size=2>sysctl -w net.ipv4.tcp_rmem="4096 87380 268435456"</font><br><font size=2>sysctl -w net.ipv4.tcp_wmem="4096 65536 268435456"</font><br><font size=2>sysctl -w net.core.netdev_max_backlog=250000</font><br><font size=2>sysctl -w net.ipv4.tcp_congestion_control=htcp</font><br><font size=2>sysctl -w net.ipv4.tcp_mtu_probing=1</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">I modified a couple of small things on
the AFM “cache” side to see if it’d make a difference such as:</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">mmchconfig afmNumWriteThreads=4</font><br><font size=2 face="Calibri">mmchconfig afmNumReadThreads=4</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">But no difference so far.</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">Thoughts would be appreciated. I’ve done
this before over much shorter distances (30Km) and I’ve flattened a 10GbE
wire without really tuning…anything. Are my large in-flight-packets numbers/long-time-to-acknowledgement
semantics going to hurt here? I really thought AFM might be well designed
for exactly this kind of work at long distance *<b>and</b>* high throughput
– so I must be missing something!</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">-jc</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri"> </font><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br></font></tt><br><br><BR>