<font size=2 face="sans-serif">So you are using the NSD protocol for data
transfers over multi-cluster? If so the TCP and thread tuning should help
as well. </font><br><br><br><font size=2 face="sans-serif">Scott Fadden<br>Spectrum Scale - Technical Marketing <br>Phone: (503) 880-5833 <br>sfadden@us.ibm.com<br></font><a href=http://www.ibm.com/systems/storage/spectrum/scale><font size=2 face="sans-serif">http://www.ibm.com/systems/storage/spectrum/scale</font></a><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Jake Carroll <jake.carroll@uq.edu.au></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">"gpfsug-discuss@spectrumscale.org"
<gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">11/09/2016 10:09 AM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
Tuning AFM for high throughput/high IO over _really_ long distances (Jan-Frode
Myklebust)</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><tt><font size=2>Hi jf$B!D(B<br><br> <br>>> Mostly curious, don't have experience in such environments,
but ... Is this<br> AFM over NFS or NSD protocol? Might be interesting to try
the other option<br> -- and also check how nsdperf performs over such distance/latency.<br> <br>As it turns out, it seems, very few people do. <br><br>I will test nsdperf over it and see how it performs. And yes, it is AFM
$B"*(B AFM. No NFS involved here!<br><br>-jc<br><br><br> <br> ------------------------------<br> <br> Message: 2<br> Date: Wed, 9 Nov 2016 17:39:05 +0000<br> From: Jake Carroll <jake.carroll@uq.edu.au><br> To: "gpfsug-discuss@spectrumscale.org"<br>
<gpfsug-discuss@spectrumscale.org><br> Subject: [gpfsug-discuss] Tuning AFM for high throughput/high
IO over<br>
_really_ long distances<br> Message-ID: <83652C3D-0802-4CC2-B636-9FAA31EF5AF0@uq.edu.au><br> Content-Type: text/plain; charset="utf-8"<br> <br> Hi.<br> <br> I?ve got an GPFS to GPFS AFM cache/home (IW) relationship
set up over a really long distance. About 180ms of latency between the
two clusters and around 13,000km of optical path. Fortunately for me, I?ve
actually got near theoretical maximum IO over the NIC?s between the clusters
and I?m iPerf?ing at around 8.90 to 9.2Gbit/sec over a 10GbE circuit. All
MTU9000 all the way through.<br> <br> Anyway ? I?m finding my AFM traffic to be dragging its feet
and I don?t really understand why that might be. I?ve verified the links
and transports ability as I said above with iPerf, and CERN?s FDT to near
10Gbit/sec.<br> <br> I also verified the clusters on both sides in terms of disk
IO and they both seem easily capable in IOZone and IOR tests of multiple
GB/sec of throughput.<br> <br> So ? my questions:<br> <br> <br> 1. Are there very specific tunings AFM
needs for high latency/long distance IO?<br> <br> 2. Are there very specific NIC/TCP-stack
tunings (beyond the type of thing we already have in place) that benefits
AFM over really long distances and high latency?<br> <br> 3. We are seeing on the ?cache? side
really lazy/sticky ?ls ?als? in the home mount. It sometimes takes 20 to
30 seconds before the command line will report back with a long listing
of files. Any ideas why it?d take that long to get a response from ?home?.<br> <br> We?ve got our TCP stack setup fairly aggressively, on all
hosts that participate in these two clusters.<br> <br> ethtool -C enp2s0f0 adaptive-rx off<br> ifconfig enp2s0f0 txqueuelen 10000<br> sysctl -w net.core.rmem_max=536870912<br> sysctl -w net.core.wmem_max=536870912<br> sysctl -w net.ipv4.tcp_rmem="4096 87380 268435456"<br> sysctl -w net.ipv4.tcp_wmem="4096 65536 268435456"<br> sysctl -w net.core.netdev_max_backlog=250000<br> sysctl -w net.ipv4.tcp_congestion_control=htcp<br> sysctl -w net.ipv4.tcp_mtu_probing=1<br> <br> I modified a couple of small things on the AFM ?cache? side
to see if it?d make a difference such as:<br> <br> mmchconfig afmNumWriteThreads=4<br> mmchconfig afmNumReadThreads=4<br> <br> But no difference so far.<br> <br> Thoughts would be appreciated. I?ve done this before over
much shorter distances (30Km) and I?ve flattened a 10GbE wire without really
tuning?anything. Are my large in-flight-packets numbers/long-time-to-acknowledgement
semantics going to hurt here? I really thought AFM might be well designed
for exactly this kind of work at long distance *and* high throughput ?
so I must be missing something!<br> <br> -jc<br> <br> <br> <br> -------------- next part --------------<br> An HTML attachment was scrubbed...<br> URL: <</font></tt><a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161109/d4f4d9a7/attachment-0001.html"><tt><font size=2>http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161109/d4f4d9a7/attachment-0001.html</font></tt></a><tt><font size=2>><br> <br> ------------------------------<br> <br> Message: 3<br> Date: Wed, 09 Nov 2016 18:05:21 +0000<br> From: Jan-Frode Myklebust <janfrode@tanso.net><br> To: "gpfsug-discuss@spectrumscale.org"<br>
<gpfsug-discuss@spectrumscale.org><br> Subject: Re: [gpfsug-discuss] Tuning AFM for high throughput/high
IO<br>
over _really_ long distances<br> Message-ID:<br>
<CAHwPathy=4z=jDXN5qa3ys+Z-_7n=tsJh7cZ3ZKzFwQMG34zwg@mail.gmail.com><br> Content-Type: text/plain; charset="utf-8"<br> <br> Mostly curious, don't have experience in such environments,
but ... Is this<br> AFM over NFS or NSD protocol? Might be interesting to try
the other option<br> -- and also check how nsdperf performs over such distance/latency.<br> <br> <br> <br> -jf<br> ons. 9. nov. 2016 kl. 18.39 skrev Jake Carroll <jake.carroll@uq.edu.au>:<br> <br> > Hi.<br> ><br> ><br> ><br> > I?ve got an GPFS to GPFS AFM cache/home (IW) relationship
set up over a<br> > really long distance. About 180ms of latency between
the two clusters and<br> > around 13,000km of optical path. Fortunately for me,
I?ve actually got near<br> > theoretical maximum IO over the NIC?s between the clusters
and I?m<br> > iPerf?ing at around 8.90 to 9.2Gbit/sec over a 10GbE
circuit. All MTU9000<br> > all the way through.<br> ><br> ><br> ><br> > Anyway ? I?m finding my AFM traffic to be dragging its
feet and I don?t<br> > really understand why that might be. I?ve verified the
links and transports<br> > ability as I said above with iPerf, and CERN?s FDT to
near 10Gbit/sec.<br> ><br> ><br> ><br> > I also verified the clusters on both sides in terms
of disk IO and they<br> > both seem easily capable in IOZone and IOR tests of
multiple GB/sec of<br> > throughput.<br> ><br> ><br> ><br> > So ? my questions:<br> ><br> ><br> ><br> > 1. Are there very specific tunings
AFM needs for high latency/long<br> > distance IO?<br> ><br> > 2. Are there very specific NIC/TCP-stack
tunings (beyond the type<br> > of thing we already have in place) that benefits AFM
over really long<br> > distances and high latency?<br> ><br> > 3. We are seeing on the ?cache?
side really lazy/sticky ?ls ?als?<br> > in the home mount. It sometimes takes 20 to 30 seconds
before the command<br> > line will report back with a long listing of files.
Any ideas why it?d take<br> > that long to get a response from ?home?.<br> ><br> ><br> ><br> > We?ve got our TCP stack setup fairly aggressively, on
all hosts that<br> > participate in these two clusters.<br> ><br> ><br> ><br> > ethtool -C enp2s0f0 adaptive-rx off<br> ><br> > ifconfig enp2s0f0 txqueuelen 10000<br> ><br> > sysctl -w net.core.rmem_max=536870912<br> ><br> > sysctl -w net.core.wmem_max=536870912<br> ><br> > sysctl -w net.ipv4.tcp_rmem="4096 87380 268435456"<br> ><br> > sysctl -w net.ipv4.tcp_wmem="4096 65536 268435456"<br> ><br> > sysctl -w net.core.netdev_max_backlog=250000<br> ><br> > sysctl -w net.ipv4.tcp_congestion_control=htcp<br> ><br> > sysctl -w net.ipv4.tcp_mtu_probing=1<br> ><br> ><br> ><br> > I modified a couple of small things on the AFM ?cache?
side to see if it?d<br> > make a difference such as:<br> ><br> ><br> ><br> > mmchconfig afmNumWriteThreads=4<br> ><br> > mmchconfig afmNumReadThreads=4<br> ><br> ><br> ><br> > But no difference so far.<br> ><br> ><br> ><br> > Thoughts would be appreciated. I?ve done this before
over much shorter<br> > distances (30Km) and I?ve flattened a 10GbE wire without
really<br> > tuning?anything. Are my large in-flight-packets<br> > numbers/long-time-to-acknowledgement semantics going
to hurt here? I really<br> > thought AFM might be well designed for exactly this
kind of work at long<br> > distance **and** high throughput ? so I must be missing
something!<br> ><br> ><br> ><br> > -jc<br> ><br> ><br> ><br> ><br> ><br> ><br> > _______________________________________________<br> > gpfsug-discuss mailing list<br> > gpfsug-discuss at spectrumscale.org<br> > </font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br> ><br> -------------- next part --------------<br> An HTML attachment was scrubbed...<br> URL: <</font></tt><a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161109/f44369ab/attachment.html"><tt><font size=2>http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161109/f44369ab/attachment.html</font></tt></a><tt><font size=2>><br> <br> ------------------------------<br> <br> _______________________________________________<br> gpfsug-discuss mailing list<br> gpfsug-discuss at spectrumscale.org<br> </font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br> <br> <br> End of gpfsug-discuss Digest, Vol 58, Issue 12<br> **********************************************<br> <br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br></font></tt><br><br><BR>