<html><body><p>Hi Jake,<br><br>I would tackle this programmatically:<br><br>a) Mount the remote FS directly from a GPFS client (using multi-cluster) and evaluate the performance using GPFS directly. The key factors affecting performance here will be<br>- number of nsd servers at remote site (home site): The client creates a new TCP connection to each NSD server. The TCP connections will slowly expand their send/receive window as more data is read/written. The more TCP connections the better since it allows multiple windows to better fill the pipe. Note that TCP windows close quickly when no data is sent..and must resume tcp slow start on each benchmark run.<br>- size of tcp buffers that gpfs is using: You want to be able to have the sum of the TCP windows fill the pipe. <br>- type of workload you are running: Small files occur the full latency, heavy metadata occur the full latency, but large files allow tcp slow start to expand the tcp window to the full size and fill the pipe.<br><br>b) One this is done then move to AFM. Note that with writes the only way to evaluate performance is to monitor the network B/W on the GW. With reads, note that the GW writes data to storage before clients read it off the local disk. So you can either monitor read network B/W on the GW, or run your read tests directly on the GW (since then data is delivered to the application benchmark directly from the pagepool).<br><br>Also note that AFM writes data in at most (by default) 1GB chunks. This can be increased with afmMaxWriteMergeLen, but be careful since if the network fails in the middle of the write to the home, it may need to restart from the beginning when the network connection is fixed.<br><br>c) Using multiple GWs with AFM with parallel I/O. This allows more available nodes the opportunity to create more TCP connections, all giving a better chance to fill the pipe. The additional TCP connections also mitigates the impact of packet loss or delay since it will only affect one of the connections. Playing with chunk size here can be useful. <br><br>Dean <br><br><br><img width="16" height="16" src="cid:1__=07BB0AF5DFFE22418f9e8a93df938690918c07B@" border="0" alt="Inactive hide details for Jake Carroll ---11/09/2016 10:28:16 AM---Scott, Nar, very much pure AFM to AFM here, hence we are a l"><font color="#424282">Jake Carroll ---11/09/2016 10:28:16 AM---Scott, Nar, very much pure AFM to AFM here, hence we are a little surprised. Last time we did this o</font><br><br><font size="2" color="#5F5F5F">From: </font><font size="2">Jake Carroll <jake.carroll@uq.edu.au></font><br><font size="2" color="#5F5F5F">To: </font><font size="2">"gpfsug-discuss@spectrumscale.org" <gpfsug-discuss@spectrumscale.org></font><br><font size="2" color="#5F5F5F">Date: </font><font size="2">11/09/2016 10:28 AM</font><br><font size="2" color="#5F5F5F">Subject: </font><font size="2">[gpfsug-discuss] Tuning AFM for high throughput/high IO over _really_ long</font><br><font size="2" color="#5F5F5F">Sent by: </font><font size="2">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br><br><br><tt>Scott,<br><br>Nar, very much pure AFM to AFM here, hence we are a little surprised. Last time we did this over a longish link we almost caused an outage with the ease at which we attained throughput - but maybe there are some magic tolerances we are hitting in latency and in flight IO semantics that SS/GPFS/AFM is not well tweaked for (yet...)...<br><br>Yes - we are catching up at SC. I think it's all been arranged? We are also talking to one of your resources about this AFM throughput behaviour this afternoon. John I believe his name is?<br><br>Anyway - if you've got any ideas, am all ears!<br>> <br>> <br>> Today's Topics:<br>> <br>> 1. Re: Tuning AFM for high throughput/high IO over _really_ long<br>> distances (Scott Fadden)<br>> 2. Re: Tuning AFM for high throughput/high IO over _really_ long<br>> distances (Jan-Frode Myklebust) (Jake Carroll)<br>> <br>> <br>> ----------------------------------------------------------------------<br>> <br>> Message: 1<br>> Date: Wed, 9 Nov 2016 10:08:42 -0800<br>> From: "Scott Fadden" <sfadden@us.ibm.com><br>> To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>> Subject: Re: [gpfsug-discuss] Tuning AFM for high throughput/high IO<br>> over _really_ long distances<br>> Message-ID:<br>> <OF438B4E0A.3DF65F65-ON88258066.006179EA-88258066.0063ABE4@notes.na.collabserv.com><br>> <br>> Content-Type: text/plain; charset="utf-8"<br>> <br>> Jake,<br>> <br>> If AFM is using NFS it is all about NFS tuning. The copy from one side to <br>> the other is basically just a client writing to an NFS mount. Thee are a <br>> few things you can look at:<br>> 1. NFS Transfer size (Make is 1MiB, I think that is the max)<br>> 2. TCP Tuning for large window size. This is discussed on Tuning active <br>> file management home communications in the docs. On this page you will <br>> find some discussion on increasing gateway threads, and other things <br>> similar that may help as well.<br>> <br>> We can discuss further as I understand we will be meeting at SC16.<br>> <br>> Scott Fadden<br>> Spectrum Scale - Technical Marketing <br>> Phone: (503) 880-5833 <br>> sfadden@us.ibm.com<br>> </tt><tt><a href="http://www.ibm.com/systems/storage/spectrum/scale">http://www.ibm.com/systems/storage/spectrum/scale</a></tt><tt><br>> <br>> <br>> <br>> From: Jake Carroll <jake.carroll@uq.edu.au><br>> To: "gpfsug-discuss@spectrumscale.org" <br>> <gpfsug-discuss@spectrumscale.org><br>> Date: 11/09/2016 09:39 AM<br>> Subject: [gpfsug-discuss] Tuning AFM for high throughput/high IO <br>> over _really_ long distances<br>> Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>> <br>> <br>> <br>> Hi.<br>> <br>> I?ve got an GPFS to GPFS AFM cache/home (IW) relationship set up over a <br>> really long distance. About 180ms of latency between the two clusters and <br>> around 13,000km of optical path. Fortunately for me, I?ve actually got <br>> near theoretical maximum IO over the NIC?s between the clusters and I?m <br>> iPerf?ing at around 8.90 to 9.2Gbit/sec over a 10GbE circuit. All MTU9000 <br>> all the way through.<br>> <br>> Anyway ? I?m finding my AFM traffic to be dragging its feet and I don?t <br>> really understand why that might be. I?ve verified the links and <br>> transports ability as I said above with iPerf, and CERN?s FDT to near <br>> 10Gbit/sec. <br>> <br>> I also verified the clusters on both sides in terms of disk IO and they <br>> both seem easily capable in IOZone and IOR tests of multiple GB/sec of <br>> throughput.<br>> <br>> So ? my questions:<br>> <br>> 1. Are there very specific tunings AFM needs for high latency/long <br>> distance IO? <br>> 2. Are there very specific NIC/TCP-stack tunings (beyond the type of <br>> thing we already have in place) that benefits AFM over really long <br>> distances and high latency?<br>> 3. We are seeing on the ?cache? side really lazy/sticky ?ls ?als? in <br>> the home mount. It sometimes takes 20 to 30 seconds before the command <br>> line will report back with a long listing of files. Any ideas why it?d <br>> take that long to get a response from ?home?.<br>> <br>> We?ve got our TCP stack setup fairly aggressively, on all hosts that <br>> participate in these two clusters.<br>> <br>> ethtool -C enp2s0f0 adaptive-rx off<br>> ifconfig enp2s0f0 txqueuelen 10000<br>> sysctl -w net.core.rmem_max=536870912<br>> sysctl -w net.core.wmem_max=536870912<br>> sysctl -w net.ipv4.tcp_rmem="4096 87380 268435456"<br>> sysctl -w net.ipv4.tcp_wmem="4096 65536 268435456"<br>> sysctl -w net.core.netdev_max_backlog=250000<br>> sysctl -w net.ipv4.tcp_congestion_control=htcp<br>> sysctl -w net.ipv4.tcp_mtu_probing=1<br>> <br>> I modified a couple of small things on the AFM ?cache? side to see if it?d <br>> make a difference such as:<br>> <br>> mmchconfig afmNumWriteThreads=4<br>> mmchconfig afmNumReadThreads=4<br>> <br>> But no difference so far.<br>> <br>> Thoughts would be appreciated. I?ve done this before over much shorter <br>> distances (30Km) and I?ve flattened a 10GbE wire without really <br>> tuning?anything. Are my large in-flight-packets <br>> numbers/long-time-to-acknowledgement semantics going to hurt here? I <br>> really thought AFM might be well designed for exactly this kind of work at <br>> long distance *and* high throughput ? so I must be missing something!<br>> <br>> -jc<br>> <br>> <br>> _______________________________________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at spectrumscale.org<br>> </tt><tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></tt><tt><br>> <br>> <br>> <br>> <br>> -------------- next part --------------<br>> An HTML attachment was scrubbed...<br>> URL: <</tt><tt><a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161109/c775cf5a/attachment-0001.html">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161109/c775cf5a/attachment-0001.html</a></tt><tt>><br>> <br>> ------------------------------<br>> <br>> Message: 2<br>> Date: Wed, 9 Nov 2016 18:09:14 +0000<br>> From: Jake Carroll <jake.carroll@uq.edu.au><br>> To: "gpfsug-discuss@spectrumscale.org"<br>> <gpfsug-discuss@spectrumscale.org><br>> Subject: Re: [gpfsug-discuss] Tuning AFM for high throughput/high IO<br>> over _really_ long distances (Jan-Frode Myklebust)<br>> Message-ID: <5D327C63-84EC-4F59-86E7-158308E91013@uq.edu.au><br>> Content-Type: text/plain; charset="utf-8"<br>> <br>> Hi jf?<br>> <br>> <br>>>> Mostly curious, don't have experience in such environments, but ... Is this<br>> AFM over NFS or NSD protocol? Might be interesting to try the other option<br>> -- and also check how nsdperf performs over such distance/latency.<br>> <br>> As it turns out, it seems, very few people do. <br>> <br>> I will test nsdperf over it and see how it performs. And yes, it is AFM ? AFM. No NFS involved here!<br>> <br>> -jc<br>> <br>> <br>> <br>> ------------------------------<br>> <br>> Message: 2<br>> Date: Wed, 9 Nov 2016 17:39:05 +0000<br>> From: Jake Carroll <jake.carroll@uq.edu.au><br>> To: "gpfsug-discuss@spectrumscale.org"<br>> <gpfsug-discuss@spectrumscale.org><br>> Subject: [gpfsug-discuss] Tuning AFM for high throughput/high IO over<br>> _really_ long distances<br>> Message-ID: <83652C3D-0802-4CC2-B636-9FAA31EF5AF0@uq.edu.au><br>> Content-Type: text/plain; charset="utf-8"<br>> <br>> Hi.<br>> <br>> I?ve got an GPFS to GPFS AFM cache/home (IW) relationship set up over a really long distance. About 180ms of latency between the two clusters and around 13,000km of optical path. Fortunately for me, I?ve actually got near theoretical maximum IO over the NIC?s between the clusters and I?m iPerf?ing at around 8.90 to 9.2Gbit/sec over a 10GbE circuit. All MTU9000 all the way through.<br>> <br>> Anyway ? I?m finding my AFM traffic to be dragging its feet and I don?t really understand why that might be. I?ve verified the links and transports ability as I said above with iPerf, and CERN?s FDT to near 10Gbit/sec.<br>> <br>> I also verified the clusters on both sides in terms of disk IO and they both seem easily capable in IOZone and IOR tests of multiple GB/sec of throughput.<br>> <br>> So ? my questions:<br>> <br>> <br>> 1. Are there very specific tunings AFM needs for high latency/long distance IO?<br>> <br>> 2. Are there very specific NIC/TCP-stack tunings (beyond the type of thing we already have in place) that benefits AFM over really long distances and high latency?<br>> <br>> 3. We are seeing on the ?cache? side really lazy/sticky ?ls ?als? in the home mount. It sometimes takes 20 to 30 seconds before the command line will report back with a long listing of files. Any ideas why it?d take that long to get a response from ?home?.<br>> <br>> We?ve got our TCP stack setup fairly aggressively, on all hosts that participate in these two clusters.<br>> <br>> ethtool -C enp2s0f0 adaptive-rx off<br>> ifconfig enp2s0f0 txqueuelen 10000<br>> sysctl -w net.core.rmem_max=536870912<br>> sysctl -w net.core.wmem_max=536870912<br>> sysctl -w net.ipv4.tcp_rmem="4096 87380 268435456"<br>> sysctl -w net.ipv4.tcp_wmem="4096 65536 268435456"<br>> sysctl -w net.core.netdev_max_backlog=250000<br>> sysctl -w net.ipv4.tcp_congestion_control=htcp<br>> sysctl -w net.ipv4.tcp_mtu_probing=1<br>> <br>> I modified a couple of small things on the AFM ?cache? side to see if it?d make a difference such as:<br>> <br>> mmchconfig afmNumWriteThreads=4<br>> mmchconfig afmNumReadThreads=4<br>> <br>> But no difference so far.<br>> <br>> Thoughts would be appreciated. I?ve done this before over much shorter distances (30Km) and I?ve flattened a 10GbE wire without really tuning?anything. Are my large in-flight-packets numbers/long-time-to-acknowledgement semantics going to hurt here? I really thought AFM might be well designed for exactly this kind of work at long distance *and* high throughput ? so I must be missing something!<br>> <br>> -jc<br>> <br>> <br>> <br>> -------------- next part --------------<br>> An HTML attachment was scrubbed...<br>> URL: <</tt><tt><a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161109/d4f4d9a7/attachment-0001.html">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161109/d4f4d9a7/attachment-0001.html</a></tt><tt>><br>> <br>> ------------------------------<br>> <br>> Message: 3<br>> Date: Wed, 09 Nov 2016 18:05:21 +0000<br>> From: Jan-Frode Myklebust <janfrode@tanso.net><br>> To: "gpfsug-discuss@spectrumscale.org"<br>> <gpfsug-discuss@spectrumscale.org><br>> Subject: Re: [gpfsug-discuss] Tuning AFM for high throughput/high IO<br>> over _really_ long distances<br>> Message-ID:<br>> <CAHwPathy=4z=jDXN5qa3ys+Z-_7n=tsJh7cZ3ZKzFwQMG34zwg@mail.gmail.com><br>> Content-Type: text/plain; charset="utf-8"<br>> <br>> Mostly curious, don't have experience in such environments, but ... Is this<br>> AFM over NFS or NSD protocol? Might be interesting to try the other option<br>> -- and also check how nsdperf performs over such distance/latency.<br>> <br>> <br>> <br>> -jf<br>>> ons. 9. nov. 2016 kl. 18.39 skrev Jake Carroll <jake.carroll@uq.edu.au>:<br>>> <br>>> Hi.<br>>> <br>>> <br>>> <br>>> I?ve got an GPFS to GPFS AFM cache/home (IW) relationship set up over a<br>>> really long distance. About 180ms of latency between the two clusters and<br>>> around 13,000km of optical path. Fortunately for me, I?ve actually got near<br>>> theoretical maximum IO over the NIC?s between the clusters and I?m<br>>> iPerf?ing at around 8.90 to 9.2Gbit/sec over a 10GbE circuit. All MTU9000<br>>> all the way through.<br>>> <br>>> <br>>> <br>>> Anyway ? I?m finding my AFM traffic to be dragging its feet and I don?t<br>>> really understand why that might be. I?ve verified the links and transports<br>>> ability as I said above with iPerf, and CERN?s FDT to near 10Gbit/sec.<br>>> <br>>> <br>>> <br>>> I also verified the clusters on both sides in terms of disk IO and they<br>>> both seem easily capable in IOZone and IOR tests of multiple GB/sec of<br>>> throughput.<br>>> <br>>> <br>>> <br>>> So ? my questions:<br>>> <br>>> <br>>> <br>>> 1. Are there very specific tunings AFM needs for high latency/long<br>>> distance IO?<br>>> <br>>> 2. Are there very specific NIC/TCP-stack tunings (beyond the type<br>>> of thing we already have in place) that benefits AFM over really long<br>>> distances and high latency?<br>>> <br>>> 3. We are seeing on the ?cache? side really lazy/sticky ?ls ?als?<br>>> in the home mount. It sometimes takes 20 to 30 seconds before the command<br>>> line will report back with a long listing of files. Any ideas why it?d take<br>>> that long to get a response from ?home?.<br>>> <br>>> <br>>> <br>>> We?ve got our TCP stack setup fairly aggressively, on all hosts that<br>>> participate in these two clusters.<br>>> <br>>> <br>>> <br>>> ethtool -C enp2s0f0 adaptive-rx off<br>>> <br>>> ifconfig enp2s0f0 txqueuelen 10000<br>>> <br>>> sysctl -w net.core.rmem_max=536870912<br>>> <br>>> sysctl -w net.core.wmem_max=536870912<br>>> <br>>> sysctl -w net.ipv4.tcp_rmem="4096 87380 268435456"<br>>> <br>>> sysctl -w net.ipv4.tcp_wmem="4096 65536 268435456"<br>>> <br>>> sysctl -w net.core.netdev_max_backlog=250000<br>>> <br>>> sysctl -w net.ipv4.tcp_congestion_control=htcp<br>>> <br>>> sysctl -w net.ipv4.tcp_mtu_probing=1<br>>> <br>>> <br>>> <br>>> I modified a couple of small things on the AFM ?cache? side to see if it?d<br>>> make a difference such as:<br>>> <br>>> <br>>> <br>>> mmchconfig afmNumWriteThreads=4<br>>> <br>>> mmchconfig afmNumReadThreads=4<br>>> <br>>> <br>>> <br>>> But no difference so far.<br>>> <br>>> <br>>> <br>>> Thoughts would be appreciated. I?ve done this before over much shorter<br>>> distances (30Km) and I?ve flattened a 10GbE wire without really<br>>> tuning?anything. Are my large in-flight-packets<br>>> numbers/long-time-to-acknowledgement semantics going to hurt here? I really<br>>> thought AFM might be well designed for exactly this kind of work at long<br>>> distance **and** high throughput ? so I must be missing something!<br>>> <br>>> <br>>> <br>>> -jc<br>>> <br>>> <br>>> <br>>> <br>>> <br>>> <br>>> _______________________________________________<br>>> gpfsug-discuss mailing list<br>>> gpfsug-discuss at spectrumscale.org<br>>> </tt><tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></tt><tt><br>>> <br>> -------------- next part --------------<br>> An HTML attachment was scrubbed...<br>> URL: <</tt><tt><a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161109/f44369ab/attachment.html">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161109/f44369ab/attachment.html</a></tt><tt>><br>> <br>> ------------------------------<br>> <br>> _______________________________________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at spectrumscale.org<br>> </tt><tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></tt><tt><br>> <br>> <br>> End of gpfsug-discuss Digest, Vol 58, Issue 12<br>> **********************************************<br>> <br>> <br>> <br>> ------------------------------<br>> <br>> _______________________________________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at spectrumscale.org<br>> </tt><tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></tt><tt><br>> <br>> <br>> End of gpfsug-discuss Digest, Vol 58, Issue 13<br>> **********************************************<br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></tt><tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></tt><tt><br><br></tt><br><br><BR>
</body></html>