<html><body><p><font size="2">mmnetverify is a new tool that aims to make it easier to identify network problems.</font><br><br><font size="2">Regarding bandwidth commands, in 4.2.2, there are two options:</font><ul type="disc"><li><font size="2">mmnetverify bandwidth-node - [1 to 1] this will communicate from local node (or one or more nodes specified with -N option) to one or more target nodes. The bandwidth tests are executed serially from nodes in node list to target, iterating through each target node, one by one. The serially calculated bandwidth with each node is reported.</font><li><font size="2">mmnetverify bandwidth-cluster - [1 to many] this is a measure of parallel communication from the local node (or one or more nodes specified with -N option) to all of the other nodes in the cluster. The concurrent bandwidth with each target node in the cluster is reported.</font></ul><br><font size="2">In both of these tests, we establish a socket connection, and pass a fixed number of bytes over the connection and calculate bandwidth based on how long that transmission took.</font><br><br><font size="2">For 4.2.3, there is a new bandwidth test called gnr-bandwidth. It is similar to the bandwidth-cluster [1 to many] except that it uses the following steps:</font><br><font size="2">1. establish connection from node to all other target nodes in the cluster</font><br><font size="2">2. start sending data to target for some ramp up period</font><br><font size="2">3. after ramp up period, continue sending data for test period</font><br><font size="2">4. calculate bandwidth based on bytes transmitted during test period</font><br><br><font size="2">The bandwidth to each node is summed to return a total bandwidth from the command node to the other nodes in the cluster.</font><br><br><font size="2">In future releases, we may modify bandwidth-node & bandwidth-cluster tests to use the gnr-bandwidth methodology (and deprecate gnr-bandwidth). Your feedback on how to improve mmnetverify is appreciated.</font><br><br><font size="2">Regarding:</font><br><font size="2">> </font><tt><font size="2">We found some weird looking numbers that i don't quite understand and not in the places we might expect. </font></tt><br><tt><font size="2">> For example between hosts on the same switch, traffic flowing to another switch and traffic flowing to </font></tt><br><tt><font size="2">> nodes in another data centre where it's several switch hops. Some nodes over there were significantly </font></tt><br><tt><font size="2">> faster than switch local nodes.</font></tt><font size="2"><br>Note that system load can impact the test results. Is it possible that the slow nodes on the local switch were heavily loaded? Or is it possible they are using an interface that is lower bandwidth? (sorry, i had to ask that one to be sure...)</font><br><br><font size="2">Regards,<br>Bill Owen <br>billowen@us.ibm.com<br>Spectrum Scale Development<br>520-799-4829<br></font><br><br><img width="16" height="16" src="cid:1__=88BB0A67DFF744108f9e8a93df938690918c88B@" border="0" alt="Inactive hide details for "Simon Thompson (Research Computing - IT Services)" ---03/17/2017 01:13:50 PM---It looks to run seque"><font size="2" color="#424282">"Simon Thompson (Research Computing - IT Services)" ---03/17/2017 01:13:50 PM---It looks to run sequential tests to each node one at a time and isn't using NSD protocol but echo se</font><br><br><font size="2" color="#5F5F5F">From: </font><font size="2">"Simon Thompson (Research Computing - IT Services)" <S.J.Thompson@bham.ac.uk></font><br><font size="2" color="#5F5F5F">To: </font><font size="2">gpfsug main discussion list <gpfsug-discuss@spectrumscale.org></font><br><font size="2" color="#5F5F5F">Date: </font><font size="2">03/17/2017 01:13 PM</font><br><font size="2" color="#5F5F5F">Subject: </font><font size="2">Re: [gpfsug-discuss] mmnetverify</font><br><font size="2" color="#5F5F5F">Sent by: </font><font size="2">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br><br><br><tt><font size="2">It looks to run sequential tests to each node one at a time and isn't using NSD protocol but echo server.<br><br>We found some weird looking numbers that i don't quite understand and not in the places we might expect. For example between hosts on the same switch, traffic flowing to another switch and traffic flowing to nodes in another data centre where it's several switch hops. Some nodes over there were significantly faster than switch local nodes.<br><br>I think it was only added in 4.2.2 and is listed as "not yet a replacement for nsdperf". I get that is different as it's using NSD protocol, but was struggling a bit with what mmnetverify might be doing.<br><br>Simon<br><br>From: gpfsug-discuss-bounces@spectrumscale.org [gpfsug-discuss-bounces@spectrumscale.org] on behalf of Sanchez, Paul [Paul.Sanchez@deshaw.com]<br>Sent: 17 March 2017 19:43<br>To: gpfsug-discuss@spectrumscale.org<br>Subject: Re: [gpfsug-discuss] mmnetverify<br><br>Sven will tell you: "RPC isn't streaming" and that may account for the discrepancy. If the tests are doing any "fan-in" where multiple nodes are sending to single node, then it's also possible that you are exhausting switch buffer memory in a way that a 1:1 iperf wouldn't.<br><br>For our internal benchmarking we've used /usr/lpp/mmfs/samples/net/nsdperf to more closely estimate the real performance. I haven't played with mmnetverify yet though.<br><br>-Paul<br><br>-----Original Message-----<br>From: gpfsug-discuss-bounces@spectrumscale.org [</font></tt><tt><font size="2"><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">mailto:gpfsug-discuss-bounces@spectrumscale.org</a></font></tt><tt><font size="2">] On Behalf Of Simon Thompson (Research Computing - IT Services)<br>Sent: Friday, March 17, 2017 2:50 PM<br>To: gpfsug-discuss@spectrumscale.org<br>Subject: [gpfsug-discuss] mmnetverify<br><br>Hi all,<br><br>Just wondering if anyone has used the mmnetverify tool at all?<br><br>Having made some changes to our internal L3 routing this week, I was interested to see what it claimed.<br><br>As a side-note, it picked up some DNS resolution issues, though I'm not clear on some of those why it was claiming this as doing a "dig" on the node, it resolved fine (but adding the NSD servers to the hosts files cleared the error).<br><br>Its actually the bandwidth tests that I'm interested in hearing other people's experience with as the numbers that some out from it are very different (lower) than if we use iperf to test performance between two nodes.<br><br>Anyone any thoughts at all on this?<br><br>Thanks<br>Simon<br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><tt><font size="2"><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></tt><tt><font size="2"><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><tt><font size="2"><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></tt><tt><font size="2"><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><tt><font size="2"><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></tt><tt><font size="2"><br><br></font></tt><br><br><BR>
</body></html>