<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hello All,<div class=""><br class=""></div><div class="">I would like to discuss or understand on which ethernet networking switches/architecture seems to work best with GPFS. </div><div class="">We had thought about infiniband, but are not yet ready to move to infiniband because of the complexity/upgrade and debugging issues that come with it. </div><div class=""><br class=""></div><div class=""><div class=""><u class="">Current hardware:</u></div><div class=""><br class=""></div><div class="">We are currently using Arista 7328x 100G core switch for networking among the GPFS clusters and the compute nodes.</div></div><div class=""><br class=""></div><div class="">It is heterogeneous network, with some of the servers on 10G/25G/100G with LACP and without LACP.</div><div class=""><br class=""></div><div class="">For example: </div><div class=""><br class=""></div><div class="">GPFS storage clusters either have 25G LACP, or 10G LACP, or a single 100G network port.</div><div class="">Compute nodes range from 10G to 100G.</div><div class="">Login nodes/transfer servers etc have 25G bonded.</div><div class=""><br class=""></div><div class="">Most of the servers have Mellanox ConnectX-4 or ConnectX-5 adapters. But we also have few older Intel,Broadcom and Chelsio network cards in the clusters.</div><div class=""><br class=""></div><div class="">Most of the transceivers that we use are Mellanox,Finisar,Intel.</div><div class=""><br class=""></div><div class=""><u class="">Issue</u>:</div><div class=""><br class=""></div><div class="">We had upgraded to the above switch recently, and we had seen that it is not able to handle the network traffic because of higher NSD servers bandwidth vs lower compute node bandwidth.</div><div class=""><br class=""></div><div class="">One issue that we did see was a lot of network discards on the switch side and network congestion with slow IO performance on respective compute nodes.</div><div class=""><br class=""></div><div class="">Once we enabled ECN - we did see that it had reduced the network congestion.</div><div class=""><br class=""></div><div class="">We do see expels once in a while, but that is mostly related to the network errors or the host not responding. We observed that bonding/LACP does make expels much more trickier, so we have decided to go with no LACP until GPFS code gets better at handling LACP - which I think they are working on.</div><div class=""><br class=""></div><div class="">We have heard that our current switch is a shallow buffer switch, and we would need a higher/deep buffer Arista switch to perform better with no congestion/lesser latency and more throughput.</div><div class=""><br class=""></div><div class="">On the other side, Mellanox promises to use better ASIC design and buffer architecture with spine leaf design, instead of one deep buffer core switch to get better performance than Arista.</div><div class=""><br class=""></div><div class=""><div class="">Most of the applications that run on the clusters are either genomic applications on CPUs and deep learning applications on GPUs. </div></div><div class=""><br class=""></div><div class="">All of our GPFS storage cluster versions are above 5.0.2 with the compute filesystems at 16M block size on near line rotating disks, and Flash storage at 512K block size.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">May I know if could feedback from anyone who is using Arista or Mellanox switches on the clusters to understand the pros and cons, stability and the performance numbers of the same?</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Thank you,</div><div class="">Lohit</div></body></html>