<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body>
<div dir="auto" style="font-family: Aptos, Aptos_MSFontService, -apple-system, Roboto, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(33, 33, 33);">
Hi</div>
<div dir="auto" style="font-family: Aptos, Aptos_MSFontService, -apple-system, Roboto, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(33, 33, 33);">
<br>
</div>
<div dir="auto" style="font-family: Aptos, Aptos_MSFontService, -apple-system, Roboto, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(33, 33, 33);">
We are</div>
<div dir="auto" style="font-family: Aptos, Aptos_MSFontService, -apple-system, Roboto, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(33, 33, 33);">
<br>
</div>
<div dir="auto" style="font-family: Aptos, Aptos_MSFontService, -apple-system, Roboto, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(33, 33, 33);">
Somethings is odd</div>
<div dir="auto" style="font-family: Aptos, Aptos_MSFontService, -apple-system, Roboto, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(33, 33, 33);">
<br>
</div>
<div dir="auto" style="font-family: Aptos, Aptos_MSFontService, -apple-system, Roboto, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(33, 33, 33);">
We had pretty good results. Latency halved which had a great improvement on throughput</div>
<div dir="auto" style="font-family: Aptos, Aptos_MSFontService, -apple-system, Roboto, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(33, 33, 33);">
<br>
</div>
<div dir="auto" style="font-family: Aptos, Aptos_MSFontService, -apple-system, Roboto, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(33, 33, 33);">
We use two separated fabrics. Configuration was not straight forward as this case is not covered by essgennetworks but worth it </div>
<div dir="auto" style="font-family: Aptos, Aptos_MSFontService, -apple-system, Roboto, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(33, 33, 33);">
<br>
</div>
<div dir="auto" style="font-family: Aptos, Aptos_MSFontService, -apple-system, Roboto, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(33, 33, 33);">
In your case you are down to one fabric as clients have one 25G port </div>
<div dir="auto" style="font-family: Aptos, Aptos_MSFontService, -apple-system, Roboto, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(33, 33, 33);">
<br>
</div>
<div dir="auto" style="font-family: Aptos, Aptos_MSFontService, -apple-system, Roboto, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(33, 33, 33);">
<br>
</div>
<div id="ms-outlook-mobile-body-separator-line" data-applydefaultfontstyles="true" dir="auto" style="font-family: Aptos, Aptos_MSFontService, -apple-system, Roboto, Arial, Helvetica, sans-serif; font-size: 12pt;">
<div style="font-family: Aptos, Aptos_MSFontService, -apple-system, Roboto, Arial, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> gpfsug-discuss <gpfsug-discuss-bounces@gpfsug.org> on behalf of Luke Sudbery <l.r.sudbery@bham.ac.uk><br>
<b>Sent:</b> Friday, January 23, 2026 5:38:43 PM<br>
<b>To:</b> gpfsug main discussion list <gpfsug-discuss@gpfsug.org><br>
<b>Subject:</b> [EXTERNAL] [gpfsug-discuss] Anyone using RoCE?</font>
<div> </div>
</div>
<style>
<!--
@font-face
{font-family:Wingdings}
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Aptos}
@font-face
{font-family:"Cascadia Code"}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0cm;
font-size:12.0pt;
font-family:"Aptos",sans-serif}
p.x_MsoListParagraph, li.x_MsoListParagraph, div.x_MsoListParagraph
{margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
font-size:12.0pt;
font-family:"Aptos",sans-serif}
span.x_EmailStyle19
{font-family:"Aptos",sans-serif;
color:windowtext}
.x_MsoChpDefault
{}
@page WordSection1
{margin:72.0pt 72.0pt 72.0pt 72.0pt}
div.x_WordSection1
{}
ol
{margin-bottom:0cm}
ul
{margin-bottom:0cm}
-->
</style>
<div lang="EN-GB" link="#467886" vlink="#96607D" style="word-wrap:break-word">
<div class="x_WordSection1">
<p class="x_MsoNormal"><span style="font-size:11.0pt">Is anyone using RoCE with good results? We are planning on it, but initial tests are not great we get much better performance using plain Ethernet over the exact same links.</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt"> </span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt">Its up and working, I can see RDMA connections and counters, no errors, but performance is unstable. And worse than Ethernet, which was just meant to be a sanity check!</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt"> </span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt">Things Ive looked at based on Lenovo and IBM guides, which I think are all configured correctly:</span></p>
<ul type="disc" style="margin-top:0cm">
<li class="x_MsoListParagraph" style="margin-left:0cm"><span style="font-size:11.0pt">RoCE interfaces all on the same subnet</span></li><li class="x_MsoListParagraph" style="margin-left:0cm"><span style="font-size:11.0pt">They all have IPv6 enabled with addresses using eui64 addr-gen-mode</span></li><li class="x_MsoListParagraph" style="margin-left:0cm"><span style="font-size:11.0pt">DSCP trust mode on NICs</span></li><li class="x_MsoListParagraph" style="margin-left:0cm"><span style="font-size:11.0pt">PFC flow control on NICs</span></li><li class="x_MsoListParagraph" style="margin-left:0cm"><span style="font-size:11.0pt">Global Pause disabled on NICs</span></li><li class="x_MsoListParagraph" style="margin-left:0cm"><span style="font-size:11.0pt">ToS configured for RDMA_CM</span></li><li class="x_MsoListParagraph" style="margin-left:0cm"><span style="font-size:11.0pt">Source based routing for multiple interfaces on the same subnet.</span></li><li class="x_MsoListParagraph" style="margin-left:0cm"><span style="font-size:11.0pt">Switches (nvidia cumulus) all enabled for RoCE QOS
</span></li></ul>
<p class="x_MsoNormal"><span style="font-size:11.0pt"> </span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt">Iperf and GPFS over plain Ethernet get nearly 3GB/s, which is near the line speed of the NIC in question 25Gbps. Testing basic RDMA connections with ib_send_bw gets about the same. But GPFS over RoCE gets
from 0.7GB/s to 1.9GB/s.</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt"> </span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt">The servers have 4x 200G Mellanox cards. The client has 1x 25G card. Whats frustrating and confusing is that we get better performance when we just enable 1 card at the server end, and also get better performance
if we have 1 fabric ID per NIC on the server (with all 4 fabric ID on the same NIC at the client end).</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt"> </span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt">I can go into more details if anyone has experience! Does this sound familiar to anyone? I am planning to open a call with Lenovo and/or IBM as Im not quite sure where to look next.</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt"> </span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt">Cheers,</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt"> </span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt">Luke</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt"> </span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; color:#1F497D">-- </span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; color:#1F497D">Luke Sudbery</span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; color:#1F497D">Principal Engineer (HPC and Storage).</span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; color:#1F497D">Architecture, Infrastructure and Systems</span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; color:#1F497D">Advanced Research Computing, IT Services</span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; color:#1F497D">Room 132, Computer Centre G5, Elms Road</span></p>
<p class="x_MsoNormal"><span style="font-size:9.0pt; color:#1F497D"> </span></p>
<p class="x_MsoNormal"><b><span style="font-size:9.0pt; color:#1F497D">Please note I dont work on Monday.</span></b></p>
<p class="x_MsoNormal"> </p>
</div>
</div>
<br>
<p class="MsoNormal" style="MARGIN: 0in 0in 10pt"><font size="3"><span style="font-family: Calibri;">###################################################################################<!--?xml:namespace prefix = "o" ns = "urn:schemas-microsoft-com:office:office" /--><o:p></o:p></span></font></p>
<p class="MsoNormal" style="MARGIN: 0in 0in 10pt"><font size="3"><span style="font-family: Calibri;">The information contained in this communication is confidential, may be<o:p></o:p></span></font></p>
<p class="MsoNormal" style="MARGIN: 0in 0in 10pt"><font size="3"><span style="font-family: Calibri;">subject to legal privilege, and is intended only for the individual named.<o:p></o:p></span></font></p>
<p class="MsoNormal" style="MARGIN: 0in 0in 10pt"><font size="3"><span style="font-family: Calibri;">If you are not the named addressee, please notify the sender immediately and<o:p></o:p></span></font></p>
<p class="MsoNormal" style="MARGIN: 0in 0in 10pt"><span style="font-family: Calibri;"><font size="3">delete this email from your system.</font></span><font size="3"><span style="mso-spacerun: yes"><font size="3"> </font></span><font size="3">The views expressed in this email are<o:p></o:p></font></font></p>
<p class="MsoNormal" style="MARGIN: 0in 0in 10pt"><span style="font-family: Calibri;"><font size="3">the views of the sender only.</font></span><font size="3"><span style="mso-spacerun: yes"><font size="3"> </font></span><font size="3">Outgoing and incoming electronic communications<o:p></o:p></font></font></p>
<p class="MsoNormal" style="MARGIN: 0in 0in 10pt"><font size="3"><span style="font-family: Calibri;">to this address are electronically archived and subject to review and/or disclosure<o:p></o:p></span></font></p>
<p class="MsoNormal" style="MARGIN: 0in 0in 10pt"><font size="3"><span style="font-family: Calibri;">to someone other than the recipient.<o:p></o:p></span></font></p><span style="FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; LINE-HEIGHT: 115%; mso-ascii-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin; mso-hansi-theme-font: minor-latin; mso-bidi-font-family: "Times New Roman"; mso-bidi-theme-font: minor-bidi; mso-ansi-language: EN-US; mso-fareast-language: EN-US; mso-bidi-language: AR-SA">###################################################################################<br><br></span></body>
</html>