<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Ryan,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">This sounds very interesting.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Do you have more details or references of how they connected together, and what any pain points were?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Daniel<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> gpfsug-discuss <gpfsug-discuss-bounces@gpfsug.org>
<b>On Behalf Of </b>Ryan Novosielski<br>
<b>Sent:</b> 21 August 2023 19:07<br>
<b>To:</b> gpfsug main discussion list <gpfsug-discuss@gpfsug.org><br>
<b>Cc:</b> gpfsug-discuss@spectrumscale.org<br>
<b>Subject:</b> Re: [gpfsug-discuss] Joining RDMA over different networks?<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">If I understand what you’re asking correctly, we used to have a cluster that did this. GPFS was on Infininiband, some of the compute nodes were too, and the rest were on Omnipath. There were routers in between
with both types.<o:p></o:p></p>
<div>
<p class="MsoNormal">Sent from my iPhone<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-bottom:12.0pt">On Aug 21, 2023, at 13:55, Kidger, Daniel <<a href="mailto:daniel.kidger@hpe.com">daniel.kidger@hpe.com</a>> wrote:<o:p></o:p></p>
</blockquote>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">I know in the Lustre world that LNET routers are used to provide RDMA over heterogeneous networks.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Is there an equivalent for Storage Scale?<o:p></o:p></p>
<p class="MsoNormal">eg if an ESS uses Infiniband to connect directly to Cluster A, could that InfiniBand RDMA fabric be “routed” to ClusterB that has RoCE connecting all its nodes together and hence the filesystem mounted?<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">ps. The same question would apply to other usually incompatible RDMA networks like Omnipath, Slingshot, Cornelis, … ?<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Daniel<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt;line-height:12.0pt"><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">Daniel Kidger<br>
</span></b><span style="color:black">HPC Storage Solutions Architect, EMEA</span><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"><br>
</span><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#040400"><a href="mailto:daniel.kidger@hpe.com"><span style="color:#040400">daniel.kidger@hpe.com</span></a><br>
<br>
</span><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">+44 (0)7818 522266 <br>
<br>
</span><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#040400"><a href="http://www.hpe.com/"><b><span style="color:black;text-decoration:none">hpe.com</span></b></a><br>
<br>
</span><o:p></o:p></p>
<div>
<p class="MsoNormal"><image001.png><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at gpfsug.org<br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org">http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org</a><o:p></o:p></p>
</div>
</blockquote>
</div>
</body>
</html>