<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">I think what you want is to use fabric numbers with verbsPorts, e.g. we have two IB fabrics and in the config we do thinks like:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[nodeclass1]<o:p></o:p></p>
<p class="MsoNormal">verbsPorts mlx4_0/1/1<o:p></o:p></p>
<p class="MsoNormal">[nodeclass2]<o:p></o:p></p>
<p class="MsoNormal">verbsPorts mlx5_0/1/3<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">GPFS recognises the /1 or /3 at the end as a fabric number and knows they are separate and will Ethernet between those nodes instead.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Simon<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black"><gpfsug-discuss-bounces@spectrumscale.org> on behalf of "Stephan.Peinkofer@lrz.de" <Stephan.Peinkofer@lrz.de><br>
<b>Reply-To: </b>"gpfsug-discuss@spectrumscale.org" <gpfsug-discuss@spectrumscale.org><br>
<b>Date: </b>Thursday, 19 July 2018 at 15:13<br>
<b>To: </b>"gpfsug-discuss@spectrumscale.org" <gpfsug-discuss@spectrumscale.org><br>
<b>Subject: </b>[gpfsug-discuss] Mixing RDMA Client Fabrics for a single NSD Cluster<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">Dear GPFS List, <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">does anyone of you know, if it is possible to have multiple file systems in a GPFS Cluster that all are served primary via Ethernet but for which different “booster” connections to various IB/OPA fabrics exist.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">For example let’s say in my central Storage/NSD Cluster, I implement two file systems FS1 and FS2. FS1 is served by NSD-A and NSD-B and FS2 is served by NSD-C and NSD-D.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Now I have two client Clusters C1 and C2 which have different OPA fabrics. Both Clusters can mount the two file systems via Ethernet, but I now add OPA connections for NSD-A and NSD-B to C1’s fabric and OPA connections for NSD-C and NSD-D
to C2’s fabric and just switch on RDMA.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">As far as I understood, GPFS will use RDMA if it is available between two nodes but switch to Ethernet if RDMA is not available between the two nodes. So given just this, the above scenario could work in principle. But will it work in reality
and will it be supported by IBM?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Many thanks in advance.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Best Regards,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Stephan Peinkofer<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Helvetica;color:black">-- <br>
Stephan Peinkofer<br>
Leibniz Supercomputing Centre<br>
Data and Storage Division<br>
Boltzmannstraße 1, 85748 Garching b. München<br>
URL: <a href="http://www.lrz.de">http://www.lrz.de</a><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>