<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="DE-AT" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoPlainText">Hi Giovanni!<o:p></o:p></p>
<p class="MsoPlainText"><span lang="EN-US">Thank you very much for your offer , we really would be very grateful to be allowed to come if we run into troubles!
<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">Well, the implementation will not happen before June or later, but may I ask only one question meanwhile?<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">Did you ever run into problems with IBM support or did you get a special “OK” from them? Or do you accept to sove any rdma specific problems without support ? (it´s only because of the FAQ “not supported” )
<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">Have a great day and keep healthy!<o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US">Best regards walter <o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US" style="mso-fareast-language:DE-AT">-----Original Message-----<br>
From: Giovanni Bracco <giovanni.bracco@enea.it> <br>
Sent: Montag, 1. Februar 2021 20:42<br>
To: Walter Sklenka <Walter.Sklenka@EDV-Design.at><br>
Cc: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled</span></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">On 30/01/21 21:01, Walter Sklenka wrote:<o:p></o:p></p>
<p class="MsoPlainText">> Hi Giovanni!<o:p></o:p></p>
<p class="MsoPlainText">> Thats great! Many thanks for your fast and detailed answer!!!!<o:p></o:p></p>
<p class="MsoPlainText">> So this is the way we will go too!<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Have a nice weekend and keep healthy!<o:p></o:p></p>
<p class="MsoPlainText">> Best regards<o:p></o:p></p>
<p class="MsoPlainText">> Walter<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">I suppose you will implement the solution with more recent versions of the software components, so please let me know if everything works!<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">If yu have any issues I am ready to discuss!<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Regards<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Giovanni<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">> -----Original Message-----<o:p></o:p></p>
<p class="MsoPlainText">> From: Giovanni Bracco <<a href="mailto:giovanni.bracco@enea.it"><span style="color:windowtext;text-decoration:none">giovanni.bracco@enea.it</span></a>><o:p></o:p></p>
<p class="MsoPlainText">> Sent: Samstag, 30. Jänner 2021 18:08<o:p></o:p></p>
<p class="MsoPlainText">> To: gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org"><span style="color:windowtext;text-decoration:none">gpfsug-discuss@spectrumscale.org</span></a>>;
<o:p></o:p></p>
<p class="MsoPlainText">> Walter Sklenka <<a href="mailto:Walter.Sklenka@EDV-Design.at"><span style="color:windowtext;text-decoration:none">Walter.Sklenka@EDV-Design.at</span></a>><o:p></o:p></p>
<p class="MsoPlainText">> Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD
<o:p></o:p></p>
<p class="MsoPlainText">> Server with only ib rdma enabled<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> In our HPC infrastructure we have 6 NSD server, running CentOS 7.4, each of them with with 1 Intel QDR HCA to a QDR Cluster (now 100 nodes SandyBridge cpu it was 300 nodes CentOS 6.5), 1 OPA HCA to the main OPA Cluster (400 nodes Skylake
cpu, CentOS 7.3) and 1 Mellanox FDR to DDN storages and it works nicely using RDMA since 2018. GPFS 4.2.3-19.<o:p></o:p></p>
<p class="MsoPlainText">> See<o:p></o:p></p>
<p class="MsoPlainText">> F. Iannone et al., "CRESCO ENEA HPC clusters: a working example of a
<o:p></o:p></p>
<p class="MsoPlainText">> multifabric GPFS Spectrum Scale layout," 2019 International Conference
<o:p></o:p></p>
<p class="MsoPlainText">> on High Performance Computing & Simulation (HPCS), Dublin, Ireland,
<o:p></o:p></p>
<p class="MsoPlainText">> 2019, pp. 1051-1052, doi: 10.1109/HPCS48598.2019.918813<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> When setting up the system the main trick has been:<o:p></o:p></p>
<p class="MsoPlainText">> just use CentOS drivers and do not install OFED We do not use IPoIB.<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> Giovanni<o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> On 30/01/21 06:45, Walter Sklenka wrote:<o:p></o:p></p>
<p class="MsoPlainText">>> Hi!<o:p></o:p></p>
<p class="MsoPlainText">>><o:p> </o:p></p>
<p class="MsoPlainText">>> Is it possible to mix OPAcards and Infininiband HCAs on the same server?<o:p></o:p></p>
<p class="MsoPlainText">>><o:p> </o:p></p>
<p class="MsoPlainText">>> In the faq<o:p></o:p></p>
<p class="MsoPlainText">>> <a href="https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq">
<span style="color:windowtext;text-decoration:none">https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq</span></a>.<o:p></o:p></p>
<p class="MsoPlainText">>> html#rdma<o:p></o:p></p>
<p class="MsoPlainText">>><o:p> </o:p></p>
<p class="MsoPlainText">>><o:p> </o:p></p>
<p class="MsoPlainText">>> They talk about RDMA :<o:p></o:p></p>
<p class="MsoPlainText">>><o:p> </o:p></p>
<p class="MsoPlainText">>> "RDMA is NOT supported on a node when both Mellanox HCAs and Intel
<o:p></o:p></p>
<p class="MsoPlainText">>> Omni-Path HFIs are ENABLED for RDMA."<o:p></o:p></p>
<p class="MsoPlainText">>><o:p> </o:p></p>
<p class="MsoPlainText">>> So do I understand right: When we do NOT enable the opa interface we
<o:p></o:p></p>
<p class="MsoPlainText">>> can still enable IB ?<o:p></o:p></p>
<p class="MsoPlainText">>><o:p> </o:p></p>
<p class="MsoPlainText">>> The reason I ask is, that we have a gpfs cluster of 6 NSD Servers
<o:p></o:p></p>
<p class="MsoPlainText">>> (wih access to storage) with opa interfaces which provide access to
<o:p></o:p></p>
<p class="MsoPlainText">>> remote cluster also via OPA.<o:p></o:p></p>
<p class="MsoPlainText">>><o:p> </o:p></p>
<p class="MsoPlainText">>> A new cluster with HDR interfaces will be implemented soon<o:p></o:p></p>
<p class="MsoPlainText">>><o:p> </o:p></p>
<p class="MsoPlainText">>> They shell have access to the same filesystems<o:p></o:p></p>
<p class="MsoPlainText">>><o:p> </o:p></p>
<p class="MsoPlainText">>> When we add HDR interfaces to NSD servers and enable rdma on this
<o:p></o:p></p>
<p class="MsoPlainText">>> network while disabling rdma on opa we would accept the worse
<o:p></o:p></p>
<p class="MsoPlainText">>> performance via opa . We hope that this provides still better perf
<o:p></o:p></p>
<p class="MsoPlainText">>> and less technical overhead than using routers<o:p></o:p></p>
<p class="MsoPlainText">>><o:p> </o:p></p>
<p class="MsoPlainText">>> Or am I totally wrong?<o:p></o:p></p>
<p class="MsoPlainText">>><o:p> </o:p></p>
<p class="MsoPlainText">>> Thank you very much and keep healthy!<o:p></o:p></p>
<p class="MsoPlainText">>><o:p> </o:p></p>
<p class="MsoPlainText">>> Best regards<o:p></o:p></p>
<p class="MsoPlainText">>><o:p> </o:p></p>
<p class="MsoPlainText">>> Walter<o:p></o:p></p>
<p class="MsoPlainText">>><o:p> </o:p></p>
<p class="MsoPlainText">>> Mit freundlichen Grüßen<o:p></o:p></p>
<p class="MsoPlainText">>> */Walter Sklenka/*<o:p></o:p></p>
<p class="MsoPlainText">>> */Technical Consultant/*<o:p></o:p></p>
<p class="MsoPlainText">>><o:p> </o:p></p>
<p class="MsoPlainText">>> EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210
<o:p></o:p></p>
<p class="MsoPlainText">>> Wien<o:p></o:p></p>
<p class="MsoPlainText">>> Tel: +43 1 29 22 165-31<o:p></o:p></p>
<p class="MsoPlainText">>> Fax: +43 1 29 22 165-90<o:p></o:p></p>
<p class="MsoPlainText">>> E-Mail: <a href="mailto:sklenka@edv-design.at"><span style="color:windowtext;text-decoration:none">sklenka@edv-design.at</span></a> <<a href="mailto:sklenka@edv-design.at"><span style="color:windowtext;text-decoration:none">mailto:sklenka@edv-design.at</span></a>><o:p></o:p></p>
<p class="MsoPlainText">>> Internet: <a href="http://www.edv-design.at"><span style="color:windowtext;text-decoration:none">www.edv-design.at</span></a> <<a href="http://www.edv-design.at/"><span style="color:windowtext;text-decoration:none">http://www.edv-design.at/</span></a>><o:p></o:p></p>
<p class="MsoPlainText">>><o:p> </o:p></p>
<p class="MsoPlainText">>><o:p> </o:p></p>
<p class="MsoPlainText">>> _______________________________________________<o:p></o:p></p>
<p class="MsoPlainText">>> gpfsug-discuss mailing list<o:p></o:p></p>
<p class="MsoPlainText">>> gpfsug-discuss at spectrumscale.org<o:p></o:p></p>
<p class="MsoPlainText">>> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">
<span style="color:windowtext;text-decoration:none">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></a><o:p></o:p></p>
<p class="MsoPlainText">>><o:p> </o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText">> --<o:p></o:p></p>
<p class="MsoPlainText">> Giovanni Bracco<o:p></o:p></p>
<p class="MsoPlainText">> phone +39 351 8804788<o:p></o:p></p>
<p class="MsoPlainText">> E-mail <a href="mailto:giovanni.bracco@enea.it"><span style="color:windowtext;text-decoration:none">giovanni.bracco@enea.it</span></a><o:p></o:p></p>
<p class="MsoPlainText">> WWW <a href="http://www.afs.enea.it/bracco"><span style="color:windowtext;text-decoration:none">http://www.afs.enea.it/bracco</span></a><o:p></o:p></p>
<p class="MsoPlainText">> <o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">--<o:p></o:p></p>
<p class="MsoPlainText">Giovanni Bracco<o:p></o:p></p>
<p class="MsoPlainText">phone +39 351 8804788<o:p></o:p></p>
<p class="MsoPlainText">E-mail <a href="mailto:giovanni.bracco@enea.it"><span style="color:windowtext;text-decoration:none">giovanni.bracco@enea.it</span></a><o:p></o:p></p>
<p class="MsoPlainText">WWW <a href="http://www.afs.enea.it/bracco"><span style="color:windowtext;text-decoration:none">http://www.afs.enea.it/bracco</span></a><o:p></o:p></p>
</div>
</body>
</html>