<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style>pre,code,address {
margin: 0px;
}
h1,h2,h3,h4,h5,h6 {
margin-top: 0.2em;
margin-bottom: 0.2em;
}
ol,ul {
margin-top: 0em;
margin-bottom: 0em;
}
blockquote {
margin-top: 0em;
margin-bottom: 0em;
}
</style>
</head>
<body>
<div>Hello Sean,</div>
<div><br>
</div>
<div>you wrote:</div>
<div><br>
</div>
<div>> Does this mean that we wouldn't be able to consolidate both IB & OPA</div>
<div>> HCA's in NSD servers and would have to have 2 types of NSD servers?</div>
<div>> 1) InfiniBand exporting and 2) OmniPath exporting?</div>
<div><br>
</div>
<div>Eventually there is a solution to your problem. We will have to do some internal research and will come back to you later.</div>
<div><br>
</div>
<div><br>
</div>
<div><span>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Cantarell; font-size: 14.666666984558105px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-tap-highlight-color: rgba(0, 0, 0, 0.4); -webkit-text-stroke-width: 0px; text-decoration: none;">
Mit freundlichen Grüßen / Kind regards</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Cantarell; font-size: 14.666666984558105px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-tap-highlight-color: rgba(0, 0, 0, 0.4); -webkit-text-stroke-width: 0px; text-decoration: none;">
<br>
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Cantarell; font-size: 14.666666984558105px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-tap-highlight-color: rgba(0, 0, 0, 0.4); -webkit-text-stroke-width: 0px; text-decoration: none;">
<b><font size="4">Ralph Würthner</font></b></div>
<div><br>
</div>
<div><font size="2"><font color="#0055AA">IBM Storage Scale Development</font><br>
<font color="#0055AA">IBM Systems & Technology Group, Systems Software Development</font></font>
<table cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr valign="top">
<td class="small_font" colspan="4" width="800" valign="middle"></td>
</tr>
<tr valign="top">
<td class="small_font" colspan="4" width="800" valign="middle">
<hr style="color:#696969; " width="100%" size="2" align="left">
</td>
</tr>
<tr valign="top">
<td class="foot" colspan="4" width="800" valign="middle"><font size="2"><a href="https://www.ibm.com/privacy/us/en/"><font face="Arial" color="#0000FF"><u>IBM Data Privacy Statement</u></font></a><font face="Arial" color="#999999"> </font></font></td>
</tr>
<tr valign="top">
<td class="foot" colspan="4" width="800" valign="middle"><font face="Arial" color="#999999" size="2">IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Wolfgang Wendt<br>
Geschäftsführung: David Faller<br>
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
</font></td>
</tr>
</tbody>
</table>
<div>
<div><br>
</div>
</div>
</div>
</span></div>
<div><br>
</div>
<div>-----Original Message-----</div>
<div><b>From</b>: Sean Mc Grath <<a href="mailto:Sean%20Mc%20Grath%20%3csmcgrat@tcd.ie%3e">smcgrat@tcd.ie</a>></div>
<div><b>Reply-To</b>: gpfsug main discussion list <<a href="mailto:gpfsug%20main%20discussion%20list%20%3cgpfsug-discuss@gpfsug.org%3e">gpfsug-discuss@gpfsug.org</a>></div>
<div><b>To</b>: gpfsug-discuss@gpfsug.org <<a href="mailto:%22gpfsug-discuss@gpfsug.org%22%20%3cgpfsug-discuss@gpfsug.org%3e">gpfsug-discuss@gpfsug.org</a>></div>
<div><b>Subject</b>: [EXTERNAL] [gpfsug-discuss] InfiniBand and OmiPath NSD servers</div>
<div><b>Date</b>: 09/06/2024 05:52:55 PM</div>
<div><br>
</div>
<div>Hi,<br>
</div>
<div><br>
</div>
<div>We have a GPFS cluster where the NSD servers mount the storage over fibre channel and export the file system over InfiniBand for clients.<br>
</div>
<div><br>
</div>
<div>We will be getting some used equipment that uses OmniPath. <br>
</div>
<div><br>
</div>
<div>As per the "IBM Storage Scale Frequently Asked Questions and Answers" it states [1]:<br>
</div>
<div><br>
</div>
<blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex">
<div>RDMA is not supported on a node when both Mellanox HCAs and Cornelis Networks Omni-Path HFIs are enabled for RDMA.<br>
</div>
</blockquote>
<div><br>
</div>
<div>Does this mean that we wouldn't be able to consolidate both IB & OPA HCA's in NSD servers and would have to have 2 types of NSD servers? 1) InfiniBand exporting and 2) OmniPath exporting?<br>
</div>
<div><br>
</div>
<div>If so, is it then a matter of using the Multi-Rail over TCP "subnets =" setting in mmchonfig to distinguish which nsd server the clients should connect to? [2].<br>
</div>
<div><br>
</div>
<div>Or am I completely miss understanding all this?<br>
</div>
<div><br>
</div>
<div>Many thanks in advance.<br>
</div>
<div><br>
</div>
<div>Sean<br>
</div>
<div><br>
</div>
<div>[1] <a href="https://www.ibm.com/docs/en/STXKQY/pdf/gpfsclustersfaq.pdf">https://www.ibm.com/docs/en/STXKQY/pdf/gpfsclustersfaq.pdf</a> <br>
</div>
<div>[2] <a href="https://www.ibm.com/docs/en/storage-scale/5.1.6?topic=configuring-multi-rail-over-tcp-mrot">https://www.ibm.com/docs/en/storage-scale/5.1.6?topic=configuring-multi-rail-over-tcp-mrot</a> <br>
</div>
<div><br>
</div>
<div>---<br>
</div>
<div><br>
</div>
<div>Sean McGrath<br>
</div>
<div><br>
</div>
<div><a href="mailto:smcgrat@tcd.ie">smcgrat@tcd.ie</a><br>
</div>
<div>Senior Systems Administrator<br>
</div>
<div>Research IT, IT Services, Trinity College Dublin<br>
</div>
<div><a href="https://www.tcd.ie/itservices/">https://www.tcd.ie/itservices/</a> <br>
</div>
<div><a href="https://www.tchpc.tcd.ie/">https://www.tchpc.tcd.ie/</a> <br>
</div>
<div><br>
</div>
<div>_______________________________________________<br>
</div>
<div>gpfsug-discuss mailing list<br>
</div>
<div>gpfsug-discuss at gpfsug.org<br>
</div>
<div><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org">http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org</a> <br>
</div>
</body>
</html>