<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
We had an environment like this and had gateway nodes between the OPA and IB networks that had both cards. The DDN NetScaler in this case was IB only. Doesn’t answer your question, but that does work.
<div><br id="lineBreakAtBeginningOfMessage">
<div>
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
--<br>
#BlackLivesMatter</div>
<div style="color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
____<br>
|| \\UTGERS, |---------------------------*O*---------------------------<br>
||_// the State<span class="Apple-tab-span" style="white-space: pre;"> </span> | Ryan Novosielski - novosirj@rutgers.edu<br>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus<br>
|| \\ of NJ<span class="Apple-tab-span" style="white-space: pre;"> </span> | Office of Advanced Research Computing - MSB A555B, Newark<br>
`'</div>
</div>
</div>
</div>
</div>
</div>
<div><br>
<blockquote type="cite">
<div>On Sep 6, 2024, at 11:52, Sean Mc Grath <smcgrat@tcd.ie> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div>Hi,<br>
<br>
We have a GPFS cluster where the NSD servers mount the storage over fibre channel and export the file system over InfiniBand for clients.<br>
<br>
We will be getting some used equipment that uses OmniPath. <br>
<br>
As per the "IBM Storage Scale Frequently Asked Questions and Answers" it states [1]:<br>
<br>
<blockquote type="cite">RDMA is not supported on a node when both Mellanox HCAs and Cornelis Networks Omni-Path HFIs are enabled for RDMA.<br>
</blockquote>
<br>
Does this mean that we wouldn't be able to consolidate both IB & OPA HCA's in NSD servers and would have to have 2 types of NSD servers? 1) InfiniBand exporting and 2) OmniPath exporting?<br>
<br>
If so, is it then a matter of using the Multi-Rail over TCP "subnets =" setting in mmchonfig to distinguish which nsd server the clients should connect to? [2].<br>
<br>
Or am I completely miss understanding all this?<br>
<br>
Many thanks in advance.<br>
<br>
Sean<br>
<br>
[1] https://www.ibm.com/docs/en/STXKQY/pdf/gpfsclustersfaq.pdf <br>
[2] https://www.ibm.com/docs/en/storage-scale/5.1.6?topic=configuring-multi-rail-over-tcp-mrot
<br>
<br>
---<br>
<br>
Sean McGrath<br>
<br>
smcgrat@tcd.ie<br>
Senior Systems Administrator<br>
Research IT, IT Services, Trinity College Dublin<br>
https://www.tcd.ie/itservices/<br>
https://www.tchpc.tcd.ie/<br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at gpfsug.org<br>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</body>
</html>