[gpfsug-discuss] InfiniBand and OmiPath NSD servers

[Ralph Wurthner]

Hello Sean,

you wrote:

> Does this mean that we wouldn't be able to consolidate both IB & OPA
> HCA's in NSD servers and would have to have 2 types of NSD servers?
> 1) InfiniBand exporting and 2) OmniPath exporting?

Eventually there is a solution to your problem. We will have to do some internal research and will come back to you later.


Mit freundlichen Grüßen / Kind regards

Ralph Würthner

IBM Storage Scale Development
IBM Systems & Technology Group, Systems Software Development
________________________________
IBM Data Privacy Statement<https://www.ibm.com/privacy/us/en/>
IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Wolfgang Wendt
Geschäftsführung: David Faller
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294


-----Original Message-----
From: Sean Mc Grath <smcgrat at tcd.ie<mailto:Sean%20Mc%20Grath%20%3csmcgrat at tcd.ie%3e>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug%20main%20discussion%20list%20%3cgpfsug-discuss at gpfsug.org%3e>>
To: gpfsug-discuss at gpfsug.org <gpfsug-discuss at gpfsug.org<mailto:%22gpfsug-discuss at gpfsug.org%22%20%3cgpfsug-discuss at gpfsug.org%3e>>
Subject: [EXTERNAL] [gpfsug-discuss] InfiniBand and OmiPath NSD servers
Date: 09/06/2024 05:52:55 PM

Hi,

We have a GPFS cluster where the NSD servers mount the storage over fibre channel and export the file system over InfiniBand for clients.

We will be getting some used equipment that uses OmniPath.

As per the "IBM Storage Scale Frequently Asked Questions and Answers" it states [1]:

RDMA is not supported on a node when both Mellanox HCAs and Cornelis Networks Omni-Path HFIs are enabled for RDMA.

Does this mean that we wouldn't be able to consolidate both IB & OPA HCA's in NSD servers and would have to have 2 types of NSD servers? 1) InfiniBand exporting and 2) OmniPath exporting?

If so, is it then a matter of using the Multi-Rail over TCP "subnets =" setting in mmchonfig to distinguish which nsd server the clients should connect to? [2].

Or am I completely miss understanding all this?

Many thanks in advance.

Sean

[1] https://www.ibm.com/docs/en/STXKQY/pdf/gpfsclustersfaq.pdf
[2] https://www.ibm.com/docs/en/storage-scale/5.1.6?topic=configuring-multi-rail-over-tcp-mrot

---

Sean McGrath

smcgrat at tcd.ie<mailto:smcgrat at tcd.ie>
Senior Systems Administrator
Research IT, IT Services, Trinity College Dublin
https://www.tcd.ie/itservices/
https://www.tchpc.tcd.ie/

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20240913/ead0af03/attachment-0001.htm>