[gpfsug-discuss] EDR and omnipath

[Aaron Knister]

Here's where I read about the inter-operability of the two:

http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/omni-path-storage-white-paper.pdf

This is what Intel says:

> In a multi-homed file system server, or in a Lustre Networking (LNet) or IP router, a single OpenFabrics Al- liance (OFA) software environment supporting both an Intel OPA HFI and a Mellanox* InfiniBand HCA is required. The OFA software stack is architected to support multiple tar- geted network types. Currently, the OFA stack simultaneously supports iWARP for Ethernet, RDMA over Converged Ethernet (RoCE), and InfiniBand networks, and the Intel OPA network has been added to that list. As the OS distributions implement their OFA stacks, it will be validated to simultaneously support both Intel OPA Host

> Intel is working closely with the major Linux distributors, including Red Hat* and SUSE*, to ensure that Intel OPA support is integrated into their OFA implementation. Once this is accomplished, then simultaneous Mellanox InfiniBand and Intel OPA support will be present in the standard Linux distributions.

So it seems as though Intel is relying on the OS vendors to bridge the 
support gap between them and Mellanox.

-Aaron

On 9/19/16 4:55 PM, Aaron Knister wrote:
> Ah, that makes complete sense. Thanks!
>
> I had been doing some reading about OmniPath and for some reason was
> under the impression the OmniPath adapter could load itself as a driver
> under the verbs stack of OFED. Even so, that raises support concerns as
> you say.
>
> I wonder what folks are doing who have IB-based block storage fabrics
> but wanting to connect to OmniPath-based fabrics?
>
> I'm also curious how GNR customers would be able to serve both IB-based
> and an OmniPath-based fabrics over RDMA where performance is best? This
> is is along the lines of my GPFS protocol router question from the other
> day.
>
> -Aaron
>
> On 9/19/16 4:43 PM, Sven Oehme wrote:
>> Because they both require a different distribution of OFED, which are
>> mutual exclusive to install.
>> in theory if you deploy plain OFED it might work, but that will be hard
>> to find somebody to support.
>>
>>
>> Sent from IBM Verse
>>
>> Aaron Knister --- Re: [gpfsug-discuss] EDR and omnipath ---
>>
>> From:    "Aaron Knister" <aaron.s.knister at nasa.gov>
>> To:    gpfsug-discuss at spectrumscale.org
>> Date:    Mon, Sep 19, 2016 1:35 PM
>> Subject:    Re: [gpfsug-discuss] EDR and omnipath
>>
>> ------------------------------------------------------------------------
>>
>> I must admit, I'm curious as to why one cannot use GPFS with IB and OPA
>> both in RDMA mode. Granted, I know very little about OPA but if it just
>> presents as another verbs device I wonder why it wouldn't "Just work" as
>> long as GPFS is configured correctly.
>>
>> -Aaron
>>
>> On 9/19/16 3:11 AM, Vic Cornell wrote:
>>> Bump
>>>
>>> I can see no reason why that wouldn't work. But it would be nice to a
>>> have an official answer or evidence that it works.
>>>
>>> Vic
>>>
>>>
>>>> On 15 Sep 2016, at 5:49 pm, Brian Marshall <mimarsh2 at vt.edu
>>>> <mailto:mimarsh2 at vt.edu>> wrote:
>>>>
>>>> All,
>>>>
>>>> I see in the GPFS FAQ A6.3 the statement below.  Is it possible to
>>>> have GPFS do RDMA over EDR infiniband and non-RDMA communication over
>>>> omnipath (IP over fabric) when each NSD server has an EDR card and a
>>>> OPA card installed?
>>>>
>>>>
>>>>
>>>> RDMA is not supported on a node when both Mellanox HCAs and Intel
>>>> Omni-Path HFIs are enabled for RDMA.
>>>>
>>>> _______________________________________________
>>>> gpfsug-discuss mailing list
>>>> gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
>>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>>
>>>
>>>
>>> _______________________________________________
>>> gpfsug-discuss mailing list
>>> gpfsug-discuss at spectrumscale.org
>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>>
>>
>> --
>> Aaron Knister
>> NASA Center for Climate Simulation (Code 606.2)
>> Goddard Space Flight Center
>> (301) 286-2776
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>
>>
>>
>>
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>
>

-- 
Aaron Knister
NASA Center for Climate Simulation (Code 606.2)
Goddard Space Flight Center
(301) 286-2776