[gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled
giovanni.bracco at enea.it
Mon Feb 1 19:41:43 GMT 2021
On 30/01/21 21:01, Walter Sklenka wrote:
> Hi Giovanni!
> Thats great! Many thanks for your fast and detailed answer!!!!
> So this is the way we will go too!
> Have a nice weekend and keep healthy!
> Best regards
I suppose you will implement the solution with more recent versions of
the software components, so please let me know if everything works!
If yu have any issues I am ready to discuss!
> -----Original Message-----
> From: Giovanni Bracco <giovanni.bracco at enea.it>
> Sent: Samstag, 30. Jänner 2021 18:08
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>; Walter Sklenka <Walter.Sklenka at EDV-Design.at>
> Subject: Re: [gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled
> In our HPC infrastructure we have 6 NSD server, running CentOS 7.4, each of them with with 1 Intel QDR HCA to a QDR Cluster (now 100 nodes SandyBridge cpu it was 300 nodes CentOS 6.5), 1 OPA HCA to the main OPA Cluster (400 nodes Skylake cpu, CentOS 7.3) and 1 Mellanox FDR to DDN storages and it works nicely using RDMA since 2018. GPFS 4.2.3-19.
> F. Iannone et al., "CRESCO ENEA HPC clusters: a working example of a multifabric GPFS Spectrum Scale layout," 2019 International Conference on High Performance Computing & Simulation (HPCS), Dublin, Ireland, 2019, pp. 1051-1052, doi: 10.1109/HPCS48598.2019.918813
> When setting up the system the main trick has been:
> just use CentOS drivers and do not install OFED We do not use IPoIB.
> On 30/01/21 06:45, Walter Sklenka wrote:
>> Is it possible to mix OPAcards and Infininiband HCAs on the same server?
>> In the faq
>> They talk about RDMA :
>> "RDMA is NOT supported on a node when both Mellanox HCAs and Intel
>> Omni-Path HFIs are ENABLED for RDMA."
>> So do I understand right: When we do NOT enable the opa interface we
>> can still enable IB ?
>> The reason I ask is, that we have a gpfs cluster of 6 NSD Servers
>> (wih access to storage) with opa interfaces which provide access to
>> remote cluster also via OPA.
>> A new cluster with HDR interfaces will be implemented soon
>> They shell have access to the same filesystems
>> When we add HDR interfaces to NSD servers and enable rdma on this
>> network while disabling rdma on opa we would accept the worse
>> performance via opa . We hope that this provides still better perf
>> and less technical overhead than using routers
>> Or am I totally wrong?
>> Thank you very much and keep healthy!
>> Best regards
>> Mit freundlichen Grüßen
>> */Walter Sklenka/*
>> */Technical Consultant/*
>> EDV-Design Informationstechnologie GmbH Giefinggasse 6/1/2, A-1210
>> Tel: +43 1 29 22 165-31
>> Fax: +43 1 29 22 165-90
>> E-Mail: sklenka at edv-design.at <mailto:sklenka at edv-design.at>
>> Internet: www.edv-design.at <http://www.edv-design.at/>
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
> Giovanni Bracco
> phone +39 351 8804788
> E-mail giovanni.bracco at enea.it
> WWW http://www.afs.enea.it/bracco
phone +39 351 8804788
E-mail giovanni.bracco at enea.it
More information about the gpfsug-discuss