[gpfsug-discuss] OPA HFI and Mellanox HCA on same NSD Server with only ib rdma enabled

[Giovanni Bracco]

In our HPC infrastructure we have 6 NSD server, running CentOS 7.4, each 
of them with with 1 Intel QDR HCA to a QDR Cluster (now 100 nodes 
SandyBridge cpu it was 300 nodes CentOS 6.5), 1 OPA HCA to the main OPA 
Cluster (400 nodes Skylake cpu, CentOS 7.3) and 1 Mellanox FDR to DDN 
storages and it works nicely using RDMA since 2018. GPFS 4.2.3-19.
See
F. Iannone et al., "CRESCO ENEA HPC clusters: a working example of a 
multifabric GPFS Spectrum Scale layout," 2019 International Conference 
on High Performance Computing & Simulation (HPCS), Dublin, Ireland, 
2019, pp. 1051-1052, doi: 10.1109/HPCS48598.2019.918813

When setting up the system the main trick has been:
just use CentOS drivers and do not install OFED
We do not use IPoIB.

Giovanni

On 30/01/21 06:45, Walter Sklenka wrote:
> Hi!
> 
> Is it possible to mix OPAcards and Infininiband HCAs on the same server?
> 
> In the faq 
> https://www.ibm.com/support/knowledgecenter/en/STXKQY/gpfsclustersfaq.html#rdma 
> 
> 
> They talk about RDMA :
> 
> “RDMA is NOT  supported on a node when both Mellanox HCAs and Intel 
> Omni-Path HFIs are ENABLED for RDMA.”
> 
> So do I understand right: When we do NOT enable  the opa interface we 
> can still enable IB ?
> 
> The reason I ask  is, that we have a gpfs cluster of 6 NSD Servers  (wih 
> access to storage)  with opa interfaces which provide access to remote 
> cluster  also via OPA.
> 
> A new cluster with HDR interfaces will be implemented soon
> 
> They shell have access to the same filesystems
> 
> When we add HDR interfaces to  NSD servers  and enable rdma on this 
> network  while disabling rdma on opa we would accept the worse 
> performance via opa . We hope that this provides  still better perf and 
> less technical overhead  than using routers
> 
> Or am I totally wrong?
> 
> Thank you very much and keep healthy!
> 
> Best regards
> 
> Walter
> 
> Mit freundlichen Grüßen
> */Walter Sklenka/*
> */Technical Consultant/*
> 
> EDV-Design Informationstechnologie GmbH
> Giefinggasse 6/1/2, A-1210 Wien
> Tel: +43 1 29 22 165-31
> Fax: +43 1 29 22 165-90
> E-Mail: sklenka at edv-design.at <mailto:sklenka at edv-design.at>
> Internet: www.edv-design.at <http://www.edv-design.at/>
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 

-- 
Giovanni Bracco
phone  +39 351 8804788
E-mail  giovanni.bracco at enea.it
WWW http://www.afs.enea.it/bracco