[gpfsug-discuss] Tune OS for Mellanox IB/ETH HCA on Power hardware - should I run mlnx_affinity or mlnx_tune or sysctl tuning?

[Billich Heinrich Rainer (ID SD)]

Hello, 

We run Spectrum Scale on Power hardware - le and be - and Mellanox IB and VPI cards. We did not enable any automatic tuning at system start by the usual Mellanox scripts.

/etc/infiniband/openib.conf contains

# Run /usr/sbin/mlnx_affinity
RUN_AFFINITY_TUNER=no
# Run /usr/sbin/mlnx_tune
RUN_MLNX_TUNE=no
# Run sysctl performance tuning script
RUN_SYSCTL=no

I wonder if we should enable this scripts?  
Are they of no use on ppc64 and ppc64le or does some other script run them?  

They aren't enabled on ESS systems, either.  /proc/interrupts shows that the interrupts likely are attached and distributed to all cores on the closest cpu. Which is good.

I did google to find  advise on Mellanox tuning for power hardware but found none. I would like to get a bit more insight in this topic.

We run RHEL 7.6 and 7.7 , Scale 5.0.4 and 5.0.5 and ESS 5.3.6.1. We do the usual sysctl based tuning. 

ESS 5.3.6.1 includes a new script /xcatpost/mlnx_params.sh which adds some tuning of PCI and Mellanox FW settings :

Description: Check and update PCI Express Read Request Size to 4KB and INT_LOG_MAX_PAYLOAD_SIZE to 12

So at least some tuning is done.

Cheers,

Heiner
-- 
=======================
Heinrich Billich
ETH Zürich
Informatikdienste
Tel.: +41 44 632 72 56
heinrich.billich at id.ethz.ch
========================