[gpfsug-discuss] Spectrum Scale, InfiniBand and QoS

[Thomas Bernecker]

Hi Luis,

Thanks for your quick response. I was referring to fabric QoS, not Spectrum Scale QoS (which I have some experience with). The reference to slide 10 of said presentation says to configure the RDMA device as follows to use QoS

verbsPorts = <device>/<port>/<fabric>/<service level>
− List of RDMA ports to be used
− <device> : RDMA device, required, e.g. mlx5_0, mlx5_1
− <port>: RDMA port on device, default 1, valid values are 1 or 2
− <fabric>: virtual fabric number, default 0, valid values are >= 0
Only verbsPorts using a common <fabric> are connected
− <service level>: QoS level, default is 0, valid values defined in SM configuration
The references to SM and verbsPorts being part of RDMA configuration indicates (to me) that this is fabric QoS …

--
Best regards / Mit freundlichem Gruß
Thomas Bernecker
------------------------------------------------------------------------------------------------------------------
Manager System Integration and Support, HPCE Division
Mobile: +49 (1522) 2851523, Fax: +49 (211) 5369-199, Home Office: +49 (38821) 65091
NEC Deutschland GmbH, Fritz-Vomfelde-Straße 14-16, 40547 Düsseldorf, Germany
Geschäftsführer: Christopher Richard Jackson – Handelsregister Düsseldorf HRB 57941
-------------------------------------------------------------------------------------------------------------------

From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> On Behalf Of Luis Bolinches
Sent: Thursday, September 7, 2023 10:51 AM
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Subject: Re: [gpfsug-discuss] Spectrum Scale, InfiniBand and QoS

Hi Assuming you are talking about GPFS QoS and not any fabric QoS. It would work with any supported fabric, ESS or not. It is a filesystem feature well above all the fabrics and HW. Limits IOPS per class https: //www. ibm. com/docs/en/storage-scale/5. 1. 8?topic=reference-mmchqos-command
ZjQcmQRYFpfptBannerStart
Be Careful With This Message
The sender's identity could not be verified and someone may be impersonating the sender.
    Report Suspicious  <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/BQNorrFsuw!1ltCYpPUsLPrqvNt6ouhTp6NGkpGHjneRgeZQ4EL8X29-niUBGCBdKKkDls1qqoVuPo1KL8kYyJpf4yhW6ztYNlnYstlXTWSM5mLl3UqEvexW0whUlE-4P3Yj_DDkWnTV-cejP_cc5NNq29-2g$>   ‌
ZjQcmQRYFpfptBannerEnd
Hi

Assuming you are talking about GPFS QoS and not any fabric QoS. It would work with any supported fabric, ESS or not. It is a filesystem feature well above all the fabrics and HW. Limits IOPS per class

https://www.ibm.com/docs/en/storage-scale/5.1.8?topic=reference-mmchqos-command [ibm.com]<https://urldefense.com/v3/__https:/www.ibm.com/docs/en/storage-scale/5.1.8?topic=reference-mmchqos-command__;!!BQNorrFsuw!kLyDPEkaxGCP9yaFDTa9ncFU_wIG_irxMgBkJIrwT1H-30N9Wziwk81U93JepyFgVaBXCY7echVIaxaaWDoEvxGXUSF5bwZd$>

I recommend (depending on number of nodes and seconds) fine-stats so then you can visualize which PID on which client node is doing what. I believe I did something of that on SSUG London 18 or 19

--
Ystävällisin terveisin/Regards/Saludos/Salutations/Salutacions
Luis Bolinches
Executive IT Specialist
IBM Storage Scale Server (formerly ESS) developer
Phone: +358503112585

Ab IBM Finland Oy
Toinen linja 7
00530 Helsinki
Uusimaa - Finland

Visitors entrance: Siltasaarenkatu 22

"If you always give you will always have" --  Anonymous

https://www.credly.com/users/luis-bolinches/badges [credly.com]<https://urldefense.com/v3/__https:/www.credly.com/users/luis-bolinches/badges__;!!BQNorrFsuw!kLyDPEkaxGCP9yaFDTa9ncFU_wIG_irxMgBkJIrwT1H-30N9Wziwk81U93JepyFgVaBXCY7echVIaxaaWDoEvxGXUVT_rW2x$>

From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>> On Behalf Of Thomas Bernecker
Sent: Thursday, 7 September 2023 11.36
To: gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>
Subject: [EXTERNAL] [gpfsug-discuss] Spectrum Scale, InfiniBand and QoS

Dear all, we recently found this presentation from 2021: https: //www. spectrumscaleug. org/wp-content/uploads/2021/05/SSSD21DE-06-Improving-Spectrum-Scale-performance-using-RDMA. pdf On page 10 it explains by setting the RDMA device in a specific
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
    Report Suspicious  <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/PjiDSg!1e-v7R5zRvmblYv7uGFl7ra7I0IqiHHRBBi-CjVQYUsssj6dhwnB7Lo2ZohM0He5_MhQUpUfgGfOHPXlTwIhxLUDAE1nbLv0ArY1Ygr3eS1cGzhvDzAy1DtrxLl5hHS5FHw6KsNKzpBBnc2d$>   ‌
ZjQcmQRYFpfptBannerEnd
Dear all,

we recently found this presentation from 2021: https://www.spectrumscaleug.org/wp-content/uploads/2021/05/SSSD21DE-06-Improving-Spectrum-Scale-performance-using-RDMA.pdf [spectrumscaleug.org]<https://urldefense.com/v3/__https:/www.spectrumscaleug.org/wp-content/uploads/2021/05/SSSD21DE-06-Improving-Spectrum-Scale-performance-using-RDMA.pdf__;!!BQNorrFsuw!kLyDPEkaxGCP9yaFDTa9ncFU_wIG_irxMgBkJIrwT1H-30N9Wziwk81U93JepyFgVaBXCY7echVIaxaaWDoEvxGXUaGQ5_uh$> On page 10 it explains by setting the RDMA device in a specific way one could employ QoS with InfiniBand, which is something we would like to achieve as well. Has anyone a working environment using QoS with InfiniBand outside of the ESS domain? If so, would you share your experience? Sorry for asking a rather broad question, but it seems that this is not well-known stuff …


--
Best regards / Mit freundlichem Gruß
Thomas Bernecker
------------------------------------------------------------------------------------------------------------------
Manager System Integration and Support, HPCE Division
Mobile: +49 (1522) 2851523, Fax: +49 (211) 5369-199, Home Office: +49 (38821) 65091
NEC Deutschland GmbH, Fritz-Vomfelde-Straße 14-16, 40547 Düsseldorf, Germany
Geschäftsführer: Christopher Richard Jackson – Handelsregister Düsseldorf HRB 57941
-------------------------------------------------------------------------------------------------------------------

Unless otherwise stated above:

Oy IBM Finland Ab
PL 265, 00101 Helsinki, Finland
Business ID, Y-tunnus: 0195876-3
Registered in Finland
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230907/df55921e/attachment-0001.htm>