[gpfsug-discuss] Spectrum Scale, InfiniBand and QoS

Fri Sep 8 09:52:02 BST 2023

Dear Ralph,

Thanks a lot for answering. We tested this yesterday in a test cluster from our customer and it works as it is supposed to be. We are using the rather outdated GPFS Version 5.1.1.4, so this appears to be working since quite some time. One can clearly see the traffic is separated by using perfquery -X -C <HCAddev> …

--
Best regards / Mit freundlichem Gruß
Thomas Bernecker
------------------------------------------------------------------------------------------------------------------
Manager System Integration and Support, HPCE Division
Mobile: +49 (1522) 2851523, Fax: +49 (211) 5369-199, Home Office: +49 (38821) 65091
NEC Deutschland GmbH, Fritz-Vomfelde-Straße 14-16, 40547 Düsseldorf, Germany
Geschäftsführer: Christopher Richard Jackson – Handelsregister Düsseldorf HRB 57941
-------------------------------------------------------------------------------------------------------------------

From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org> On Behalf Of Ralph Wuerthner
Sent: Thursday, September 7, 2023 5:14 PM
To: gpfsug-discuss at gpfsug.org
Subject: Re: [gpfsug-discuss] Spectrum Scale, InfiniBand and QoS

Thomas, you are correct, in the verbsPorts configuration variable description verbsPorts = <device>/<port>/<fabric>/<service level> <service level> refers to the service level which is set for all RC queue pairs
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
    Report Suspicious  <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/BQNorrFsuw!2ztO7JNyHNNBj_2naopARCySMk2bdezjiXwVZZ6ElXCI-WwHm_8KpdBOUsbTdeSimdNt9AiqG3ycS6cFUNk_HGBP-W8NYdLSZa-4oHWTC2b_vElTPhMPSw$>   ‌
ZjQcmQRYFpfptBannerEnd
Thomas,

you are correct, in the verbsPorts configuration variable description

verbsPorts = <device>/<port>/<fabric>/<service level>

<service level> refers to the service level which is set for all RC queue pairs on the specified adapter port. This information is used by the subnet manager to control QoS. Unfortunately I'm not familiar with configuring QoS so I cannot provide additional information on this.

If found the following white paper from Mellanox/Nvidia providing some more information: https://network.nvidia.com/pdf/whitepapers/deploying_qos_wp_10_19_2005.pdf [network.nvidia.com]<https://urldefense.com/v3/__https:/network.nvidia.com/pdf/whitepapers/deploying_qos_wp_10_19_2005.pdf__;!!BQNorrFsuw!gFOqdpq7kHEXBM75HosNGDnXXgDP4yTOdW3wwrdYQ3ocasUUvMstiTx-uZgEEhBFX35YBzRhcYEXbeHsK9UzisZCHvQnxlIJog$> . It is at least a starting point.

Mit freundlichen Grüßen / Kind regards

Ralph Würthner

IBM Storage Scale Development
IBM Systems & Technology Group, Systems Software Development
________________________________

Mobile:
+49 (0) 171 3089472
 IBM Deutschland Research & Development GmbH
[cid:image002.png at 01D9E241.EA4A26A0]
Email:
ralph.wuerthner at de.ibm.com<mailto:ralph.wuerthner at de.ibm.com>
 Wilhelm-Fay-Str. 32

 65936 Frankfurt

 Germany
________________________________
IBM Data Privacy Statement [ibm.com]<https://urldefense.com/v3/__https:/www.ibm.com/privacy/us/en/__;!!BQNorrFsuw!gFOqdpq7kHEXBM75HosNGDnXXgDP4yTOdW3wwrdYQ3ocasUUvMstiTx-uZgEEhBFX35YBzRhcYEXbeHsK9UzisZCHvThvwtBmw$>
IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Gregor Pillen
Geschäftsführung: David Faller
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294

-----Original Message-----
From: Thomas Bernecker <Thomas.Bernecker at EMEA.NEC.COM<mailto:Thomas%20Bernecker%20%3cThomas.Bernecker at EMEA.NEC.COM%3e>>
Reply-To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug%20main%20discussion%20list%20%3cgpfsug-discuss at gpfsug.org%3e>>
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug%20main%20discussion%20list%20%3cgpfsug-discuss at gpfsug.org%3e>>
Subject: [EXTERNAL] Re: [gpfsug-discuss] Spectrum Scale, InfiniBand and QoS
Date: 09/07/2023 11:02:02 AM

Hi Luis, Thanks for your quick response. I was referring to fabric QoS, not Spectrum Scale QoS (which I have some experience with). The reference to slide 10 of said presentation says to configure the RDMA device as follows to use QoS verbsPorts
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
    Report Suspicious  <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/PjiDSg!1e-v7R5zRvmblYv7uGFl7pa6Y4IryZDRZDY_pFV_gcuWmVhOYY_MnKo0CryfRuaBuRmNkPYpLYCHVyCAO62RdiqmuBcGrQ1bbfxDJ3CzqkCGbKUj4BJ_KOcmBuYQq_Z4sy-kxOV4aF_bfA4U$>   ‌
ZjQcmQRYFpfptBannerEnd
Hi Luis,

Thanks for your quick response. I was referring to fabric QoS, not Spectrum Scale QoS (which I have some experience with). The reference to slide 10 of said presentation says to configure the RDMA device as follows to use QoS

verbsPorts = <device>/<port>/<fabric>/<service level>
− List of RDMA ports to be used
− <device> : RDMA device, required, e.g. mlx5_0, mlx5_1
− <port>: RDMA port on device, default 1, valid values are 1 or 2
− <fabric>: virtual fabric number, default 0, valid values are >= 0
Only verbsPorts using a common <fabric> are connected
− <service level>: QoS level, default is 0, valid values defined in SM configuration
The references to SM and verbsPorts being part of RDMA configuration indicates (to me) that this is fabric QoS …

--
Best regards / Mit freundlichem Gruß
Thomas Bernecker
------------------------------------------------------------------------------------------------------------------
Manager System Integration and Support, HPCE Division
Mobile: +49 (1522) 2851523, Fax: +49 (211) 5369-199, Home Office: +49 (38821) 65091
NEC Deutschland GmbH, Fritz-Vomfelde-Straße 14-16, 40547 Düsseldorf, Germany
Geschäftsführer: Christopher Richard Jackson – Handelsregister Düsseldorf HRB 57941
-------------------------------------------------------------------------------------------------------------------

From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>>On Behalf Of Luis Bolinches
Sent: Thursday, September 7, 2023 10:51 AM
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>>
Subject: Re: [gpfsug-discuss] Spectrum Scale, InfiniBand and QoS

Hi Assuming you are talking about GPFS QoS and not any fabric QoS. It would work with any supported fabric, ESS or not. It is a filesystem feature well above all the fabrics and HW. Limits IOPS per class https: //www. ibm. com/docs/en/storage-scale/5. 1. 8?topic=reference-mmchqos-command
ZjQcmQRYFpfptBannerStart
Be Careful With This Message
The sender's identity could not be verified and someone may be impersonating the sender.
    Report Suspicious  <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/BQNorrFsuw!1ltCYpPUsLPrqvNt6ouhTp6NGkpGHjneRgeZQ4EL8X29-niUBGCBdKKkDls1qqoVuPo1KL8kYyJpf4yhW6ztYNlnYstlXTWSM5mLl3UqEvexW0whUlE-4P3Yj_DDkWnTV-cejP_cc5NNq29-2g$>   ‌
ZjQcmQRYFpfptBannerEnd
Hi

Assuming you are talking about GPFS QoS and not any fabric QoS. It would work with any supported fabric, ESS or not. It is a filesystem feature well above all the fabrics and HW. Limits IOPS per class

https://www.ibm.com/docs/en/storage-scale/5.1.8?topic=reference-mmchqos-command [ibm.com] [ibm.com]<https://urldefense.com/v3/__https:/www.ibm.com/docs/en/storage-scale/5.1.8?topic=reference-mmchqos-command__;!!BQNorrFsuw!gFOqdpq7kHEXBM75HosNGDnXXgDP4yTOdW3wwrdYQ3ocasUUvMstiTx-uZgEEhBFX35YBzRhcYEXbeHsK9UzisZCHvSjx_6fmA$>

I recommend (depending on number of nodes and seconds) fine-stats so then you can visualize which PID on which client node is doing what. I believe I did something of that on SSUG London 18 or 19

--
Ystävällisin terveisin/Regards/Saludos/Salutations/Salutacions
Luis Bolinches
Executive IT Specialist
IBM Storage ScaleServer (formerly ESS) developer
Phone: +358503112585

Ab IBM Finland Oy
Toinen linja 7
00530 Helsinki
Uusimaa - Finland

Visitors entrance: Siltasaarenkatu 22

"If you always give you will always have" --  Anonymous

https://www.credly.com/users/luis-bolinches/badges [credly.com] [credly.com]<https://urldefense.com/v3/__https:/www.credly.com/users/luis-bolinches/badges__;!!BQNorrFsuw!gFOqdpq7kHEXBM75HosNGDnXXgDP4yTOdW3wwrdYQ3ocasUUvMstiTx-uZgEEhBFX35YBzRhcYEXbeHsK9UzisZCHvR17Vss7A$>

From: gpfsug-discuss <gpfsug-discuss-bounces at gpfsug.org<mailto:gpfsug-discuss-bounces at gpfsug.org>>On Behalf Of Thomas Bernecker
Sent: Thursday, 7 September 2023 11.36
To: gpfsug-discuss at gpfsug.org<mailto:gpfsug-discuss at gpfsug.org>
Subject: [EXTERNAL] [gpfsug-discuss] Spectrum Scale, InfiniBand and QoS

Dear all, we recently found this presentation from 2021: https: //www. spectrumscaleug. org/wp-content/uploads/2021/05/SSSD21DE-06-Improving-Spectrum-Scale-performance-using-RDMA. pdf On page 10 it explains by setting the RDMA device in a specific
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.
    Report Suspicious  <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/PjiDSg!1e-v7R5zRvmblYv7uGFl7ra7I0IqiHHRBBi-CjVQYUsssj6dhwnB7Lo2ZohM0He5_MhQUpUfgGfOHPXlTwIhxLUDAE1nbLv0ArY1Ygr3eS1cGzhvDzAy1DtrxLl5hHS5FHw6KsNKzpBBnc2d$>   ‌
ZjQcmQRYFpfptBannerEnd
Dear all,

we recently found this presentation from 2021:https://www.spectrumscaleug.org/wp-content/uploads/2021/05/SSSD21DE-06-Improving-Spectrum-Scale-performance-using-RDMA.pdf [spectrumscaleug.org] [spectrumscaleug.org]<https://urldefense.com/v3/__https:/www.spectrumscaleug.org/wp-content/uploads/2021/05/SSSD21DE-06-Improving-Spectrum-Scale-performance-using-RDMA.pdf__;!!BQNorrFsuw!gFOqdpq7kHEXBM75HosNGDnXXgDP4yTOdW3wwrdYQ3ocasUUvMstiTx-uZgEEhBFX35YBzRhcYEXbeHsK9UzisZCHvREwAuGsg$> On page 10 it explains by setting the RDMA device in a specific way one could employ QoS with InfiniBand, which is something we would like to achieve as well. Has anyone a working environment using QoS with InfiniBand outside of the ESS domain? If so, would you share your experience? Sorry for asking a rather broad question, but it seems that this is not well-known stuff …

--
Best regards / Mit freundlichem Gruß
Thomas Bernecker
------------------------------------------------------------------------------------------------------------------
Manager System Integration and Support, HPCE Division
Mobile: +49 (1522) 2851523, Fax: +49 (211) 5369-199, Home Office: +49 (38821) 65091
NEC Deutschland GmbH, Fritz-Vomfelde-Straße 14-16, 40547 Düsseldorf, Germany
Geschäftsführer: Christopher Richard Jackson – Handelsregister Düsseldorf HRB 57941
-------------------------------------------------------------------------------------------------------------------

Unless otherwise stated above:

Oy IBM Finland Ab
PL 265, 00101 Helsinki, Finland
Business ID, Y-tunnus: 0195876-3
Registered in Finland
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org [gpfsug.org]<https://urldefense.com/v3/__http:/gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org__;!!BQNorrFsuw!gFOqdpq7kHEXBM75HosNGDnXXgDP4yTOdW3wwrdYQ3ocasUUvMstiTx-uZgEEhBFX35YBzRhcYEXbeHsK9UzisZCHvSjbs8rzg$>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230908/63e12ceb/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 70 bytes
Desc: image001.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230908/63e12ceb/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 354 bytes
Desc: image002.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230908/63e12ceb/attachment-0003.png>