<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:"Yu Gothic";
panose-1:2 11 4 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@Yu Gothic";
panose-1:2 11 4 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Hi Luis,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks for your quick response. I was referring to fabric QoS, not Spectrum Scale QoS (which I have some experience with). The reference to slide 10 of said presentation says to configure the RDMA device as
follows to use QoS<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="FR" style="font-size:11.0pt">verbsPorts = <device>/<port>/<fabric>/<service level><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">− List of RDMA ports to be used<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">− <device> : RDMA device, required, e.g. mlx5_0, mlx5_1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">− <port>: RDMA port on device, default 1, valid values are 1 or 2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">− <fabric>: virtual fabric number, default 0, valid values are >= 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Only verbsPorts using a common <fabric> are connected<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">− <service level>: QoS level, default is 0, valid values defined in SM configuration<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The references to SM and verbsPorts being part of RDMA configuration indicates (to me) that this is fabric QoS …<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div name="divtagdefaultwrapper">
<div name="divtagdefaultwrapper">
<div>
<div name="divtagdefaultwrapper">
<div name="divtagdefaultwrapper">
<div>
<p class="MsoNormal"><span lang="DE" style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">--</span><span lang="DE" style="font-size:11.0pt;mso-fareast-language:DE"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="DE" style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">Best regards / Mit freundlichem Gruß<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="DE" style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">Thomas Bernecker<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">------------------------------------------------------------------------------------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">Manager System Integration and Support, HPCE Division<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">Mobile: +49 (1522) 2851523, Fax: +49 (211) 5369-199, Home Office: +49 (38821) 65091<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">NEC Deutschland GmbH, Fritz-Vomfelde-Straße 14-16, 40547 Düsseldorf, Germany<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="DE" style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">Geschäftsführer: Christopher Richard Jackson – Handelsregister Düsseldorf HRB 57941<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:DE">-------------------------------------------------------------------------------------------------------------------</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:11.0pt">From:</span></b><span style="font-size:11.0pt"> gpfsug-discuss <gpfsug-discuss-bounces@gpfsug.org>
<b>On Behalf Of </b>Luis Bolinches<br>
<b>Sent:</b> Thursday, September 7, 2023 10:51 AM<br>
<b>To:</b> gpfsug main discussion list <gpfsug-discuss@gpfsug.org><br>
<b>Subject:</b> Re: [gpfsug-discuss] Spectrum Scale, InfiniBand and QoS<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">Hi Assuming you are talking about GPFS QoS and not any fabric QoS. It would work with any supported fabric, ESS or not. It is a filesystem feature well above all
the fabrics and HW. Limits IOPS per class https: //www. ibm. com/docs/en/storage-scale/5. 1. 8?topic=reference-mmchqos-command
<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerStart<o:p></o:p></span></p>
</div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;border-radius:4px">
<tbody>
<tr>
<td style="padding:12.0pt 0cm 12.0pt 0cm">
<table class="MsoNormalTable" border="1" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;background:#FADF9B;border:none;border-top:solid #F2B004 3.0pt">
<tbody>
<tr>
<td valign="top" style="border:none;padding:0cm 7.5pt 3.75pt 4.5pt">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left">
<tbody>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<p class="MsoNormal"><b><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">Be Careful With This Message
<o:p></o:p></span></b></p>
</td>
</tr>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:black">The sender's identity could not be verified and someone may be impersonating the sender.
<o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="right">
<tbody>
<tr>
<td style="padding:3.0pt 0cm 3.0pt 0cm">
<p class="MsoNormal"> <a href="https://us-phishalarm-ewt.proofpoint.com/EWT/v1/BQNorrFsuw!1ltCYpPUsLPrqvNt6ouhTp6NGkpGHjneRgeZQ4EL8X29-niUBGCBdKKkDls1qqoVuPo1KL8kYyJpf4yhW6ztYNlnYstlXTWSM5mLl3UqEvexW0whUlE-4P3Yj_DDkWnTV-cejP_cc5NNq29-2g$" target="_blank"><strong><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;border:solid #666666 1.0pt;padding:6.0pt;font-weight:normal;text-decoration:none"> Report Suspicious </span></strong></a>
<span style="font-size:11.0pt"><o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerEnd<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">Hi<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">Assuming you are talking about GPFS QoS and not any fabric QoS. It would work with any supported fabric, ESS or not. It is a filesystem feature well above all the fabrics and HW.
Limits IOPS per class<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><a href="https://urldefense.com/v3/__https:/www.ibm.com/docs/en/storage-scale/5.1.8?topic=reference-mmchqos-command__;!!BQNorrFsuw!kLyDPEkaxGCP9yaFDTa9ncFU_wIG_irxMgBkJIrwT1H-30N9Wziwk81U93JepyFgVaBXCY7echVIaxaaWDoEvxGXUSF5bwZd$">https://www.ibm.com/docs/en/storage-scale/5.1.8?topic=reference-mmchqos-command
[ibm.com]</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">I recommend (depending on number of nodes and seconds) fine-stats so then you can visualize which PID on which client node is doing what. I believe I did something of that on SSUG
London 18 or 19<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">--</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">Ystävällisin terveisin/Regards/Saludos/Salutations/Salutacions<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">Luis Bolinches<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">Executive IT Specialist<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">IBM Storage Scale
</span><span style="font-size:11.0pt;mso-ligatures:standardcontextual">Server (formerly ESS) developer
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">Phone: +358503112585<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">Ab IBM Finland Oy<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">Toinen linja 7<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">00530 Helsinki<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">Uusimaa - Finland<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">Visitors entrance:</span><span style="font-size:11.0pt;mso-ligatures:standardcontextual"> Siltasaarenkatu 22<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">"If you always give you will always have" -- Anonymous<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual"><a href="https://urldefense.com/v3/__https:/www.credly.com/users/luis-bolinches/badges__;!!BQNorrFsuw!kLyDPEkaxGCP9yaFDTa9ncFU_wIG_irxMgBkJIrwT1H-30N9Wziwk81U93JepyFgVaBXCY7echVIaxaaWDoEvxGXUVT_rW2x$">https://www.credly.com/users/luis-bolinches/badges
[credly.com]</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt">From:</span></b><span lang="EN-US" style="font-size:11.0pt"> gpfsug-discuss <<a href="mailto:gpfsug-discuss-bounces@gpfsug.org">gpfsug-discuss-bounces@gpfsug.org</a>>
<b>On Behalf Of </b>Thomas Bernecker<br>
<b>Sent:</b> Thursday, 7 September 2023 11.36<br>
<b>To:</b> <a href="mailto:gpfsug-discuss@gpfsug.org">gpfsug-discuss@gpfsug.org</a><br>
<b>Subject:</b> [EXTERNAL] [gpfsug-discuss] Spectrum Scale, InfiniBand and QoS<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">Dear all, we recently found this presentation from 2021: https: //www. spectrumscaleug. org/wp-content/uploads/2021/05/SSSD21DE-06-Improving-Spectrum-Scale-performance-using-RDMA. pdf
On page 10 it explains by setting the RDMA device in a specific <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerStart<o:p></o:p></span></p>
</div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;border-radius:4px">
<tbody>
<tr>
<td style="padding:12.0pt 0cm 12.0pt 0cm">
<table class="MsoNormalTable" border="1" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;background:#D0D8DC;border:none;border-top:solid #90A4AE 3.0pt">
<tbody>
<tr>
<td valign="top" style="border:none;padding:0cm 7.5pt 3.75pt 4.5pt">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left">
<tbody>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<p class="MsoNormal"><b><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">This Message Is From an External Sender
<o:p></o:p></span></b></p>
</td>
</tr>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:black">This message came from outside your organization.
<o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="right">
<tbody>
<tr>
<td style="padding:3.0pt 0cm 3.0pt 0cm">
<p class="MsoNormal"> <a href="https://us-phishalarm-ewt.proofpoint.com/EWT/v1/PjiDSg!1e-v7R5zRvmblYv7uGFl7ra7I0IqiHHRBBi-CjVQYUsssj6dhwnB7Lo2ZohM0He5_MhQUpUfgGfOHPXlTwIhxLUDAE1nbLv0ArY1Ygr3eS1cGzhvDzAy1DtrxLl5hHS5FHw6KsNKzpBBnc2d$" target="_blank"><strong><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;border:solid #666666 1.0pt;padding:6.0pt;font-weight:normal;text-decoration:none"> Report Suspicious </span></strong></a>
<span style="font-size:11.0pt"><o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerEnd<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">Dear all,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">we recently found this presentation from 2021:
<a href="https://urldefense.com/v3/__https:/www.spectrumscaleug.org/wp-content/uploads/2021/05/SSSD21DE-06-Improving-Spectrum-Scale-performance-using-RDMA.pdf__;!!BQNorrFsuw!kLyDPEkaxGCP9yaFDTa9ncFU_wIG_irxMgBkJIrwT1H-30N9Wziwk81U93JepyFgVaBXCY7echVIaxaaWDoEvxGXUaGQ5_uh$">
https://www.spectrumscaleug.org/wp-content/uploads/2021/05/SSSD21DE-06-Improving-Spectrum-Scale-performance-using-RDMA.pdf [spectrumscaleug.org]</a> On page 10 it explains by setting the RDMA device in a specific way one could employ QoS with InfiniBand, which
is something we would like to achieve as well. Has anyone a working environment using QoS with InfiniBand outside of the ESS domain? If so, would you share your experience? Sorry for asking a rather broad question, but it seems that this is not well-known
stuff …<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual"><o:p> </o:p></span></p>
<div name="divtagdefaultwrapper">
<div name="divtagdefaultwrapper">
<div>
<div name="divtagdefaultwrapper">
<div name="divtagdefaultwrapper">
<div>
<p class="MsoNormal"><span lang="DE" style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">--</span><span lang="DE" style="font-size:11.0pt;mso-fareast-language:DE"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="DE" style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">Best regards / Mit freundlichem Gruß<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="DE" style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">Thomas Bernecker<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">------------------------------------------------------------------------------------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">Manager System Integration and Support, HPCE Division<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">Mobile: +49 (1522) 2851523, Fax: +49 (211) 5369-199, Home Office: +49 (38821) 65091<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">NEC Deutschland GmbH, Fritz-Vomfelde-Straße 14-16, 40547 Düsseldorf, Germany<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="DE" style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">Geschäftsführer: Christopher Richard Jackson – Handelsregister Düsseldorf HRB 57941<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:DE">-------------------------------------------------------------------------------------------------------------------</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Unless otherwise stated above:<br>
<br>
Oy IBM Finland Ab<br>
PL 265, 00101 Helsinki, Finland<br>
Business ID, Y-tunnus: 0195876-3<br>
Registered in Finland<o:p></o:p></span></p>
</div>
</div>
</body>
</html>