<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:"Yu Gothic";
panose-1:2 11 4 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@Yu Gothic";
panose-1:2 11 4 0 0 0 0 0 0 0;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:Cantarell;
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
mso-ligatures:none;}
span.EmailStyle32
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Dear Ralph,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks a lot for answering. We tested this yesterday in a test cluster from our customer and it works as it is supposed to be. We are using the rather outdated GPFS Version 5.1.1.4, so this appears to be working
since quite some time. One can clearly see the traffic is separated by using perfquery -X -C <HCAddev> …<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div name="divtagdefaultwrapper">
<div name="divtagdefaultwrapper">
<div>
<div name="divtagdefaultwrapper">
<div name="divtagdefaultwrapper">
<div>
<p class="MsoNormal"><span lang="DE" style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">--</span><span lang="DE" style="font-size:11.0pt;mso-fareast-language:DE"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="DE" style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">Best regards / Mit freundlichem Gruß<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="DE" style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">Thomas Bernecker<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">------------------------------------------------------------------------------------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">Manager System Integration and Support, HPCE Division<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">Mobile: +49 (1522) 2851523, Fax: +49 (211) 5369-199, Home Office: +49 (38821) 65091<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">NEC Deutschland GmbH, Fritz-Vomfelde-Straße 14-16, 40547 Düsseldorf, Germany<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="DE" style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">Geschäftsführer: Christopher Richard Jackson – Handelsregister Düsseldorf HRB 57941<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:DE">-------------------------------------------------------------------------------------------------------------------</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:11.0pt">From:</span></b><span style="font-size:11.0pt"> gpfsug-discuss <gpfsug-discuss-bounces@gpfsug.org>
<b>On Behalf Of </b>Ralph Wuerthner<br>
<b>Sent:</b> Thursday, September 7, 2023 5:14 PM<br>
<b>To:</b> gpfsug-discuss@gpfsug.org<br>
<b>Subject:</b> Re: [gpfsug-discuss] Spectrum Scale, InfiniBand and QoS<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">Thomas, you are correct, in the verbsPorts configuration variable description verbsPorts = <device>/<port>/<fabric>/<service level> <service level> refers to the
service level which is set for all RC queue pairs <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerStart<o:p></o:p></span></p>
</div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;border-radius:4px">
<tbody>
<tr>
<td style="padding:12.0pt 0cm 12.0pt 0cm">
<table class="MsoNormalTable" border="1" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;background:#D0D8DC;border:none;border-top:solid #90A4AE 3.0pt">
<tbody>
<tr>
<td valign="top" style="border:none;padding:0cm 7.5pt 3.75pt 4.5pt">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left">
<tbody>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<p class="MsoNormal"><b><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">This Message Is From an External Sender
<o:p></o:p></span></b></p>
</td>
</tr>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:black">This message came from outside your organization.
<o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="right">
<tbody>
<tr>
<td style="padding:3.0pt 0cm 3.0pt 0cm">
<p class="MsoNormal"> <a href="https://us-phishalarm-ewt.proofpoint.com/EWT/v1/BQNorrFsuw!2ztO7JNyHNNBj_2naopARCySMk2bdezjiXwVZZ6ElXCI-WwHm_8KpdBOUsbTdeSimdNt9AiqG3ycS6cFUNk_HGBP-W8NYdLSZa-4oHWTC2b_vElTPhMPSw$" target="_blank"><strong><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;border:solid #666666 1.0pt;padding:6.0pt;font-weight:normal;text-decoration:none"> Report Suspicious </span></strong></a>
<span style="font-size:11.0pt"><o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerEnd<o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Thomas,<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">you are correct, in the verbsPorts configuration variable description<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<pre style="margin-left:3ch">verbsPorts = <device>/<port>/<fabric>/<service level><o:p></o:p></pre>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cantarell",serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cantarell",serif;color:black"><service level> refers to the service level which is set for all RC queue pairs on the specified adapter port. This information is used by the subnet manager to control
QoS. Unfortunately I'm not familiar with configuring QoS so I cannot provide additional information on this.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cantarell",serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cantarell",serif;color:black">If found the following white paper from Mellanox/Nvidia providing some more information: <a href="https://urldefense.com/v3/__https:/network.nvidia.com/pdf/whitepapers/deploying_qos_wp_10_19_2005.pdf__;!!BQNorrFsuw!gFOqdpq7kHEXBM75HosNGDnXXgDP4yTOdW3wwrdYQ3ocasUUvMstiTx-uZgEEhBFX35YBzRhcYEXbeHsK9UzisZCHvQnxlIJog$">https://network.nvidia.com/pdf/whitepapers/deploying_qos_wp_10_19_2005.pdf
[network.nvidia.com]</a> . It is at least a starting point.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cantarell",serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cantarell",serif;color:black">Mit freundlichen Grüßen / Kind regards<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Cantarell",serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><b><span style="font-size:13.5pt;font-family:"Cantarell",serif;color:black">Ralph Würthner</span></b><span style="font-size:11.0pt;font-family:"Cantarell",serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#0055AA">IBM Storage Scale Development</span><br>
<span style="color:#0055AA">IBM Systems & Technology Group, Systems Software Development</span><span style="font-size:11.0pt">
<o:p></o:p></span></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td width="800" colspan="4" style="width:600.0pt;padding:0cm 0cm 0cm 0cm">
<div class="MsoNormal"><span style="font-size:11.0pt">
<hr size="1" width="100%" noshade="" style="color:dimgray" align="left">
</span></div>
</td>
</tr>
<tr>
<td width="800" colspan="4" style="width:600.0pt;padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><span style="font-size:11.0pt"><img border="0" width="1" height="1" style="width:.0138in;height:.0138in" id="Picture_x0020_2" src="cid:image001.png@01D9E241.EA4A26A0"></span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</td>
</tr>
<tr>
<td width="62" style="width:46.5pt;padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><span style="color:#0055AA">Mobile:</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</td>
<td width="235" style="width:176.25pt;padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><span style="color:#0055AA">+49 (0) 171 3089472</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</td>
<td width="405" style="width:303.75pt;padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><span style="color:#0055AA"> IBM Deutschland Research & Development GmbH</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</td>
<td width="98" rowspan="4" style="width:73.5pt;padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal" align="right" style="text-align:right"><img border="0" width="87" height="30" style="width:.9027in;height:.3125in" id="Picture_x0020_3" src="cid:image002.png@01D9E241.EA4A26A0"><span style="font-size:11.0pt"><o:p></o:p></span></p>
</td>
</tr>
<tr>
<td width="62" style="width:46.5pt;padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><span style="color:#0055AA">Email:</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</td>
<td width="235" style="width:176.25pt;padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><span style="color:#0055AA"><a href="mailto:ralph.wuerthner@de.ibm.com">ralph.wuerthner@de.ibm.com</a></span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</td>
<td width="405" style="width:303.75pt;padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><span style="color:#0055AA"> Wilhelm-Fay-Str. 32</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</td>
</tr>
<tr>
<td width="62" style="width:46.5pt;padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><img border="0" width="1" height="1" style="width:.0069in;height:.0069in" id="Picture_x0020_4" src="cid:image001.png@01D9E241.EA4A26A0"><span style="font-size:11.0pt"><o:p></o:p></span></p>
</td>
<td width="235" style="width:176.25pt;padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><img border="0" width="1" height="1" style="width:.0069in;height:.0069in" id="Picture_x0020_5" src="cid:image001.png@01D9E241.EA4A26A0"><span style="font-size:11.0pt"><o:p></o:p></span></p>
</td>
<td width="405" style="width:303.75pt;padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><span style="color:#0055AA"> 65936 Frankfurt</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</td>
</tr>
<tr>
<td width="62" style="width:46.5pt;padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><img border="0" width="1" height="1" style="width:.0069in;height:.0069in" id="Picture_x0020_6" src="cid:image001.png@01D9E241.EA4A26A0"><span style="font-size:11.0pt"><o:p></o:p></span></p>
</td>
<td width="235" style="width:176.25pt;padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><img border="0" width="1" height="1" style="width:.0069in;height:.0069in" id="Picture_x0020_7" src="cid:image001.png@01D9E241.EA4A26A0"><span style="font-size:11.0pt"><o:p></o:p></span></p>
</td>
<td width="405" style="width:303.75pt;padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:#0055AA"> </span><span style="color:#0055AA">Germany</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</td>
</tr>
<tr>
<td width="800" colspan="4" style="width:600.0pt;padding:0cm 0cm 0cm 0cm">
<div class="MsoNormal"><span style="font-size:11.0pt">
<hr size="1" width="100%" noshade="" style="color:dimgray" align="left">
</span></div>
</td>
</tr>
<tr>
<td width="800" colspan="4" style="width:600.0pt;padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><a href="https://urldefense.com/v3/__https:/www.ibm.com/privacy/us/en/__;!!BQNorrFsuw!gFOqdpq7kHEXBM75HosNGDnXXgDP4yTOdW3wwrdYQ3ocasUUvMstiTx-uZgEEhBFX35YBzRhcYEXbeHsK9UzisZCHvThvwtBmw$"><span style="font-family:"Arial",sans-serif;color:blue">IBM
Data Privacy Statement</span> [ibm.com]</a><span style="font-family:"Arial",sans-serif;color:#999999"> </span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</td>
</tr>
<tr>
<td width="800" colspan="4" style="width:600.0pt;padding:0cm 0cm 0cm 0cm">
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:#999999">IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Gregor Pillen<br>
Geschäftsführung: David Faller<br>
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">-----Original Message-----<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><b><span style="font-size:11.0pt">From</span></b><span style="font-size:11.0pt">: Thomas Bernecker <<a href="mailto:Thomas%20Bernecker%20%3cThomas.Bernecker@EMEA.NEC.COM%3e">Thomas.Bernecker@EMEA.NEC.COM</a>><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><b><span style="font-size:11.0pt">Reply-To</span></b><span style="font-size:11.0pt">: gpfsug main discussion list <<a href="mailto:gpfsug%20main%20discussion%20list%20%3cgpfsug-discuss@gpfsug.org%3e">gpfsug-discuss@gpfsug.org</a>><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><b><span style="font-size:11.0pt">To</span></b><span style="font-size:11.0pt">: gpfsug main discussion list <<a href="mailto:gpfsug%20main%20discussion%20list%20%3cgpfsug-discuss@gpfsug.org%3e">gpfsug-discuss@gpfsug.org</a>><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><b><span style="font-size:11.0pt">Subject</span></b><span style="font-size:11.0pt">: [EXTERNAL] Re: [gpfsug-discuss] Spectrum Scale, InfiniBand and QoS<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><b><span style="font-size:11.0pt">Date</span></b><span style="font-size:11.0pt">: 09/07/2023 11:02:02 AM<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">Hi Luis, Thanks for your quick response. I was referring to fabric QoS, not Spectrum Scale QoS (which I have some experience with). The reference to slide 10 of
said presentation says to configure the RDMA device as follows to use QoS verbsPorts<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerStart<o:p></o:p></span></p>
</div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;border-radius:4px">
<tbody>
<tr>
<td style="padding:12.0pt 0cm 12.0pt 0cm">
<table class="MsoNormalTable" border="1" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;background:#D0D8DC;border:none;border-top:solid #90A4AE 3.0pt">
<tbody>
<tr>
<td valign="top" style="border:none;padding:0cm 7.5pt 3.75pt 4.5pt">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left">
<tbody>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<p class="MsoNormal"><b><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">This Message Is From an External Sender
<o:p></o:p></span></b></p>
</td>
</tr>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:black">This message came from outside your organization.
<o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="right">
<tbody>
<tr>
<td style="padding:3.0pt 0cm 3.0pt 0cm">
<p class="MsoNormal"><span style="font-size:11.0pt"> <a href="https://us-phishalarm-ewt.proofpoint.com/EWT/v1/PjiDSg!1e-v7R5zRvmblYv7uGFl7pa6Y4IryZDRZDY_pFV_gcuWmVhOYY_MnKo0CryfRuaBuRmNkPYpLYCHVyCAO62RdiqmuBcGrQ1bbfxDJ3CzqkCGbKUj4BJ_KOcmBuYQq_Z4sy-kxOV4aF_bfA4U$" target="_blank"><strong><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;border:solid #666666 1.0pt;padding:6.0pt;font-weight:normal;text-decoration:none"> Report Suspicious </span></strong></a>
<o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerEnd<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Hi Luis,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks for your quick response. I was referring to fabric QoS, not Spectrum Scale QoS (which I have some experience with). The reference to slide 10 of said presentation says to configure the RDMA device as
follows to use QoS</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="FR" style="font-size:11.0pt">verbsPorts = <device>/<port>/<fabric>/<service level></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">− List of RDMA ports to be used</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">− <device> : RDMA device, required, e.g. mlx5_0, mlx5_1</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">− <port>: RDMA port on device, default 1, valid values are 1 or 2</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">− <fabric>: virtual fabric number, default 0, valid values are >= 0</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Only verbsPorts using a common <fabric> are connected</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">− <service level>: QoS level, default is 0, valid values defined in SM configuration</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The references to SM and verbsPorts being part of RDMA configuration indicates (to me) that this is fabric QoS …</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div name="divtagdefaultwrapper">
<div name="divtagdefaultwrapper">
<div>
<div name="divtagdefaultwrapper">
<div name="divtagdefaultwrapper">
<div>
<p class="MsoNormal"><span lang="DE" style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">--</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="DE" style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">Best regards / Mit freundlichem Gruß</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="DE" style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">Thomas Bernecker</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">------------------------------------------------------------------------------------------------------------------</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">Manager System Integration and Support, HPCE Division</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">Mobile: +49 (1522) 2851523, Fax: +49 (211) 5369-199, Home Office: +49 (38821) 65091</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">NEC Deutschland GmbH, Fritz-Vomfelde-Straße 14-16, 40547 Düsseldorf, Germany</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="DE" style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">Geschäftsführer: Christopher Richard Jackson – Handelsregister Düsseldorf HRB 57941</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:DE">-------------------------------------------------------------------------------------------------------------------</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:11.0pt">From:</span></b><span style="font-size:11.0pt"> gpfsug-discuss <<a href="mailto:gpfsug-discuss-bounces@gpfsug.org">gpfsug-discuss-bounces@gpfsug.org</a>><b>On Behalf Of
</b>Luis Bolinches<br>
<b>Sent:</b> Thursday, September 7, 2023 10:51 AM<br>
<b>To:</b> gpfsug main discussion list <<a href="mailto:gpfsug-discuss@gpfsug.org">gpfsug-discuss@gpfsug.org</a>><br>
<b>Subject:</b> Re: [gpfsug-discuss] Spectrum Scale, InfiniBand and QoS</span><o:p></o:p></p>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">Hi Assuming you are talking about GPFS QoS and not any fabric QoS. It would work with any supported fabric, ESS or not. It is a filesystem feature well above all
the fabrics and HW. Limits IOPS per class https: //www. ibm. com/docs/en/storage-scale/5. 1. 8?topic=reference-mmchqos-command</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerStart</span><o:p></o:p></p>
</div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;border-radius:4px">
<tbody>
<tr>
<td style="padding:12.0pt 0cm 12.0pt 0cm">
<table class="MsoNormalTable" border="1" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;background:#FADF9B;border:none;border-top:solid #F2B004 3.0pt">
<tbody>
<tr>
<td valign="top" style="border:none;padding:0cm 7.5pt 3.75pt 4.5pt">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left">
<tbody>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<p class="MsoNormal"><b><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">Be Careful With This Message</span></b><o:p></o:p></p>
</td>
</tr>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:black">The sender's identity could not be verified and someone may be impersonating the sender.</span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="right">
<tbody>
<tr>
<td style="padding:3.0pt 0cm 3.0pt 0cm">
<p class="MsoNormal"> <a href="https://us-phishalarm-ewt.proofpoint.com/EWT/v1/BQNorrFsuw!1ltCYpPUsLPrqvNt6ouhTp6NGkpGHjneRgeZQ4EL8X29-niUBGCBdKKkDls1qqoVuPo1KL8kYyJpf4yhW6ztYNlnYstlXTWSM5mLl3UqEvexW0whUlE-4P3Yj_DDkWnTV-cejP_cc5NNq29-2g$" target="_blank"><strong><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;border:solid #666666 1.0pt;padding:6.0pt;font-weight:normal;text-decoration:none"> Report Suspicious </span></strong></a> <o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerEnd</span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">Hi</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">Assuming you are talking about GPFS QoS and not any fabric QoS. It would work with any supported fabric, ESS or not. It is a filesystem feature well above all the fabrics and HW.
Limits IOPS per class</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><a href="https://urldefense.com/v3/__https:/www.ibm.com/docs/en/storage-scale/5.1.8?topic=reference-mmchqos-command__;!!BQNorrFsuw!gFOqdpq7kHEXBM75HosNGDnXXgDP4yTOdW3wwrdYQ3ocasUUvMstiTx-uZgEEhBFX35YBzRhcYEXbeHsK9UzisZCHvSjx_6fmA$">https://www.ibm.com/docs/en/storage-scale/5.1.8?topic=reference-mmchqos-command
[ibm.com] [ibm.com]</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">I recommend (depending on number of nodes and seconds) fine-stats so then you can visualize which PID on which client node is doing what. I believe I did something of that on SSUG
London 18 or 19</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">--</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">Ystävällisin terveisin/Regards/Saludos/Salutations/Salutacions</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">Luis Bolinches</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">Executive IT Specialist</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">IBM Storage ScaleServer (formerly ESS) developer</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">Phone: +358503112585</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">Ab IBM Finland Oy</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">Toinen linja 7</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">00530 Helsinki</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">Uusimaa - Finland</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">Visitors entrance: Siltasaarenkatu 22</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">"If you always give you will always have" -- Anonymous</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual"><a href="https://urldefense.com/v3/__https:/www.credly.com/users/luis-bolinches/badges__;!!BQNorrFsuw!gFOqdpq7kHEXBM75HosNGDnXXgDP4yTOdW3wwrdYQ3ocasUUvMstiTx-uZgEEhBFX35YBzRhcYEXbeHsK9UzisZCHvR17Vss7A$">https://www.credly.com/users/luis-bolinches/badges
[credly.com] [credly.com]</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt">From:</span></b><span lang="EN-US" style="font-size:11.0pt"> gpfsug-discuss <<a href="mailto:gpfsug-discuss-bounces@gpfsug.org">gpfsug-discuss-bounces@gpfsug.org</a>><b>On Behalf Of
</b>Thomas Bernecker<br>
<b>Sent:</b> Thursday, 7 September 2023 11.36<br>
<b>To:</b> <a href="mailto:gpfsug-discuss@gpfsug.org">gpfsug-discuss@gpfsug.org</a><br>
<b>Subject:</b> [EXTERNAL] [gpfsug-discuss] Spectrum Scale, InfiniBand and QoS</span><o:p></o:p></p>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">Dear all, we recently found this presentation from 2021: https: //www. spectrumscaleug. org/wp-content/uploads/2021/05/SSSD21DE-06-Improving-Spectrum-Scale-performance-using-RDMA. pdf
On page 10 it explains by setting the RDMA device in a specific </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerStart</span><o:p></o:p></p>
</div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;border-radius:4px">
<tbody>
<tr>
<td style="padding:12.0pt 0cm 12.0pt 0cm">
<table class="MsoNormalTable" border="1" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;background:#D0D8DC;border:none;border-top:solid #90A4AE 3.0pt">
<tbody>
<tr>
<td valign="top" style="border:none;padding:0cm 7.5pt 3.75pt 4.5pt">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left">
<tbody>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<p class="MsoNormal"><b><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">This Message Is From an External Sender</span></b><o:p></o:p></p>
</td>
</tr>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:black">This message came from outside your organization.</span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="right">
<tbody>
<tr>
<td style="padding:3.0pt 0cm 3.0pt 0cm">
<p class="MsoNormal"> <a href="https://us-phishalarm-ewt.proofpoint.com/EWT/v1/PjiDSg!1e-v7R5zRvmblYv7uGFl7ra7I0IqiHHRBBi-CjVQYUsssj6dhwnB7Lo2ZohM0He5_MhQUpUfgGfOHPXlTwIhxLUDAE1nbLv0ArY1Ygr3eS1cGzhvDzAy1DtrxLl5hHS5FHw6KsNKzpBBnc2d$" target="_blank"><strong><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black;border:solid #666666 1.0pt;padding:6.0pt;font-weight:normal;text-decoration:none"> Report Suspicious </span></strong></a> <o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerEnd</span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">Dear all,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual">we recently found this presentation from 2021:<a href="https://urldefense.com/v3/__https:/www.spectrumscaleug.org/wp-content/uploads/2021/05/SSSD21DE-06-Improving-Spectrum-Scale-performance-using-RDMA.pdf__;!!BQNorrFsuw!gFOqdpq7kHEXBM75HosNGDnXXgDP4yTOdW3wwrdYQ3ocasUUvMstiTx-uZgEEhBFX35YBzRhcYEXbeHsK9UzisZCHvREwAuGsg$">https://www.spectrumscaleug.org/wp-content/uploads/2021/05/SSSD21DE-06-Improving-Spectrum-Scale-performance-using-RDMA.pdf
[spectrumscaleug.org] [spectrumscaleug.org]</a> On page 10 it explains by setting the RDMA device in a specific way one could employ QoS with InfiniBand, which is something we would like to achieve as well. Has anyone a working environment using QoS with InfiniBand
outside of the ESS domain? If so, would you share your experience? Sorry for asking a rather broad question, but it seems that this is not well-known stuff …</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual"> </span><o:p></o:p></p>
<div name="divtagdefaultwrapper">
<div name="divtagdefaultwrapper">
<div>
<div name="divtagdefaultwrapper">
<div name="divtagdefaultwrapper">
<div>
<p class="MsoNormal"><span lang="DE" style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">--</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="DE" style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">Best regards / Mit freundlichem Gruß</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="DE" style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">Thomas Bernecker</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">------------------------------------------------------------------------------------------------------------------</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">Manager System Integration and Support, HPCE Division</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">Mobile: +49 (1522) 2851523, Fax: +49 (211) 5369-199, Home Office: +49 (38821) 65091</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">NEC Deutschland GmbH, Fritz-Vomfelde-Straße 14-16, 40547 Düsseldorf, Germany</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="DE" style="font-size:11.0pt;mso-ligatures:standardcontextual;mso-fareast-language:DE">Geschäftsführer: Christopher Richard Jackson – Handelsregister Düsseldorf HRB 57941</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:DE">-------------------------------------------------------------------------------------------------------------------</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-ligatures:standardcontextual"> </span><o:p></o:p></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Unless otherwise stated above:<br>
<br>
Oy IBM Finland Ab<br>
PL 265, 00101 Helsinki, Finland<br>
Business ID, Y-tunnus: 0195876-3<br>
Registered in Finland</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">_______________________________________________<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">gpfsug-discuss mailing list<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">gpfsug-discuss at gpfsug.org<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><a href="https://urldefense.com/v3/__http:/gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org__;!!BQNorrFsuw!gFOqdpq7kHEXBM75HosNGDnXXgDP4yTOdW3wwrdYQ3ocasUUvMstiTx-uZgEEhBFX35YBzRhcYEXbeHsK9UzisZCHvSjbs8rzg$">http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
[gpfsug.org]</a> <o:p></o:p></span></p>
</div>
</div>
</body>
</html>