<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css">BODY {
FONT-FAMILY: arial
}
</style>
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
tt
{mso-style-priority:99;
font-family:"Courier New";}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="DE" link="blue" vlink="purple" style="word-wrap:break-word">
<p></p>
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt;font-family:"Arial",sans-serif;mso-fareast-language:EN-US">Hallo Walter,<br>
we had many experiences now to change our Storage-Systems in our Backup-Environment to RDMA-IB with HDR and EDR Connections. What we see now (came from a 16Gbit FC Infrastructure) we enhance our throuhput from 7 GB/s to 30 GB/s. The main reason are the elimination
of the driver-layers in the client-systems and make a Buffer to Buffer communication because of RDMA. The latency reduction are significant.<br>
Regards Renar.<br>
We use now ESS3k and ESS5k systems with 6.1.1.2-Code level.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div></div>
</div>
<p></p>
<p><font style="font-family:Arial;">Renar</font> <font style="font-family:Arial;">Grunenberg</font><br>
<font style="font-family:Arial;">Abteilung Informatik - Betrieb</font><br>
<br>
<font style="font-family:Arial;">HUK-COBURG</font><br>
<font style="font-family:Arial;">Bahnhofsplatz</font><br>
<font style="font-family:Arial;">96444</font> <font style="font-family:Arial;">Coburg</font><br>
<table class="a55a6537-94c9-4b47-ba6f-ae8df6fb2ba6Table" style="font-family:Arial;">
<tbody>
<tr>
<td><font style="font-family:Arial;"><font style="font-family:Arial;"><font style="font-family:Arial;">Telefon:</font></font></font></td>
<td><font style="font-family:Arial;"><font style="font-family:Arial;"><font style="font-family:Arial;">09561 96-44110</font></font></font></td>
</tr>
<tr>
<td><font style="font-family:Arial;"><font style="font-family:Arial;"><font style="font-family:Arial;">Telefax:</font></font></font></td>
<td><font style="font-family:Arial;"><font style="font-family:Arial;"><font style="font-family:Arial;">09561 96-44104</font></font></font></td>
</tr>
<tr>
<td><font style="font-family:Arial;"><font style="font-family:Arial;"><font style="font-family:Arial;">E-Mail:</font></font></font></td>
<td><font style="font-family:Arial;"><font style="font-family:Arial;">Renar.Grunenberg@huk-coburg.de</font></font></td>
</tr>
<tr>
<td><font style="font-family:Arial;"><font style="font-family:Arial;"><font style="font-family:Arial;">Internet:</font></font></font></td>
<td><font style="font-family:Arial;"><font style="font-family:Arial;"><font style="font-family:Arial;">www.huk.de</font></font></font></td>
</tr>
</tbody>
</table>
</p>
<hr id="HR1">
<font face="Arial"><font size="1">HUK-COBURG Haftpflicht-Unterstützungs-Kasse kraftfahrender Beamter Deutschlands a. G. in Coburg<br>
Reg.-Gericht Coburg HRB 100; St.-Nr. 9212/101/00021<br>
Sitz der Gesellschaft: Bahnhofsplatz, 96444 Coburg<br>
Vorsitzender des Aufsichtsrats: Prof. Dr. Heinrich R. Schradin.<br>
Vorstand: Klaus-Jürgen Heitmann (Sprecher), Stefan Gronbach, Dr. Hans Olav Herøy, Dr. Jörg Rheinländer, Thomas Sehn, Daniel Thomas.<br>
</font>
<hr id="HR1">
</font><font size="1" face="Arial">Diese Nachricht enthält vertrauliche und/oder rechtlich geschützte Informationen.<br>
Wenn Sie nicht der richtige Adressat sind oder diese Nachricht irrtümlich erhalten haben,<br>
informieren Sie bitte sofort den Absender und vernichten Sie diese Nachricht.<br>
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Nachricht ist nicht gestattet.<br>
<br>
This information may contain confidential and/or privileged information.<br>
If you are not the intended recipient (or have received this information in error) please notify the<br>
sender immediately and destroy this information.<br>
Any unauthorized copying, disclosure or distribution of the material in this information is strictly forbidden.<br>
</font>
<hr id="HR1">
<p></p>
<p></p>
<p></p>
<div class="WordSection1">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b>Von:</b> gpfsug-discuss-bounces@spectrumscale.org <gpfsug-discuss-bounces@spectrumscale.org>
<b>Im Auftrag von </b>Walter Sklenka<br>
<b>Gesendet:</b> Freitag, 10. Dezember 2021 11:17<br>
<b>An:</b> gpfsug-discuss@spectrumscale.org<br>
<b>Betreff:</b> Re: [gpfsug-discuss] WAS: alternative path; Now: RDMA<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Hello Douglas!<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">May I ask a basic question regarding
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">GPUdirect Storage or all local attached storage like NVME disks. Do you think it outerperforms “classical” shared storagesystems which are attached via FC connected to NSD servers
HDR attached? <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">With FC you have also bounce copies and more delay , isn´t it?
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif">There are solutions around which work with local NVME disks building some protection level with Raid (or duplication) . I am curious if it would be a better approach
than shared storage which has it´s limitation (cost intensive scale out, extra infrstructure, max 64Gb at this time … )<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Best regards<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Walter
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">
gpfsug-discuss-bounces@spectrumscale.org</a> <<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">gpfsug-discuss-bounces@spectrumscale.org</a>>
<b>On Behalf Of </b>Douglas O'flaherty<br>
<b>Sent:</b> Freitag, 10. Dezember 2021 05:24<br>
<b>To:</b> <a href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.org</a><br>
<b>Subject:</b> Re: [gpfsug-discuss] WAS: alternative path; Now: RDMA<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span lang="DE-AT"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="DE-AT" style="font-size:10.0pt;font-family:"Arial",sans-serif">Jonathan:</span><span lang="DE-AT"><br>
<br>
</span><span lang="DE-AT" style="font-size:10.0pt;font-family:"Arial",sans-serif">You posed a reasonable question, which was "when is RDMA worth the hassle?" I agree with part of your premises, which is that it only matters when the bottleneck isn't somewhere
else. With a parallel file system, like Scale/GPFS, the absolute performance bottleneck is not the throughput of a single drive. In a majority of Scale/GPFS clusters the network data path is the performance limitation. If they deploy HDR or 100/200/400Gbps
Ethernet... At that point, the buffer copy time inside the server matters. </span>
<span lang="DE-AT"><br>
<br>
</span><span lang="DE-AT" style="font-size:10.0pt;font-family:"Arial",sans-serif">When the device is an accelerator, like a GPU, the benefit of RDMA (GDS) is easily demonstrated because it eliminates the bounce copy through the system memory. In our NVIDIA
DGX A100 server testing testing we were able to get around 2x the per system throughput by using RDMA direct to GPU (GUP Direct Storage). (Tested on 2 DGX system with 4x HDR links per storage node.)
</span><span lang="DE-AT"><br>
<br>
</span><span lang="DE-AT" style="font-size:10.0pt;font-family:"Arial",sans-serif">However, your question remains. Synthetic benchmarks are good indicators of technical benefit, but do your users and applications need that extra performance?
</span><span lang="DE-AT"><br>
<br>
</span><span lang="DE-AT" style="font-size:10.0pt;font-family:"Arial",sans-serif">These are probably only a handful of codes in organizations that need this. However, they are high-value use cases. We have client applications that either read a lot of data
semi-randomly and not-cached - think mini-Epics for scaling ML training. Or, demand lowest response time, like production inference on voice recognition and NLP.
</span><span lang="DE-AT"><br>
<br>
</span><span lang="DE-AT" style="font-size:10.0pt;font-family:"Arial",sans-serif">If anyone has use cases for GPU accelerated codes with truly demanding data needs, please reach out directly. We are looking for more use cases to characterize the benefit for
a new paper. f you can provide some code examples, we can help test if RDMA direct to GPU (GPUdirect Storage) is a benefit.
</span><span lang="DE-AT"><br>
<br>
</span><span lang="DE-AT" style="font-size:10.0pt;font-family:"Arial",sans-serif">Thanks,</span><span lang="DE-AT"><br>
<br>
</span><span lang="DE-AT" style="font-size:10.0pt;font-family:"Arial",sans-serif">doug</span><span lang="DE-AT"><br>
<br>
</span><span lang="DE-AT" style="font-size:10.0pt;font-family:"Arial",sans-serif">Douglas O'Flaherty<br>
<a href="mailto:douglasof@us.ibm.com">douglasof@us.ibm.com</a></span><span lang="DE-AT"><br>
<br>
<br>
<br>
</span><span lang="DE-AT" style="font-size:10.0pt;font-family:"Courier New""><br>
<br>
</span><span lang="DE-AT" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:purple"><br>
----- Message from Jonathan Buzzard <<a href="mailto:jonathan.buzzard@strath.ac.uk">jonathan.buzzard@strath.ac.uk</a>> on Fri, 10 Dec 2021 00:27:23 +0000 -----</span><span lang="DE-AT"><o:p></o:p></span></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="513" style="width:384.75pt;border-collapse:collapse">
<tbody>
<tr style="height:6.0pt">
<td width="55" style="width:41.25pt;padding:.75pt .75pt .75pt .75pt;height:6.0pt">
<p class="MsoNormal" align="right" style="text-align:right"><b><span style="font-size:12.0pt">To:</span></b><o:p></o:p></p>
</td>
<td width="454" style="width:340.5pt;padding:.75pt .75pt .75pt .75pt;height:6.0pt">
<p class="MsoNormal"><span style="font-size:12.0pt"><a href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.org</a></span><o:p></o:p></p>
</td>
</tr>
<tr style="height:6.0pt">
<td width="55" style="width:41.25pt;padding:.75pt .75pt .75pt .75pt;height:6.0pt">
<p class="MsoNormal" align="right" style="text-align:right"><b><span style="font-size:12.0pt">Subject:</span></b><o:p></o:p></p>
</td>
<td width="454" style="width:340.5pt;padding:.75pt .75pt .75pt .75pt;height:6.0pt">
<p class="MsoNormal"><span style="font-size:12.0pt">Re: [gpfsug-discuss] </span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="DE-AT"><br>
</span><tt><span lang="DE-AT" style="font-size:10.0pt">On 09/12/2021 16:04, Douglas O'flaherty wrote:</span></tt><span lang="DE-AT" style="font-size:10.0pt;font-family:"Courier New""><br>
<tt>> </tt><br>
<tt>> Though not directly about your design, our work with NVIDIA on GPUdirect </tt>
<br>
<tt>> Storage and SuperPOD has shown how sensitive RDMA (IB & RoCE) to both </tt>
<br>
<tt>> MOFED and Firmware version compatibility can be.</tt><br>
<tt>> </tt><br>
<tt>> I would suggest anyone debugging RDMA issues should look at those closely.</tt><br>
<tt>> </tt><br>
<tt>May I ask what are the alleged benefits of using RDMA in GPFS?</tt><br>
<br>
<tt>I can see there would be lower latency over a plain IP Ethernet or IPoIB </tt>
<br>
<tt>solution but surely disk latency is going to swamp that?</tt><br>
<br>
<tt>I guess SSD drives might change that calculation but I have never seen </tt><br>
<tt>proper benchmarks comparing the two, or even better yet all four </tt><br>
<tt>connection options.</tt><br>
<br>
<tt>Just seems a lot of complexity and fragility for very little gain to me.</tt><br>
<br>
<br>
<tt>JAB.</tt><br>
<br>
<tt>-- </tt><br>
<tt>Jonathan A. Buzzard Tel: +44141-5483420</tt><br>
<tt>HPC System Administrator, ARCHIE-WeSt.</tt><br>
<tt>University of Strathclyde, John Anderson Building, Glasgow. G4 0NG</tt><br>
<br>
</span><span lang="DE-AT"><br>
</span><span lang="DE-AT" style="font-size:10.0pt;font-family:"Arial",sans-serif"> </span><span lang="DE-AT"><br>
</span><span lang="DE-AT" style="font-size:10.0pt;font-family:"Arial",sans-serif"> </span><span lang="DE-AT"><br>
</span><span lang="DE-AT" style="font-size:10.0pt;font-family:"Arial",sans-serif">----- Original message -----<br>
From: "Jonathan Buzzard" <<a href="mailto:jonathan.buzzard@strath.ac.uk">jonathan.buzzard@strath.ac.uk</a>><br>
Sent by: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">gpfsug-discuss-bounces@spectrumscale.org</a><br>
To: <a href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.org</a><br>
Cc:<br>
Subject: [EXTERNAL] Re: [gpfsug-discuss] alternate path between ESS Servers for Datamigration<br>
Date: Fri, Dec 10, 2021 10:27<br>
</span><span lang="DE-AT"><br>
</span><tt><span lang="DE-AT" style="font-size:10.0pt">On 09/12/2021 16:04, Douglas O'flaherty wrote:</span></tt><span lang="DE-AT" style="font-size:10.0pt;font-family:"Courier New""><br>
<tt>></tt><br>
<tt>> Though not directly about your design, our work with NVIDIA on GPUdirect</tt><br>
<tt>> Storage and SuperPOD has shown how sensitive RDMA (IB & RoCE) to both</tt><br>
<tt>> MOFED and Firmware version compatibility can be.</tt><br>
<tt>></tt><br>
<tt>> I would suggest anyone debugging RDMA issues should look at those closely.</tt><br>
<tt>></tt><br>
<tt>May I ask what are the alleged benefits of using RDMA in GPFS?</tt><br>
<br>
<tt>I can see there would be lower latency over a plain IP Ethernet or IPoIB</tt><br>
<tt>solution but surely disk latency is going to swamp that?</tt><br>
<br>
<tt>I guess SSD drives might change that calculation but I have never seen</tt><br>
<tt>proper benchmarks comparing the two, or even better yet all four</tt><br>
<tt>connection options.</tt><br>
<br>
<tt>Just seems a lot of complexity and fragility for very little gain to me.</tt><br>
<br>
<br>
<tt>JAB.</tt><br>
<br>
<tt>--</tt><br>
<tt>Jonathan A. Buzzard Tel: +44141-5483420</tt><br>
<tt>HPC System Administrator, ARCHIE-WeSt.</tt><br>
<tt>University of Strathclyde, John Anderson Building, Glasgow. G4 0NG</tt><br>
<tt>_______________________________________________</tt><br>
<tt>gpfsug-discuss mailing list</tt><br>
<tt>gpfsug-discuss at spectrumscale.org</tt><u><span style="color:blue"><br>
</span></u></span><span lang="DE-AT"><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><tt><span style="font-size:10.0pt">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></tt></a><br>
</span><span lang="DE-AT" style="font-size:10.0pt;font-family:"Arial",sans-serif"> </span><span lang="DE-AT"><br>
<br>
<br>
<o:p></o:p></span></p>
</div>
<p></p>
<p></p>
<p></p>
<p></p>
<p></p>
<p></p>
<p></p>
<p></p>
<p></p>
<p></p>
</body>
</html>