<div dir="ltr">I feel the need to respond here... I see many responses on this User Group forum that are dismissive of the fringe / extreme use cases and of the "what do you need that for '' mindset. The thing is that Spectrum Scale is for the extreme, just take the word "Parallel" in the old moniker that was already an extreme use case.<div><br></div><div>If you have a standard workload, then sure most of the complex features of the file system are toys, but many of us DO have extreme workloads where shaking out every ounce of performance is a worthwhile and financially sound endeavor. It is also because of the efforts of those of us living on the cusp of technology that these technologies become mainstream and no-longer extreme.</div><div><br></div><div>I have an AIX LPAR that traverses more than 300TB+ of data a day on a Spectrum Scale file system, it is fully virtualized, and handles a million files. If that performance level drops, regulatory reports will be late, business decisions won't be current. However, the systems of today and the future have to traverse this much data and if they are slow then they can't keep up with real-time data feeds. So the difference between an RDMA disk IO vs a non RDMA disk IO could possibly mean what level of analytics are done to perform real time fraud prevention. Or at what cost, today many systems achieve this by keeping everything in memory in HUGE farms.. Being able to perform data operations at 30GB/s means you can traverse ALL of the census bureau data for all time from the US Govt in about 2 seconds... that's a pretty substantial capability that moves the bar forward in what we can do from a technology perspective.</div><div><br></div><div>I just did a technology garage with IBM where we were able to achieve 1.5TB/writes on an encrypted ESS off of a single VMWare Host and 4 VM's over IP... That's over 2PB of data writes a day on a single VM server. Being able to demonstrate that there are production virtualized environments capable of this type of capacity, helps to show where the point of engineering a proper storage architecture outweighs the benefits of just throwing more GPU compute farms at the problem with ever dithering disk I/O. It also helps to demonstrate how a virtual storage optimized farm could be leveraged to host many in-memory or data analytic heavy workloads in a shared configuration.</div><div><br></div><div>Douglas's response is the right one, how much IO does the application / environment need, it's nice to see Spectrum Scale have the flexibility to deliver. I'm pretty confident that if I can't deliver the required I/O performance on Spectrum Scale, nobody else can on any other storage platform within reasonable limits.</div><div><br></div><div>Alec Effrat</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Dec 9, 2021 at 8:24 PM Douglas O'flaherty <<a href="mailto:douglasof@us.ibm.com">douglasof@us.ibm.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span style="font-size:10pt;font-family:sans-serif">Jonathan:</span><br><br><span style="font-size:10pt;font-family:sans-serif">You posed a reasonable
question, which was "when is RDMA worth the hassle?" I
agree with part of your premises, which is that it only matters when the
bottleneck isn't somewhere else. With a parallel file system, like Scale/GPFS,
the absolute performance bottleneck is not the throughput of a single drive.
In a majority of Scale/GPFS clusters the network data path is the performance
limitation. If they deploy HDR or 100/200/400Gbps Ethernet... At
that point, the buffer copy time inside the server matters. </span><br><br><span style="font-size:10pt;font-family:sans-serif">When the device
is an accelerator, like a GPU, the benefit of RDMA (GDS) is easily demonstrated
because it eliminates the bounce copy through the system memory. In our
NVIDIA DGX A100 server testing testing we were able to get around 2x the
per system throughput by using RDMA direct to GPU (GUP Direct Storage).
(Tested on 2 DGX system with 4x HDR links per storage node.) </span><br><br><span style="font-size:10pt;font-family:sans-serif">However, your
question remains. Synthetic benchmarks are good indicators of technical
benefit, but do your users and applications need that extra performance?
</span><br><br><span style="font-size:10pt;font-family:sans-serif">These are probably
only a handful of codes in organizations that need this. However, they
are high-value use cases. We have client applications that either read
a lot of data semi-randomly and not-cached - think mini-Epics for scaling
ML training. Or, demand lowest response time, like production inference
on voice recognition and NLP. </span><br><br><span style="font-size:10pt;font-family:sans-serif">If anyone has
use cases for GPU accelerated codes with truly demanding data needs, please
reach out directly. We are looking for more use cases to characterize the
benefit for a new paper. f you can provide some code examples, we can help
test if RDMA direct to GPU (GPUdirect Storage) is a benefit. </span><br><br><span style="font-size:10pt;font-family:sans-serif">Thanks,</span><br><br><span style="font-size:10pt;font-family:sans-serif">doug</span><br><br><span style="font-size:10pt;font-family:sans-serif">Douglas O'Flaherty<br><a href="mailto:douglasof@us.ibm.com" target="_blank">douglasof@us.ibm.com</a></span><br><br><br><br><tt><span style="font-size:10pt"><br><br></span></tt><span style="font-size:10pt;color:rgb(128,0,128);font-family:sans-serif"><br>----- Message from Jonathan Buzzard <<a href="mailto:jonathan.buzzard@strath.ac.uk" target="_blank">jonathan.buzzard@strath.ac.uk</a>>
on Fri, 10 Dec 2021 00:27:23 +0000 -----</span><table width="513" style="border-collapse:collapse"><tbody><tr height="8"><td width="55" style="border-style:none;border-color:rgb(0,0,0);border-width:0px;padding:1px"><div align="right"><span style="font-size:12pt"><b>To:</b></span></div></td><td width="454" style="border-style:none;border-color:rgb(0,0,0);border-width:0px;padding:1px"><span style="font-size:12pt"><a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a></span></td></tr><tr height="8"><td width="55" style="border-style:none;border-color:rgb(0,0,0);border-width:0px;padding:1px"><div align="right"><span style="font-size:12pt"><b>Subject:</b></span></div></td><td width="454" style="border-style:none;border-color:rgb(0,0,0);border-width:0px;padding:1px"><span style="font-size:12pt">Re:
[gpfsug-discuss] </span></td></tr></tbody></table><br><tt><span style="font-size:10pt">On 09/12/2021 16:04, Douglas O'flaherty
wrote:<br>> <br>> Though not directly about your design, our work with NVIDIA on GPUdirect
<br>> Storage and SuperPOD has shown how sensitive RDMA (IB & RoCE)
to both <br>> MOFED and Firmware version compatibility can be.<br>> <br>> I would suggest anyone debugging RDMA issues should look at those
closely.<br>> <br>May I ask what are the alleged benefits of using RDMA in GPFS?<br><br>I can see there would be lower latency over a plain IP Ethernet or IPoIB
<br>solution but surely disk latency is going to swamp that?<br><br>I guess SSD drives might change that calculation but I have never seen
<br>proper benchmarks comparing the two, or even better yet all four <br>connection options.<br><br>Just seems a lot of complexity and fragility for very little gain to me.<br><br><br>JAB.<br><br>-- <br>Jonathan A. Buzzard
Tel: +44141-5483420<br>HPC System Administrator, ARCHIE-WeSt.<br>University of Strathclyde, John Anderson Building, Glasgow. G4 0NG<br><br></span></tt><br><span style="font-size:10pt;font-family:Arial"> </span><br><span style="font-size:10pt;font-family:Arial"> </span><br><span style="font-size:10pt;font-family:Arial">----- Original message
-----<br>From: "Jonathan Buzzard" <<a href="mailto:jonathan.buzzard@strath.ac.uk" target="_blank">jonathan.buzzard@strath.ac.uk</a>><br>Sent by: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a><br>To: <a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a><br>Cc:<br>Subject: [EXTERNAL] Re: [gpfsug-discuss] alternate path between ESS Servers
for Datamigration<br>Date: Fri, Dec 10, 2021 10:27<br> </span><br><tt><span style="font-size:10pt">On 09/12/2021 16:04, Douglas O'flaherty
wrote:<br>><br>> Though not directly about your design, our work with NVIDIA on GPUdirect<br>> Storage and SuperPOD has shown how sensitive RDMA (IB & RoCE)
to both<br>> MOFED and Firmware version compatibility can be.<br>><br>> I would suggest anyone debugging RDMA issues should look at those
closely.<br>><br>May I ask what are the alleged benefits of using RDMA in GPFS?<br><br>I can see there would be lower latency over a plain IP Ethernet or IPoIB<br>solution but surely disk latency is going to swamp that?<br><br>I guess SSD drives might change that calculation but I have never seen<br>proper benchmarks comparing the two, or even better yet all four<br>connection options.<br><br>Just seems a lot of complexity and fragility for very little gain to me.<br><br><br>JAB.<br><br>--<br>Jonathan A. Buzzard
Tel: +44141-5483420<br>HPC System Administrator, ARCHIE-WeSt.<br>University of Strathclyde, John Anderson Building, Glasgow. G4 0NG<br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a></span></tt><tt><span style="font-size:10pt;color:blue"><u><br></u></span></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><tt><span style="font-size:10pt;color:blue"><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></span></tt></a><tt><span style="font-size:10pt"></span></tt><br><span style="font-size:10pt;font-family:Arial"> </span><br><br><br><br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
</blockquote></div>