<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body dir="auto">
In my experience, most people just don’t need the bandwidth, unless you are working in the HPC area. Even then, in many cases. We have our system hooked up correctly, and can get something like 20 GB per second with multinode jobs. But almost nobody does that.
In reality, it prevents lots of small I/O from saturating the thing. I’m not saying that’s a good reason not to care about this, but it explains why it might not be obvious to non-HBC people.
<div><br>
</div>
<div>Back before we had robust monitoring (you know, get it running yesterday) to make sure that nodes didn’t come up without RDMA enabled, we ran for quite some time with it accidentally disabled without anyone reporting anything until a job with bug I/O finally
came along and clobbered everything. I would’ve thought that would have been immediately apparent. Not so, at least in our environment.<br>
<br>
<div dir="ltr">Sent from my iPhone</div>
<div dir="ltr"><br>
<blockquote type="cite">On Jun 5, 2023, at 13:17, Alec <anacreo@gmail.com> wrote:<br>
<br>
</blockquote>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="auto">Sadly no I'm not kidding... Network engineers tend to be more focused on availability and general needs not special needs of a high speed data environment and so I give them a pass.
<div dir="auto"><br>
</div>
<div dir="auto">As I used to say SAN engineers were Unix engineers who couldn't do Unix.. so they are what they are.</div>
<div dir="auto"><br>
</div>
<div dir="auto">I can't tell you how many times I've seen millions of dollars or hardware under perming to a tiny fraction because someone didn't cable it with enough cables to do the job. Like constrained ISL or something... Or they'll just discount the
two storage ports that are RED with saturation and say 99% of ports are green... So no problem here.</div>
<div dir="auto"><br>
</div>
<div dir="auto">Alec</div>
<div dir="auto"><br>
</div>
<div dir="auto">Alec</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Jun 5, 2023, 10:05 AM Jonathan Buzzard <<a href="mailto:jonathan.buzzard@strath.ac.uk">jonathan.buzzard@strath.ac.uk</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On 05/06/2023 17:41, Alec wrote:<br>
><br>
> Many network storage engineers simply don't understand bandwidth (or the <br>
> importance of port groups)... With an ESS you are talking GIGA*BYTES* <br>
> per second and storage and networking architects simply see 10Gbe and <br>
> assume that's good enough. A 10Gbe connection can do about 1.4GB/s.. <br>
> 100Gbe can do 12.5GB/s. To the wrong engineer you can explain this <br>
> until you're blue in the face and they won't get it. You need to divide <br>
> Gbe by 8 to get about the GB/s throughput. Explain to them that a USB-C <br>
> interface is capable of 10Gbe... So you're throttling millions of <br>
> dollars in technology to the speed of a consumer grade USB-C interface. <br>
> When infact an ESS can drive at least 2x100Gbe to saturation.<br>
> <br>
> I don't have an ESS but a classic SAN array and Spectrum Scale and I can <br>
> saturate 32*8Gbe fiber connections.<br>
> <br>
<br>
Your kidding right? That's basic competency for the job!!!<br>
<br>
<br>
JAB.<br>
<br>
-- <br>
Jonathan A. Buzzard Tel: +44141-5483420<br>
HPC System Administrator, ARCHIE-WeSt.<br>
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG<br>
<br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://gpfsug.org" rel="noreferrer noreferrer" target="_blank">
gpfsug.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org" rel="noreferrer noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org</a><br>
</blockquote>
</div>
<span>_______________________________________________</span><br>
<span>gpfsug-discuss mailing list</span><br>
<span>gpfsug-discuss at gpfsug.org</span><br>
<span>http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org</span><br>
</div>
</blockquote>
</div>
</body>
</html>