<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div dir="ltr" >JF</div>
<div dir="ltr" > </div>
<div dir="ltr" >From my experience - NSD over 10GB doesn't translate into good performance, regardless of how good your back end storage architecture is, so I guess it depends on what your trying to achieve.</div>
<div dir="ltr" > </div>
<div dir="ltr" >if your trying to achieve a "we build a cluster and presented a filesystem" then yes NSD over Ethernet is probably fine, if your aiming to deliver a guaranteed performance profile of 10's of GB/s then Ethernet is probably not the network architecture of choice. </div>
<div dir="ltr" > </div>
<div dir="ltr" >Just in the same way that if you want a low latency block storage access environment you won't choose ISCSI you will use FC or IB access.</div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<div dir="ltr" >in reference to the original post, I would be curious to know how many Switches / how many ports were proposed in the 16GB FC environment and did the person who proposed it have a good understanding of the actual requirements ? I think at least in Australia you would have a good chance of getting a pair of switches to connect 2 NSD servers to a single FC storage array for far less than that.</div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<div dir="ltr" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div dir="ltr" style="margin-top: 20px;" ><div style="font-size: 12pt; font-weight: bold; font-family: sans-serif; color: #7C7C5F;" >Andrew Beattie</div>
<div style="font-size: 10pt; font-weight: bold; font-family: sans-serif;" >Software Defined Storage - IT Specialist</div>
<div style="font-size: 8pt; font-family: sans-serif; margin-top: 10px;" ><div><span style="font-weight: bold; color: #336699;" >Phone: </span>614-2133-7927</div>
<div><span style="font-weight: bold; color: #336699;" >E-mail: </span><a href="mailto:abeattie@au1.ibm.com" style="color: #555">abeattie@au1.ibm.com</a></div></div></div></div></div></div></div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Original message -----<br>From: Jan-Frode Myklebust <janfrode@tanso.net><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Cc:<br>Subject: Re: [gpfsug-discuss] Anybody running GPFS over iSCSI?<br>Date: Mon, Dec 17, 2018 5:50 PM<br> <br>I’d be curious to hear if all these arguments against iSCSI shouldn’t also apply to NSD protocol over TCP/IP?<br><br><br>-jf
<div><div dir="ltr" >man. 17. des. 2018 kl. 01:22 skrev Jonathan Buzzard <<a href="mailto:jonathan.buzzard@strath.ac.uk" target="_blank">jonathan.buzzard@strath.ac.uk</a>>:</div>
<blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex" >On 13/12/2018 20:54, Buterbaugh, Kevin L wrote:<br><br>[SNIP]<br><br>><br>> Two things that I am already aware of are: 1) use jumbo frames, and 2)<br>> run iSCSI over it’s own private network. Other things I should be aware<br>> of?!?<br>><br><br>Yes, don't do it. Really do not do it unless you have datacenter<br>Ethernet switches and adapters. Those are the ones required for FCoE.<br>Basically unless you have per channel pause on your Ethernet fabric then<br>performance will at some point all go to shit.<br><br>So what happens is your NSD makes a whole bunch of requests to read<br>blocks off the storage array. Requests are small, response is not. The<br>response can overwhelms the Ethernet channel at which point performance<br>falls through the floor. Now you might be lucky not to see this,<br>especially if you have say have 10Gbps links from the storage and 40Gbps<br>links to the NSD servers, but you are taking a gamble. Also the more<br>storage arrays you have the more likely you are to see the problem.<br><br>To fix this you have two options. The first is datacenter Ethernet with<br>per channel pause. This option is expensive, probably in the same ball<br>park as fibre channel. At least it was last time I looked, though this<br>was some time ago now.<br><br>The second option is dedicated links between the storage array and the<br>NSD server. That is the cable goes directly between the storage array<br>and the NSD server with no switches involved. This option is a<br>maintenance nightmare.<br><br>At he site where I did this, we had to go option two because I need to<br>make it work, We ended up ripping it all out are replacing with FC.<br><br>Personally I would see what price you can get DSS storage for, or use<br>SAS arrays.<br><br>Note iSCSI can in theory work, it's just the issue with GPFS scattering<br>stuff to the winds over multiple storage arrays so your ethernet channel<br>gets swamped and standard ethernet pauses all the upstream traffic. The<br>vast majority of iSCSI use cases don't see this effect.<br><br>There is a reason that to run FC over ethernet they had to turn ethernet<br>lossless.<br><br><br>JAB.<br><br>--<br>Jonathan A. Buzzard Tel: +44141-5483420<br>HPC System Administrator, ARCHIE-WeSt.<br>University of Strathclyde, John Anderson Building, Glasgow. G4 0NG<br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></blockquote></div>
<div><font size="2" face="Default Monospace,Courier New,Courier,monospace" >_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></div></blockquote>
<div dir="ltr" > </div></div><BR>