[gpfsug-discuss] Anybody running GPFS over iSCSI?

[Jan-Frode Myklebust]

I’d be curious to hear if all these arguments against iSCSI shouldn’t also
apply to NSD protocol over TCP/IP?


-jf
man. 17. des. 2018 kl. 01:22 skrev Jonathan Buzzard <
jonathan.buzzard at strath.ac.uk>:

> On 13/12/2018 20:54, Buterbaugh, Kevin L wrote:
>
> [SNIP]
>
> >
> > Two things that I am already aware of are:  1) use jumbo frames, and 2)
> > run iSCSI over it’s own private network.  Other things I should be aware
> > of?!?
> >
>
> Yes, don't do it. Really do not do it unless you have datacenter
> Ethernet switches and adapters. Those are the ones required for FCoE.
> Basically unless you have per channel pause on your Ethernet fabric then
> performance will at some point all go to shit.
>
> So what happens is your NSD makes a whole bunch of requests to read
> blocks off the storage array. Requests are small, response is not. The
> response can overwhelms the Ethernet channel at which point performance
> falls through the floor. Now you might be lucky not to see this,
> especially if you have say have 10Gbps links from the storage and 40Gbps
> links to the NSD servers, but you are taking a gamble. Also the more
> storage arrays you have the more likely you are to see the problem.
>
> To fix this you have two options. The first is datacenter Ethernet with
> per channel pause. This option is expensive, probably in the same ball
> park as fibre channel. At least it was last time I looked, though this
> was some time ago now.
>
> The second option is dedicated links between the storage array and the
> NSD server. That is the cable goes directly between the storage array
> and the NSD server with no switches involved. This option is a
> maintenance nightmare.
>
> At he site where I did this, we had to go option two because I need to
> make it work, We ended up ripping it all out are replacing with FC.
>
> Personally I would see what price you can get DSS storage for, or use
> SAS arrays.
>
> Note iSCSI can in theory work, it's just the issue with GPFS scattering
> stuff to the winds over multiple storage arrays so your ethernet channel
> gets swamped and standard ethernet pauses all the upstream traffic. The
> vast majority of iSCSI use cases don't see this effect.
>
> There is a reason that to run FC over ethernet they had to turn ethernet
> lossless.
>
>
> JAB.
>
> --
> Jonathan A. Buzzard                         Tel: +44141-5483420
> HPC System Administrator, ARCHIE-WeSt.
> University of Strathclyde, John Anderson Building, Glasgow. G4 0NG
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20181217/a2c482fd/attachment-0002.htm>