[gpfsug-discuss] Anybody running GPFS over iSCSI?

[Jonathan Buzzard]

On 13/12/2018 20:54, Buterbaugh, Kevin L wrote:

[SNIP]

> 
> Two things that I am already aware of are:  1) use jumbo frames, and 2) 
> run iSCSI over it’s own private network.  Other things I should be aware 
> of?!?
> 

Yes, don't do it. Really do not do it unless you have datacenter 
Ethernet switches and adapters. Those are the ones required for FCoE. 
Basically unless you have per channel pause on your Ethernet fabric then 
performance will at some point all go to shit.

So what happens is your NSD makes a whole bunch of requests to read 
blocks off the storage array. Requests are small, response is not. The 
response can overwhelms the Ethernet channel at which point performance 
falls through the floor. Now you might be lucky not to see this, 
especially if you have say have 10Gbps links from the storage and 40Gbps 
links to the NSD servers, but you are taking a gamble. Also the more 
storage arrays you have the more likely you are to see the problem.

To fix this you have two options. The first is datacenter Ethernet with 
per channel pause. This option is expensive, probably in the same ball 
park as fibre channel. At least it was last time I looked, though this 
was some time ago now.

The second option is dedicated links between the storage array and the 
NSD server. That is the cable goes directly between the storage array 
and the NSD server with no switches involved. This option is a 
maintenance nightmare.

At he site where I did this, we had to go option two because I need to 
make it work, We ended up ripping it all out are replacing with FC.

Personally I would see what price you can get DSS storage for, or use 
SAS arrays.

Note iSCSI can in theory work, it's just the issue with GPFS scattering 
stuff to the winds over multiple storage arrays so your ethernet channel 
gets swamped and standard ethernet pauses all the upstream traffic. The 
vast majority of iSCSI use cases don't see this effect.

There is a reason that to run FC over ethernet they had to turn ethernet 
lossless.


JAB.

-- 
Jonathan A. Buzzard                         Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG