[gpfsug-discuss] IB routers in ESS configuration + 3 different subnets - valid config?

[Jake Carroll]

Hi.

I am just in the process of sanity-checking a potential future configuration.

Let's say I have an ESS 5000 and an ESS 3000 placed on the data centre floor to form the basis of a new scratch array.

Let's then suppose that I have three existing supercomputers in that same location. Each of those supercomputers has a separate IB subnet and their networks are unrelated to each other, IB-wise.

My understanding is that it is valid and possible to use MLNX EDR IB *routers* in order to be able to transport NSD communications back and forth across those separate subnets, back to the ESS (which lives on its own unique subnet). So at this point, I've got four unique subnets - one for the ESS, one for each super. As I understand it, there is an upper limit of *SIX* unique subnets on those EDR IB routers.

As I understand it - for IPoIB transport, I'd also need some "gateway" boxes more or less - essentially some decent servers which I put EDR/HDR cards in as dog legs that act as an IPoIB gateway interface to each subnet.

I appreciate that there is devil in the detail - but what I'm asking is if it is valid to "route" NSD with IB Routers (not switches) this way to separate subnets.

Colleagues at IBM have all said "yeah....should work....we've not done it....but should be fine?"

Colleagues at Mellanox (uhhh...nvidia...) say "Yes, this is valid and does exactly as the IB Router should and there is nothing unusual about this".

If someone has experience doing this or could call out any oddity/weirdness/gotchas, I'd be very appreciative. I'm fairly sure this is all very low risk - but given nobody locally could tell me "Yeah, all certified and valid!" I'd like the wisdom of the wider crowd.

Thank you.

--jc

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20201125/f088aa8d/attachment-0001.htm>