[gpfsug-discuss] storage-based replication for Spectrum Scale

valdis.kletnieks at vt.edu valdis.kletnieks at vt.edu
Wed Jan 24 16:31:26 GMT 2018


On Tue, 23 Jan 2018 20:39:52 -0500, Harold Morales said:

> - site A simple three node cluster configured  (AIX nodes). two failure
> groups. two filesystems. No quota, no ILM.
> - Site B the same cluster config as in site A configured (AIX nodes) same
> hdisk device names used as in site A for each and every NSDs
> - NO TCPIP connection between those two sites. Same ip addresses available
> locally on each one. and same non-scale based filesystems. Same scale
> version installed. (esentially site B is a copy from site A).

You're going to have a *really* bad time with semantic stability at site B,
as while it's reading the replica from site A, it will continually be reading metadata
and data blocks that it didn't write there. (Thought experiment - what happens
when site B thinks there's 3 files in a directory, and a replication block from A
changes that to 2 without B's knowledge?)

Be ready to be dealing with filesystem issues at B - it's *always* bad karma
to do writes to a filesystem that the operating system doesn't know about....

If B is a *cold* backup site, this should work just fine and you can ignore all
this. But if B has the filesystem mounted while A is making changes, it *will*
go pear shaped on you.

> People here don't want a stretch cluster (without any apparent reason, which
> talks very bad about them).

For what it's worth, we're running a stretch cluster with about 95 cable miles
separating the two halves.  Biggest problem we have is that a network hiccup
can cause the cluster to partition and down the NSD's at the remote site (yes,
it's designed that way - 5 nodes at each site, and on a partition the main site
has enough quorum nodes to stay up, but the remote site doesn't.  So it downs
the NSDs' there.  Just means a 'mmchdisk start -a' once the cluster recovers
all the nodes at the far end (which is a lot less painful than the alternative, which
is a split-brain situation).

(And yeah, we 're working on getting reliable alternate network connections.
Just a bit hard to find another 10G link to light up that isn't fate-sharing
*and* plays nice with the rest of our BGP routing and can be fixed to have
failover fast enough for GPFS to not notice the blink. We're in the part of
Virginia that doesn't have dark fiber all over the place.)

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 486 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20180124/65bebd91/attachment-0002.sig>


More information about the gpfsug-discuss mailing list