<html><body bgcolor="#FFFFFF"><p>There are multiple ways to accomplish active-active two-side synchronous DR, aka "stretch cluster".<br><br>The most common approach is to have 3 sites: two main sites A and B, plus tiebreaker site C. The two main sites host all data/metadata disks and each has some even number of quorum nodes. There's no stretched SAN, each site has its own set of NSDs defined. The tiebreaker site consists of a single quorum node with a small descOnly LUN. In this config, any of the 3 sites can do down or be disconnected from the rest without affecting the other two. The tiebreaker site is essential: it provides a quorum node for node majority quorum to function, and a descOnly disk for the file system descriptor quorum. Technically speaking, one do away with the need to have a quorum node at site C by using "minority quorum", i.e. tiebreaker disks, but this model is more complex and it is harder to predict its behavior under various failure conditions. The basic problem with the minority quorum is that it allows a minority of nodes to win in a network partition scenario, just like the name implies. In the extreme case this leads to the "dictator problem", when a single partitioned node could manage to win the disk election and thus kick everyone else out. And since a tiebreaker disk needs to be visible from all quorum nodes, you do need a stretched SAN that extends between sites. The classic active-active stretch cluster only requires a good TCP/IP network. <br><br>The question that gets asked a lot is "how good should be network connection between sites be". There's no simple answer, unfortunately. It would be completely impractical to try to frame this in simple thresholds. The worse the network connection is, the more pain it produces, but everyone has a different level of pain tolerance. And everyone's workload is different. In any GPFS configuration that uses data replication, writes are impacted far more by replication than reads. So a read-mostly workload may run fine with a dodgy inter-site link, while a write-heavy workload may just run into the ground, as IOs may be submitted faster than they could be completed. The buffering model could make a big difference. An application that does a fair amount of write bursts, with those writes being buffered in a generously sized pagepool, may perform acceptably, while a different application that uses O_SYNC or O_DIRECT semantics for writes may run a lot worse, all other things being equal. As long as all nodes can renew their disk leases within the configured disk lease interval (35 sec by default), GPFS will basically work, so the absolute threshold for the network link quality is not particularly stringent, but beyond that it all depends on your workload and your level of pain tolerance. Practically speaking, you want a network link with low-double-digits RTT at worst, almost no packet loss, and bandwidth commensurate with your application IO needs (fudged some to allow for write amplification -- another factor that's entirely workload-dependent). So a link with, say, 100ms RTT and 2% packet loss is not going to be usable to almost anyone, in my opinion, a link with 30ms RTT and 0.1% packet loss may work for some undemanding read-mostly workloads, and so on. So you pretty much have to try it out to see.<br><br>The disk configuration is another tricky angle. The simplest approach is to have two groups of data/metadata NSDs, on sites A and B, and not have any sort of SAN reaching across sites. Historically, such a config was actually preferred over a stretched SAN, because it allowed for a basic site topology definition. When multiple replicas of the same logical block are present, it is obviously better/faster to read the replica that resides on a disk that's local to a given site. This is conceptually simple, but how would GPFS know what a site is and what disks are local vs remote? To GPFS, all disks are equal. Historically, the readReplicaPolicy=local config parameter was put forward to work around the problem. The basic idea was: if the reader node is on the same subnet as the primary NSD server for a given replica, this replica is "local", and is thus preferred. This sort of works, but requires a very specific network configuration, which isn't always practical. Starting with GPFS 4.1.1, GPFS implements readReplicaPolicy=fastest, where the best replica for reads is picked based on observed disk IO latency. This is more general and works for all disk topologies, including a stretched SAN. <br><br>yuri<br><br><img width="16" height="16" src="cid:1__=07BBF56BDFCC00AB8f9e8a93df938690918c07B@" border="0" alt="Inactive hide details for "Mark.Bush@siriuscom.com" ---07/21/2016 05:45:42 AM---This is where my confusion sits. So if I have "><font color="#424282">"Mark.Bush@siriuscom.com" ---07/21/2016 05:45:42 AM---This is where my confusion sits. So if I have two sites, and two NDS Nodes per site with 1 NSD (to</font><br><br><font size="2" color="#5F5F5F">From: </font><font size="2">"Mark.Bush@siriuscom.com" <Mark.Bush@siriuscom.com></font><br><font size="2" color="#5F5F5F">To: </font><font size="2">gpfsug main discussion list <gpfsug-discuss@spectrumscale.org>, </font><br><font size="2" color="#5F5F5F">Date: </font><font size="2">07/21/2016 05:45 AM</font><br><font size="2" color="#5F5F5F">Subject: </font><font size="2">Re: [gpfsug-discuss] NDS in Two Site scenario</font><br><font size="2" color="#5F5F5F">Sent by: </font><font size="2">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br><br><br><font face="Calibri">This is where my confusion sits. So if I have two sites, and two NDS Nodes per site with 1 NSD (to keep it simple), do I just present the physical LUN in Site1 to Site1 NDS Nodes and physical LUN in Site2 to Site2 NSD Nodes? Or is it that I present physical LUN in Site1 to all 4 NDS Nodes and the same at Site2? (Assuming SAN and not direct attached in this case). I know I’m being persistent but this for some reason confuses me. </font><br><font face="Calibri"> </font><br><font face="Calibri">Site1</font><br><font face="Calibri">NSD Node1</font><br><font face="Calibri"> ---NSD1 ---Physical LUN1 from SAN1 </font><br><font face="Calibri">NSD Node2</font><br><font face="Calibri"> </font><br><font face="Calibri"> </font><br><font face="Calibri">Site2</font><br><font face="Calibri">NSD Node3 </font><ul><ul><font face="Calibri">---NSD2 –Physical LUN2 from SAN2</font></ul></ul><font face="Calibri">NSD Node4</font><br><font face="Calibri"> </font><br><font face="Calibri"> </font><br><font face="Calibri">Or </font><br><font face="Calibri"> </font><br><font face="Calibri"> </font><br><font face="Calibri">Site1</font><br><font face="Calibri">NSD Node1</font><br><font face="Calibri"> ----NSD1 –Physical LUN1 from SAN1</font><br><font face="Calibri"> ----NSD2 –Physical LUN2 from SAN2</font><br><font face="Calibri">NSD Node2</font><br><font face="Calibri"> </font><br><font face="Calibri">Site 2</font><br><font face="Calibri">NSD Node3 </font><br><font face="Calibri"> ---NSD2 – Physical LUN2 from SAN2</font><br><font face="Calibri"> ---NSD1 --Physical LUN1 from SAN1</font><br><font face="Calibri">NSD Node4</font><br><font face="Calibri"> </font><br><font face="Calibri"> </font><br><font face="Calibri">Site 3</font><br><font face="Calibri">Node5 Quorum</font><br><font face="Calibri"> </font><br><font face="Calibri"> </font><br><font face="Calibri"> </font><br><b><font size="4" face="Calibri">From: </font></b><font size="4" face="Calibri"><gpfsug-discuss-bounces@spectrumscale.org> on behalf of Ken Hill <kenh@us.ibm.com></font><b><font size="4" face="Calibri"><br>Reply-To: </font></b><font size="4" face="Calibri">gpfsug main discussion list <gpfsug-discuss@spectrumscale.org></font><b><font size="4" face="Calibri"><br>Date: </font></b><font size="4" face="Calibri">Wednesday, July 20, 2016 at 7:02 PM</font><b><font size="4" face="Calibri"><br>To: </font></b><font size="4" face="Calibri">gpfsug main discussion list <gpfsug-discuss@spectrumscale.org></font><b><font size="4" face="Calibri"><br>Subject: </font></b><font size="4" face="Calibri">Re: [gpfsug-discuss] NDS in Two Site scenario</font><br><font size="4" face="Times New Roman"> </font><br>Yes - it is a cluster.<font size="4" face="Times New Roman"><br></font><br>The sites should NOT be further than a MAN - or Campus network. If you're looking to do this over a large distance - it would be best to choose another GPFS solution (Multi-Cluster, AFM, etc).<font size="4" face="Times New Roman"><br></font><br>Regards,<font size="4" face="Times New Roman"><br></font><b><font size="4" color="#8F8F8F" face="Arial"><br>Ken Hill</font></b><font face="Arial"><br>Technical Sales Specialist | Software Defined Solution Sales<br>IBM Systems</font><font size="4" face="Times New Roman"> </font><p><table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0"><tr valign="top"><td width="776" colspan="3" valign="middle"><div align="center"><hr width="100%" size="2" align="center"></div></td></tr>
<tr valign="top"><td width="158" valign="middle"><b><font size="2" color="#4181C0" face="Arial">Phone:</font></b><font size="2" face="Arial">1-540-207-7270</font><b><font size="2" color="#4181C0" face="Arial"><br>E-mail:</font></b><font size="4" face="Arial"> </font><a href="mailto:kenh@us.ibm.com" target="_blank"><u><font size="2" color="#5F5F5F" face="Arial">kenh@us.ibm.com</font></u></a></td><td width="612" valign="middle"><div align="right"><a href="http://www.ibm.com/us-en/"><img src="cid:2__=07BBF56BDFCC00AB8f9e8a93df938690918c07B@" width="74" height="30" border="0"></a><font size="4" face="Times New Roman"> </font><a href="http://www-03.ibm.com/systems/platformcomputing/products/lsf/"><img src="cid:3__=07BBF56BDFCC00AB8f9e8a93df938690918c07B@" width="31" height="30" border="0"></a><font size="4" face="Times New Roman"> </font><a href="http://www-03.ibm.com/systems/platformcomputing/products/high-performance-services/index.html"><img src="cid:4__=07BBF56BDFCC00AB8f9e8a93df938690918c07B@" width="28" height="30" border="0"></a><font size="4" face="Times New Roman"> </font><a href="http://www-03.ibm.com/systems/platformcomputing/products/symphony/index.html"><img src="cid:5__=07BBF56BDFCC00AB8f9e8a93df938690918c07B@" width="29" height="30" border="0"></a><font size="4" face="Times New Roman"> </font><a href="http://www-03.ibm.com/systems/storage/spectrum/"><img src="cid:6__=07BBF56BDFCC00AB8f9e8a93df938690918c07B@" width="26" height="30" border="0"></a><font size="4" face="Times New Roman"> </font><a href="http://www-01.ibm.com/software/tivoli/csi/cloud-storage/"><img src="cid:7__=07BBF56BDFCC00AB8f9e8a93df938690918c07B@" width="26" height="30" border="0"></a><font size="4" face="Times New Roman"> </font><a href="http://www-01.ibm.com/software/tivoli/csi/backup-recovery/"><img src="cid:8__=07BBF56BDFCC00AB8f9e8a93df938690918c07B@" width="26" height="30" border="0"></a><font size="4" face="Times New Roman"> </font><a href="http://www-03.ibm.com/systems/storage/tape/ltfs/index.html"><img src="cid:9__=07BBF56BDFCC00AB8f9e8a93df938690918c07B@" width="26" height="30" border="0"></a><font size="4" face="Times New Roman"> </font><a href="http://www-03.ibm.com/systems/storage/spectrum/"><img src="cid:A__=07BBF56BDFCC00AB8f9e8a93df938690918c07B@" width="27" height="30" border="0"></a><font size="4" face="Times New Roman"> </font><a href="http://www-03.ibm.com/systems/storage/spectrum/scale/"><img src="cid:B__=07BBF56BDFCC00AB8f9e8a93df938690918c07B@" width="26" height="30" border="0"></a><font size="4" face="Times New Roman"> </font><a href="https://www.ibm.com/marketplace/cloud/object-storage/us/en-us"><img src="cid:C__=07BBF56BDFCC00AB8f9e8a93df938690918c07B@" width="26" height="30" border="0"></a><font size="4" face="Times New Roman"> </font><font size="2" face="Arial"><br><br>2300 Dulles Station Blvd<br>Herndon, VA 20171-6133<br>United States</font></div></td><td width="6"><img width="1" height="1" src="cid:D__=07BBF56BDFCC00AB8f9e8a93df938690918c07B@" border="0" alt=""></td></tr></table><font size="4" face="Times New Roman"><br><br><br><br><br></font><font size="2" color="#5F5F5F"><br>From: </font><font size="2">"Mark.Bush@siriuscom.com" <Mark.Bush@siriuscom.com></font><font size="2" color="#5F5F5F"><br>To: </font><font size="2">gpfsug main discussion list <gpfsug-discuss@spectrumscale.org></font><font size="2" color="#5F5F5F"><br>Date: </font><font size="2">07/20/2016 07:33 PM</font><font size="2" color="#5F5F5F"><br>Subject: </font><font size="2">Re: [gpfsug-discuss] NDS in Two Site scenario</font><font size="2" color="#5F5F5F"><br>Sent by: </font><font size="2">gpfsug-discuss-bounces@spectrumscale.org</font><div align="center"><hr width="100%" size="2" align="center" noshade></div><font size="4" face="Times New Roman"><br><br></font><font face="Calibri"><br>So in this scenario Ken, can server3 see any disks in site1? <br> </font><b><font size="4" face="Calibri"><br>From: </font></b><font size="4" face="Calibri"><gpfsug-discuss-bounces@spectrumscale.org> on behalf of Ken Hill <kenh@us.ibm.com></font><b><font size="4" face="Calibri"><br>Reply-To: </font></b><font size="4" face="Calibri">gpfsug main discussion list <gpfsug-discuss@spectrumscale.org></font><b><font size="4" face="Calibri"><br>Date: </font></b><font size="4" face="Calibri">Wednesday, July 20, 2016 at 4:15 PM</font><b><font size="4" face="Calibri"><br>To: </font></b><font size="4" face="Calibri">gpfsug main discussion list <gpfsug-discuss@spectrumscale.org></font><b><font size="4" face="Calibri"><br>Subject: </font></b><font size="4" face="Calibri">Re: [gpfsug-discuss] NDS in Two Site scenario</font><font size="4" face="Times New Roman"><br> </font><br><br> Site1 Site2<br> Server1 (quorum 1) Server3 (quorum 2)<br> Server2 Server4<font size="4" face="Times New Roman"><br><br><br></font><br><br> SiteX <br> Server5 (quorum 3)<font size="4" face="Times New Roman"><br><br><br></font><br><br>You need to set up another site (or server) that is at least power isolated (if not completely infrastructure isolated) from Site1 or Site2. You would then set up a quorum node at that site | location. This insures you can still access your data even if one of your sites go down.<br><br>You can further isolate failure by increasing quorum (odd numbers).<br><br>The way quorum works is: The majority of the quorum nodes need to be up to survive an outage.<br><br>- With 3 quorum nodes you can have 1 quorum node failures and continue filesystem operations.<br>- With 5 quorum nodes you can have 2 quorum node failures and continue filesystem operations.<br>- With 7 quorum nodes you can have 3 quorum node failures and continue filesystem operations.<br>- etc<br><br>Please see <a href="http://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.0/ibmspectrumscale42_content.html?view=kc"><u><font color="#0000FF">http://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.0/ibmspectrumscale42_content.html?view=kc</font></u></a>for more information about quorum and tiebreaker disks.<b><font size="4" color="#8F8F8F" face="Arial"><br><br>Ken Hill</font></b><font face="Arial"><br>Technical Sales Specialist | Software Defined Solution Sales<br>IBM Systems</font><font size="4" face="Times New Roman"> </font><p><table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0"><tr valign="top"><td width="775" colspan="3" valign="middle"><div align="center"><hr width="100%" size="2" align="center"></div></td></tr>
<tr valign="top"><td width="158" valign="middle"><b><font size="2" color="#4181C0" face="Arial">Phone:</font></b><font size="2" face="Arial">1-540-207-7270</font><b><font size="2" color="#4181C0" face="Arial"><br>E-mail:</font></b><font size="4" face="Arial"> </font><a href="mailto:kenh@us.ibm.com" target="_blank"><u><font size="2" color="#5F5F5F" face="Arial">kenh@us.ibm.com</font></u></a></td><td width="611" valign="middle"><div align="right"><a href="http://www.ibm.com/us-en/"><img src="cid:E__=07BBF56BDFCC00AB8f9e8a93df938690918c07B@" width="74" height="30" border="0"></a><font size="4" face="Times New Roman"> </font><a href="http://www-03.ibm.com/systems/platformcomputing/products/lsf/"><img src="cid:F__=07BBF56BDFCC00AB8f9e8a93df938690918c07B@" width="31" height="30" border="0"></a><font size="4" face="Times New Roman"> </font><a href="http://www-03.ibm.com/systems/platformcomputing/products/high-performance-services/index.html"><img src="cid:10__=07BBF56BDFCC00AB8f9e8a93df938690918c07B@" width="28" height="30" border="0"></a><font size="4" face="Times New Roman"> </font><a href="http://www-03.ibm.com/systems/platformcomputing/products/symphony/index.html"><img src="cid:11__=07BBF56BDFCC00AB8f9e8a93df938690918c07B@" width="29" height="30" border="0"></a><font size="4" face="Times New Roman"> </font><a href="http://www-03.ibm.com/systems/storage/spectrum/"><img src="cid:12__=07BBF56BDFCC00AB8f9e8a93df938690918c07B@" width="26" height="30" border="0"></a><font size="4" face="Times New Roman"> </font><a href="http://www-01.ibm.com/software/tivoli/csi/cloud-storage/"><img src="cid:13__=07BBF56BDFCC00AB8f9e8a93df938690918c07B@" width="26" height="30" border="0"></a><font size="4" face="Times New Roman"> </font><a href="http://www-01.ibm.com/software/tivoli/csi/backup-recovery/"><img src="cid:14__=07BBF56BDFCC00AB8f9e8a93df938690918c07B@" width="26" height="30" border="0"></a><font size="4" face="Times New Roman"> </font><a href="http://www-03.ibm.com/systems/storage/tape/ltfs/index.html"><img src="cid:15__=07BBF56BDFCC00AB8f9e8a93df938690918c07B@" width="26" height="30" border="0"></a><font size="4" face="Times New Roman"> </font><a href="http://www-03.ibm.com/systems/storage/spectrum/"><img src="cid:16__=07BBF56BDFCC00AB8f9e8a93df938690918c07B@" width="27" height="30" border="0"></a><font size="4" face="Times New Roman"> </font><a href="http://www-03.ibm.com/systems/storage/spectrum/scale/"><img src="cid:17__=07BBF56BDFCC00AB8f9e8a93df938690918c07B@" width="26" height="30" border="0"></a><font size="4" face="Times New Roman"> </font><a href="https://www.ibm.com/marketplace/cloud/object-storage/us/en-us"><img src="cid:18__=07BBF56BDFCC00AB8f9e8a93df938690918c07B@" width="26" height="30" border="0"></a><font size="4" face="Times New Roman"> </font><font size="2" face="Arial"><br><br>2300 Dulles Station Blvd<br>Herndon, VA 20171-6133<br>United States</font></div></td><td width="6"><img width="1" height="1" src="cid:D__=07BBF56BDFCC00AB8f9e8a93df938690918c07B@" border="0" alt=""></td></tr></table><font size="4" face="Times New Roman"><br><br><br><br></font><font size="2" color="#5F5F5F"><br><br>From: </font><font size="2">"Mark.Bush@siriuscom.com" <Mark.Bush@siriuscom.com></font><font size="2" color="#5F5F5F"><br>To: </font><font size="2">gpfsug main discussion list <gpfsug-discuss@spectrumscale.org></font><font size="2" color="#5F5F5F"><br>Date: </font><font size="2">07/20/2016 04:47 PM</font><font size="2" color="#5F5F5F"><br>Subject: </font><font size="2">[gpfsug-discuss] NDS in Two Site scenario</font><font size="2" color="#5F5F5F"><br>Sent by: </font><font size="2">gpfsug-discuss-bounces@spectrumscale.org</font><div align="center"><hr width="100%" size="2" align="center" noshade></div><font size="4" face="Times New Roman"><br><br></font><font face="Calibri"><br><br>For some reason this concept is a round peg that doesn’t fit the square hole inside my brain. Can someone please explain the best practice to setting up two sites same cluster? I get that I would likely have two NDS nodes in site 1 and two NDS nodes in site two. What I don’t understand are the failure scenarios and what would happen if I lose one or worse a whole site goes down. Do I solve this by having scale replication set to 2 for all my files? I mean a single site I think I get it’s when there are two datacenters and I don’t want two clusters typically.</font><font face="MingLiU"><br><br></font><font color="#002F80" face="Calibri"><br><br>Mark R. Bush</font><font face="Calibri">| Solutions Architect<br>Mobile: 210.237.8415 | mark.bush@siriuscom.com<br>Sirius Computer Solutions | </font><a href="http://www.siriuscom.com/"><u><font color="#0000FF" face="Calibri">www.siriuscom.com</font></u></a><font face="Calibri"><br>10100 Reunion Place, Suite 500, San Antonio, TX 78216 </font><font size="4" face="Times New Roman"><br> </font><p><font size="2" face="Cambria">This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. </font><p><a href="http://www.siriuscom.com/"><b><u><font color="#0082BF" face="Calibri">Sirius Computer Solutions</font></u></b></a><font face="Courier New">_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org</font><u><font size="4" color="#0000FF" face="Times New Roman"><br></font></u><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><u><font color="#0000FF" face="Courier New">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></u></a><p><font size="4" face="Times New Roman"> </font><font face="Courier New">_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org</font><u><font size="4" color="#0000FF" face="Times New Roman"><br></font></u><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><u><font color="#0000FF" face="Courier New">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></u></a><p><font size="4" face="Times New Roman"> </font><tt>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></tt><tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></tt><tt><br></tt><p><BR>
</body></html>