<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:Calibri;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:Calibri;
color:windowtext;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:Calibri;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt"> Some follow-up to the discussion I kicked off a few days ago.<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Using simple GPFS replication on two sites looked like a good option, until you consider it’s really RAID5, and if the replica copy of the data fails during the restripe, you lose data. It’s not as bad as
RAID5 because the data blocks for a file are spread across multiple servers versus reconstruction of a single array.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Raid 6 + Metadata replication isn’t a bad option but you are vulnerable to server failure. It’s relatively low expansion factor makes it attractive.
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">My personal recommendation is going to be use Raid 6 + Metadata Replication (use “unmountOnDiskFail=meta” option), keep a spare server around to minimize downtime if one fails. Array rebuild times will impact
performance, but it’s the price of having integrity.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Comments?<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="1117" style="width:838.0pt;border-collapse:collapse">
<tbody>
<tr style="height:38.0pt">
<td width="157" valign="bottom" style="width:118.0pt;padding:0in 5.4pt 0in 5.4pt;height:38.0pt">
<p class="MsoNormal"><b><u><span style="font-size:14.0pt;color:black">Data Distribution</span></u></b><o:p></o:p></p>
</td>
<td width="103" valign="bottom" style="width:77.0pt;padding:0in 5.4pt 0in 5.4pt;height:38.0pt">
<p class="MsoNormal"><b><u><span style="font-size:14.0pt;color:black">Expansion Factor</span></u></b><o:p></o:p></p>
</td>
<td width="141" valign="bottom" style="width:106.0pt;padding:0in 5.4pt 0in 5.4pt;height:38.0pt">
<p class="MsoNormal"><b><u><span style="font-size:14.0pt;color:black">Data Availability (Disk Failure)</span></u></b><o:p></o:p></p>
</td>
<td width="155" valign="bottom" style="width:116.0pt;padding:0in 5.4pt 0in 5.4pt;height:38.0pt">
<p class="MsoNormal"><b><u><span style="font-size:14.0pt;color:black">Data Availability (Server Failure)</span></u></b><o:p></o:p></p>
</td>
<td width="81" valign="bottom" style="width:61.0pt;padding:0in 5.4pt 0in 5.4pt;height:38.0pt">
<p class="MsoNormal"><b><u><span style="font-size:14.0pt;color:black">Data Integrity</span></u></b><o:p></o:p></p>
</td>
<td width="480" valign="bottom" style="width:5.0in;padding:0in 5.4pt 0in 5.4pt;height:38.0pt">
<p class="MsoNormal"><b><u><span style="font-size:14.0pt;color:black">Comments</span></u></b><o:p></o:p></p>
</td>
</tr>
<tr style="height:76.0pt">
<td width="157" valign="top" style="width:118.0pt;padding:0in 5.4pt 0in 5.4pt;height:76.0pt">
<p class="MsoNormal"><span style="font-size:14.0pt;color:black">Raid 6 (6+2) + Metadata replication</span><o:p></o:p></p>
</td>
<td width="103" valign="top" style="width:77.0pt;padding:0in 5.4pt 0in 5.4pt;height:76.0pt">
<p class="MsoNormal"><span style="font-size:14.0pt;color:black">1.25+</span><o:p></o:p></p>
</td>
<td width="141" valign="top" style="width:106.0pt;padding:0in 5.4pt 0in 5.4pt;height:76.0pt">
<p class="MsoNormal"><span style="font-size:14.0pt;color:black">High</span><o:p></o:p></p>
</td>
<td width="155" valign="top" style="width:116.0pt;padding:0in 5.4pt 0in 5.4pt;height:76.0pt">
<p class="MsoNormal"><span style="font-size:14.0pt;color:black">Low</span><o:p></o:p></p>
</td>
<td width="81" valign="top" style="width:61.0pt;padding:0in 5.4pt 0in 5.4pt;height:76.0pt">
<p class="MsoNormal"><span style="font-size:14.0pt;color:black">High</span><o:p></o:p></p>
</td>
<td width="480" valign="top" style="width:5.0in;padding:0in 5.4pt 0in 5.4pt;height:76.0pt">
<p class="MsoNormal"><span style="font-size:14.0pt;color:black">Single server or single LUN failure results in some data being unavailable. Single Drive failure - lower performance during array rebuild.
</span><o:p></o:p></p>
</td>
</tr>
<tr style="height:57.0pt">
<td width="157" valign="top" style="width:118.0pt;padding:0in 5.4pt 0in 5.4pt;height:57.0pt">
<p class="MsoNormal"><span style="font-size:14.0pt;color:black">2 site replication (GPFS)</span><o:p></o:p></p>
</td>
<td width="103" valign="top" style="width:77.0pt;padding:0in 5.4pt 0in 5.4pt;height:57.0pt">
<p class="MsoNormal"><span style="font-size:14.0pt;color:black">2</span><o:p></o:p></p>
</td>
<td width="141" valign="top" style="width:106.0pt;padding:0in 5.4pt 0in 5.4pt;height:57.0pt">
<p class="MsoNormal"><span style="font-size:14.0pt;color:black">High</span><o:p></o:p></p>
</td>
<td width="155" valign="top" style="width:116.0pt;padding:0in 5.4pt 0in 5.4pt;height:57.0pt">
<p class="MsoNormal"><span style="font-size:14.0pt;color:black">High</span><o:p></o:p></p>
</td>
<td width="81" valign="top" style="width:61.0pt;padding:0in 5.4pt 0in 5.4pt;height:57.0pt">
<p class="MsoNormal"><span style="font-size:14.0pt;color:black">Low</span><o:p></o:p></p>
</td>
<td width="480" valign="top" style="width:5.0in;padding:0in 5.4pt 0in 5.4pt;height:57.0pt">
<p class="MsoNormal"><span style="font-size:14.0pt;color:black">Similar to RAID 5 - vulnerable to multiple disk failures. Rebuild done via GPFS restripe. URE vulnerable during restripe, but data distribution may mitigate this.</span><o:p></o:p></p>
</td>
</tr>
<tr style="height:38.0pt">
<td width="157" valign="top" style="width:118.0pt;padding:0in 5.4pt 0in 5.4pt;height:38.0pt">
<p class="MsoNormal"><span style="font-size:14.0pt;color:black">Raid 6 (6+2) + Full 2 site replication (GPFS)</span><o:p></o:p></p>
</td>
<td width="103" valign="top" style="width:77.0pt;padding:0in 5.4pt 0in 5.4pt;height:38.0pt">
<p class="MsoNormal"><span style="font-size:14.0pt;color:black">2.5</span><o:p></o:p></p>
</td>
<td width="141" valign="top" style="width:106.0pt;padding:0in 5.4pt 0in 5.4pt;height:38.0pt">
<p class="MsoNormal"><span style="font-size:14.0pt;color:black">High</span><o:p></o:p></p>
</td>
<td width="155" valign="top" style="width:116.0pt;padding:0in 5.4pt 0in 5.4pt;height:38.0pt">
<p class="MsoNormal"><span style="font-size:14.0pt;color:black">High</span><o:p></o:p></p>
</td>
<td width="81" valign="top" style="width:61.0pt;padding:0in 5.4pt 0in 5.4pt;height:38.0pt">
<p class="MsoNormal"><span style="font-size:14.0pt;color:black">High</span><o:p></o:p></p>
</td>
<td width="480" valign="top" style="width:5.0in;padding:0in 5.4pt 0in 5.4pt;height:38.0pt">
<p class="MsoNormal"><span style="font-size:14.0pt;color:black">Protected against single server and double drive failures. Single Drive failure - lower performance during array rebuild.</span><o:p></o:p></p>
</td>
</tr>
<tr style="height:38.0pt">
<td width="157" valign="top" style="width:118.0pt;padding:0in 5.4pt 0in 5.4pt;height:38.0pt">
<p class="MsoNormal"><span style="font-size:14.0pt;color:black">Full 3 site replication (GPFS)</span><o:p></o:p></p>
</td>
<td width="103" valign="top" style="width:77.0pt;padding:0in 5.4pt 0in 5.4pt;height:38.0pt">
<p class="MsoNormal"><span style="font-size:14.0pt;color:black">3</span><o:p></o:p></p>
</td>
<td width="141" valign="top" style="width:106.0pt;padding:0in 5.4pt 0in 5.4pt;height:38.0pt">
<p class="MsoNormal"><span style="font-size:14.0pt;color:black">High</span><o:p></o:p></p>
</td>
<td width="155" valign="top" style="width:116.0pt;padding:0in 5.4pt 0in 5.4pt;height:38.0pt">
<p class="MsoNormal"><span style="font-size:14.0pt;color:black">High</span><o:p></o:p></p>
</td>
<td width="81" valign="top" style="width:61.0pt;padding:0in 5.4pt 0in 5.4pt;height:38.0pt">
<p class="MsoNormal"><span style="font-size:14.0pt;color:black">High</span><o:p></o:p></p>
</td>
<td width="480" valign="top" style="width:5.0in;padding:0in 5.4pt 0in 5.4pt;height:38.0pt">
<p class="MsoNormal"><span style="font-size:14.0pt;color:black">Similar to RAID 6. Protected against single server and double drive failures. Rebuild done via GPFS restripe.</span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal">Bob Oesterlin<br>
Sr Principal Storage Engineer, Nuance<o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Times New Roman""> </span><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</body>
</html>