<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
tt
{mso-style-priority:99;
font-family:"Courier New";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:168564542;
mso-list-type:hybrid;
mso-list-template-ids:-233825940 -1349465274 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:2016;
mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:20.25pt;
text-indent:-.25in;
font-family:Symbol;
mso-fareast-font-family:Calibri;
mso-bidi-font-family:"Times New Roman";}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:56.25pt;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:92.25pt;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:128.25pt;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:164.25pt;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:200.25pt;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:236.25pt;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:272.25pt;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:308.25pt;
text-indent:-.25in;
font-family:Wingdings;}
@list l1
{mso-list-id:884412509;
mso-list-type:hybrid;
mso-list-template-ids:468100942 625665088 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l1:level1
{mso-level-start-at:2016;
mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;
mso-fareast-font-family:Calibri;
mso-bidi-font-family:"Times New Roman";}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">All of Marc’s points are good. A few more things to be aware of with regard to replicated writes:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l1 level1 lfo2"><![if !supportLists]><span style="font-size:11.0pt;font-family:Symbol;color:#1F497D"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Each client performs its own replication when it writes file data. So if you have several clients, each writing files concurrently, the “bandwidth
burden” of the replication is distributed among them. It’s typical that your write throughput will be limited by disk in this case.<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l1 level1 lfo2"><![if !supportLists]><span style="font-size:11.0pt;font-family:Symbol;color:#1F497D"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Because clients perform their own write replication, the max write throughput of a NSD client is limited to <50% of its available network bandwidth
for 2x replication, or <33% for 3x replication, since it must share the network interface (Ethernet, IB) to access the NSDs in each failure group.<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l1 level1 lfo2"><![if !supportLists]><span style="font-size:11.0pt;font-family:Symbol;color:#1F497D"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">If your network topology is asymmetric (e.g. multiple dataceters with higher latency and limited bandwidth between them) you may also benefit from
using “readReplicaPolicy=fastest” to keep read traffic “local” and avoid crossing congested or high-latency paths.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> gpfsug-discuss-bounces@spectrumscale.org [mailto:gpfsug-discuss-bounces@spectrumscale.org]
<b>On Behalf Of </b>Marc A Kaplan<br>
<b>Sent:</b> Tuesday, December 01, 2015 12:02 PM<br>
<b>To:</b> gpfsug main discussion list<br>
<b>Subject:</b> Re: [gpfsug-discuss] IO performance of replicated GPFS filesystem<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Generally yes. When reading, more disks is always better then fewer disks, both for replication and with striping over several or many disks.</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">When writing, more disks is good with striping. But yes, replication costs you extra writes. Those writes don't necessarily cost you loss of time, provided they can be done concurrently.</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">When I write "disks" I mean storage devices that can be accessed concurrently. Watch out for virtual LUNs. </span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">With conventional controllers and drives, it does GPFS little or no good when multiple LUNs map to the same real disk device, since multiple operations to different LUNs will ultimately be serialized
at </span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">one real disk arm/head!</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">For high performance, you should not be thinking about "two NSDs" ... you should be thinking about many NSD, so data and metadata can be striped, and written and read concurrently.</span><br>
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">But yes, for replication purposes you have to consider defining and properly configuring at least two "failure groups".</span><br>
<br>
<br>
<br>
<span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">From: </span><span style="font-size:7.5pt;font-family:"Arial","sans-serif"">"<a href="mailto:Tomasz.Wolski@ts.fujitsu.com">Tomasz.Wolski@ts.fujitsu.com</a>" <<a href="mailto:Tomasz.Wolski@ts.fujitsu.com">Tomasz.Wolski@ts.fujitsu.com</a>></span><br>
<span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">To: </span><span style="font-size:7.5pt;font-family:"Arial","sans-serif"">"<a href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.org</a>" <<a href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.org</a>></span><br>
<span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">Date: </span><span style="font-size:7.5pt;font-family:"Arial","sans-serif"">11/30/2015 05:46 AM</span><br>
<span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">Subject: </span><span style="font-size:7.5pt;font-family:"Arial","sans-serif"">[gpfsug-discuss] IO performance of replicated GPFS filesystem</span><br>
<span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#5F5F5F">Sent by: </span><span style="font-size:7.5pt;font-family:"Arial","sans-serif""><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">gpfsug-discuss-bounces@spectrumscale.org</a></span><o:p></o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" noshade="" style="color:#A0A0A0" align="center">
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
<br>
<br>
<span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">Hi All,</span><br>
<span style="font-size:10.0pt;font-family:"Calibri","sans-serif""> </span><br>
<span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">I could use some help of the experts here
</span><span style="font-size:10.0pt;font-family:Wingdings">J</span><span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">Please correct me if I’m wrong: I suspect that GPFS filesystem READ performance is better when filesystem is replicated to
i.e. two failure groups, where these failure groups are placed on separate RAID controllers. In this case WRITE performance should be worse, since the same data must go to two locations. What about situation where GPFS filesystem has two metadataOnly NSDs
which are also replicated? Does metadata READ performance increase in this way as well (and WRITE decreases)?</span><br>
<span style="font-size:10.0pt;font-family:"Calibri","sans-serif""> </span><br>
<span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">Best regards,</span><br>
<span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">Tomasz Wolski</span><tt><span style="font-size:10.0pt">_______________________________________________</span></tt><span style="font-size:10.0pt;font-family:"Courier New""><br>
<tt>gpfsug-discuss mailing list</tt><br>
<tt>gpfsug-discuss at spectrumscale.org</tt><br>
</span><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><span style="font-size:10.0pt">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></tt></a><span style="font-size:10.0pt;font-family:"Courier New""><br>
<br>
</span><o:p></o:p></p>
</div>
</body>
</html>