<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Cambria;
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">You wouldn’t be alone in trying to make the “concurrent CES gateway + NSD server nodes” formula work. That doesn’t mean it will be well-supported initially,
but it does mean that others will be finding bugs and interaction issues along with you.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">On GPFS 4.1.1.2 for example, it’s possible to get a CES protocol node into a state where the mmcesmonitor is dead and requires a mmshutdown/mmstartup to recover
from. Since in a shared-nothing disk topology that would require mmchdisk/mmrestripefs to recover and rebalance, it would be operationally intensive to run CES on an NSD server with local disks. With shared SAN disks, this becomes more tractable, in my opinion.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thx<br>
Paul<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> gpfsug-discuss-bounces@spectrumscale.org [mailto:gpfsug-discuss-bounces@spectrumscale.org]
<b>On Behalf Of </b>Zachary Giles<br>
<b>Sent:</b> Friday, March 04, 2016 11:37 AM<br>
<b>To:</b> gpfsug main discussion list<br>
<b>Subject:</b> Re: [gpfsug-discuss] Small cluster<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">SMB too, eh? See this is where it starts to get hard to scale down. You could do a 3 node GPFS cluster with replication at remote sites, pulling in from AFM over the Net. If you want SMB too, you're probably going to need another pair of
servers to act as the Protocol Servers on top of the 3 GPFS servers. I think running them all together is not recommended, and probably I'd agree with that.<o:p></o:p></p>
<div>
<p class="MsoNormal">Though, you could do it anyway. If it's for read-only and updated daily, eh, who cares. Again, depends on your GPFS experience and the balance between production, price, and performance :)<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Fri, Mar 4, 2016 at 11:30 AM, <a href="mailto:Mark.Bush@siriuscom.com">
Mark.Bush@siriuscom.com</a> <<a href="mailto:Mark.Bush@siriuscom.com" target="_blank">Mark.Bush@siriuscom.com</a>> wrote:<o:p></o:p></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">Yes. Really the only other option we have (and not a bad one) is getting a v7000 Unified in there (if we can get the price down far enough). That’s not a bad
option since all they really want is SMB shares in the remote. I just keep thinking a set of servers would do the trick and be cheaper.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-family:"Calibri","sans-serif";color:black">From:
</span></b><span style="font-family:"Calibri","sans-serif";color:black">Zachary Giles <<a href="mailto:zgiles@gmail.com" target="_blank">zgiles@gmail.com</a>><br>
<b>Reply-To: </b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a>><br>
<b>Date: </b>Friday, March 4, 2016 at 10:26 AM<o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif";color:black"><br>
<b>To: </b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a>><br>
<b>Subject: </b>Re: [gpfsug-discuss] Small cluster<o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">You can do FPO for non-Hadoop workloads. It just alters the disks below the GPFS filesystem layer and looks like a normal GPFS system (mostly). I do think there
were some restrictions on non-FPO nodes mounting FPO filesystems via multi-cluster.. not sure if those are still there.. any input on that from IBM?
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">If small enough data, and with 3-way replication, it might just be wise to do internal storage and 3x rep. A 36TB 2U server is ~$10K (just common throwing out
numbers), 3 of those per site would fit in your budget.<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">Again.. depending on your requirements, stability balance between 'science experiment' vs production, GPFS knowledge level, etc etc...<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">This is actually an interesting and somewhat missing space for small enterprises. If you just want 10-20TB active-active online everywhere, say, for VMware, or
NFS, or something else, there arent all that many good solutions today that scale down far enough and are a decent price. It's easy with many many PB, but small.. idk. I think the above sounds good as anything without going SAN-crazy.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">On Fri, Mar 4, 2016 at 11:21 AM,
<a href="mailto:Mark.Bush@siriuscom.com" target="_blank">Mark.Bush@siriuscom.com</a> <<a href="mailto:Mark.Bush@siriuscom.com" target="_blank">Mark.Bush@siriuscom.com</a>> wrote:<o:p></o:p></span></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">I guess this is really my question. Budget is less than $50k per site and they need around 20TB storage. Two nodes with MD3 or something may work. But could
it work (and be successful) with just servers and internal drives? Should I do FPO for non hadoop like workloads? I didn’t think I could get native raid except in the ESS (GSS no longer exists if I remember correctly). Do I just make replicas and call it
good?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">Mark<o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-family:"Calibri","sans-serif";color:black">From:
</span></b><span style="font-family:"Calibri","sans-serif";color:black"><<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a>> on behalf of Marc A Kaplan <<a href="mailto:makaplan@us.ibm.com" target="_blank">makaplan@us.ibm.com</a>><br>
<b>Reply-To: </b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a>><br>
<b>Date: </b>Friday, March 4, 2016 at 10:09 AM<br>
<b>To: </b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a>><br>
<b>Subject: </b>Re: [gpfsug-discuss] Small cluster<o:p></o:p></span></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:black">Jon, I don't doubt your experience, but it's not quite fair or even sensible to make a decision today based on what was available in
the GPFS 2.3 era.</span><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><br>
<br>
</span><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:black">We are now at GPFS 4.2 with support for 3 way replication and FPO. </span><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><br>
</span><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:black">Also we have Raid controllers, IB, and "Native Raid" and ESS, GSS solutions and more.</span><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><br>
<br>
</span><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:black">So more choices, more options, making finding an "optimal" solution more difficult.<br>
</span><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><br>
</span><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:black">To begin with, as with any provisioning problem, one should try to state: requirements, goals, budgets, constraints, failure/tolerance models/assumptions,</span><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><br>
</span><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:black">expected workloads, desired performance, etc, etc.</span><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><br>
<br>
<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
<p style="line-height:10.0pt"><span style="font-size:8.0pt;font-family:"Cambria","serif";color:black">This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that
is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited.
This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify
Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you.
<o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Calibri","sans-serif";color:#CC0000"><a href="http://www.siriuscom.com" target="_blank">Sirius Computer Solutions</a></span></b><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">
<o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><br>
<br clear="all">
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">--
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">Zach Giles<br>
<a href="mailto:zgiles@gmail.com" target="_blank">zgiles@gmail.com</a><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><o:p></o:p></p>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<p class="MsoNormal">Zach Giles<br>
<a href="mailto:zgiles@gmail.com" target="_blank">zgiles@gmail.com</a><o:p></o:p></p>
</div>
</div>
</div>
</body>
</html>