<html><body><p>as i stated in my previous post , its a recommendation so people don't overload the NSD servers to have them become non responsive or even forced rebooted (e.g. when you configure cNFS auto reboot on same node), it doesn't mean it doesn't work or is not supported. <br>if all you are using this cluster for is NAS services, then this recommendation makes even less sense as the whole purpose on why the recommendation is there to begin with is that if NFS would overload a node that also serves as NSD server for other nodes it would impact the other nodes that use the NSD protocol, but if there are no NSD clients there is nothing to protect because if NFS is down all clients are not able to access data, even if your NSD servers are perfectly healthy...<br><br>if you have a fairly large system with many NSD Servers, many clients as well as NAS clients this recommendation is correct, but not in the scenario you described below. <br><br>i will work with the team to come up with a better wording for this in the FAQ. <br><br>------------------------------------------<br>Sven Oehme <br>Scalable Storage Research <br>email: oehmes@us.ibm.com <br>Phone: +1 (408) 824-8904 <br>IBM Almaden Research Lab <br>------------------------------------------<br><br><img width="16" height="16" src="cid:1__=4EBBF5FEDFDA58308f9e8a93df938690918c4EB@" border="0" alt="Inactive hide details for Jan-Frode Myklebust ---03/05/2016 02:17:12 PM---Regarding #1, the FAQ has recommendation to not run C"><font color="#424282">Jan-Frode Myklebust ---03/05/2016 02:17:12 PM---Regarding #1, the FAQ has recommendation to not run CES nodes directly attached to storage:</font><br><br><font size="2" color="#5F5F5F">From: </font><font size="2">Jan-Frode Myklebust <janfrode@tanso.net></font><br><font size="2" color="#5F5F5F">To: </font><font size="2">gpfsug main discussion list <gpfsug-discuss@spectrumscale.org></font><br><font size="2" color="#5F5F5F">Cc: </font><font size="2">Sven Oehme/Almaden/IBM@IBMUS</font><br><font size="2" color="#5F5F5F">Date: </font><font size="2">03/05/2016 02:17 PM</font><br><font size="2" color="#5F5F5F">Subject: </font><font size="2">Re: [gpfsug-discuss] Small cluster</font><br><hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br><br><br><font size="4">Regarding #1, the FAQ has recommendation to not run CES nodes directly attached to storage:<br><br>""" • NSD server functionality and storage attached to Protocol node. We recommend that Protocol nodes do not take on these functions<br>"""<br><br>For small CES clusters we're now configuring 2x P822L with one partition on each server owning FC adapters and acting as NSD server/quorum/manager and the other partition being CES node accessing disk via IP.<br><br>I would much rather have a plain SAN model cluster were all nodes accessed disk directly (probably still with a dedicated quorum/manager partition), but this FAQ entry is preventing that..<br><br><br>-jf</font><br><br><font size="4">fre. 4. mar. 2016 kl. 19.04 skrev Sven Oehme <</font><a href="mailto:oehmes@us.ibm.com"><u><font size="4" color="#0000FF">oehmes@us.ibm.com</font></u></a><font size="4">>:</font><ul><font size="4">Hi,<br><br>a couple of comments to the various infos in this thread. <br><br>1. the need to run CES on separate nodes is a recommendation, not a requirement and the recommendation comes from the fact that if you have heavy loaded NAS traffic that gets the system to its knees, you can take your NSD service down with you if its on the same box. so as long as you have a reasonable performance expectation and size the system correct there is no issue.<br><br>2. shared vs FPO vs shared nothing (just replication) . the main issue people overlook in this scenario is the absence of read/write caches in FPO or shared nothing configurations. every physical disk drive can only do ~100 iops and thats independent if the io size is 1 byte or 1 megabyte its pretty much the same effort. particular on metadata this bites you really badly as every of this tiny i/os eats one of your 100 iops a disk can do and quickly you used up all your iops on the drives. if you have any form of raid controller (sw or hw) it typically implements at minimum a read cache on most systems a read/write cache which will significant increase the number of logical i/os one can do against a disk , my best example is always if you have a workload that does 4k seq DIO writes to a single disk, if you have no raid controller you can do 400k/sec in this workload if you have a reasonable ok write cache in front of the cache you can do 50 times that much. so especilly if you use snapshots, CES services or anything thats metadata intensive you want some type of raid protection with caching. btw. replication in the FS makes this even worse as now each write turns into 3 iops for the data + additional iops for the log records so you eat up your iops very quick . <br><br>3. instead of shared SAN a shared SAS device is significantly cheaper but only scales to 2-4 nodes , the benefit is you only need 2 instead of 3 nodes as you can use the disks as tiebreaker disks. if you also add some SSD's for the metadata and make use of HAWC and LROC you might get away from not needing a raid controller with cache as HAWC will solve that issue for you . <br><br>just a few thoughts :-D<br><br>sven<br><br><br>------------------------------------------<br>Sven Oehme <br>Scalable Storage Research <br>email: </font><a href="mailto:oehmes@us.ibm.com" target="_blank"><u><font size="4" color="#0000FF">oehmes@us.ibm.com</font></u></a><font size="4"> <br>Phone: +1 (408) 824-8904 <br>IBM Almaden Research Lab <br>------------------------------------------<br><br></font><font size="4" color="#424282">Zachary Giles ---03/04/2016 05:36:50 PM---SMB too, eh? See this is where it starts to get hard to scale down. You could do a 3 node GPFS clust</font><font size="4"><br></font><font color="#5F5F5F"><br>From: </font>Zachary Giles <<a href="mailto:zgiles@gmail.com" target="_blank"><u><font color="#0000FF">zgiles@gmail.com</font></u></a>>
<p><font color="#5F5F5F"><br>To: </font>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank"><u><font color="#0000FF">gpfsug-discuss@spectrumscale.org</font></u></a>>
<p><font color="#5F5F5F">Date: </font>03/04/2016 05:36 PM
<p><font color="#5F5F5F"><br>Subject: </font>Re: [gpfsug-discuss] Small cluster
<p><font color="#5F5F5F">Sent by: </font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank"><u><font color="#0000FF">gpfsug-discuss-bounces@spectrumscale.org</font></u></a><font size="4"><br></font><hr width="100%" size="2" align="left" noshade><p><font size="4"><br><br></font><font size="5"><br>SMB too, eh? See this is where it starts to get hard to scale down. You could do a 3 node GPFS cluster with replication at remote sites, pulling in from AFM over the Net. If you want SMB too, you're probably going to need another pair of servers to act as the Protocol Servers on top of the 3 GPFS servers. I think running them all together is not recommended, and probably I'd agree with that.<br>Though, you could do it anyway. If it's for read-only and updated daily, eh, who cares. Again, depends on your GPFS experience and the balance between production, price, and performance :)</font><font size="4"><br></font><font size="5"><br>On Fri, Mar 4, 2016 at 11:30 AM, </font><a href="mailto:Mark.Bush@siriuscom.com" target="_blank"><u><font size="5" color="#0000FF">Mark.Bush@siriuscom.com</font></u></a><font size="5"> <</font><a href="mailto:Mark.Bush@siriuscom.com" target="_blank"><u><font size="5" color="#0000FF">Mark.Bush@siriuscom.com</font></u></a><font size="5">> wrote:</font><ul><ul><font size="4" face="Calibri">Yes. Really the only other option we have (and not a bad one) is getting a v7000 Unified in there (if we can get the price down far enough). That’s not a bad option since all they really want is SMB shares in the remote. I just keep thinking a set of servers would do the trick and be cheaper.</font><font size="4"><br><br><br></font><b><font size="5" face="Calibri"><br>From: </font></b><font size="5" face="Calibri">Zachary Giles <</font><a href="mailto:zgiles@gmail.com" target="_blank"><u><font size="5" color="#0000FF" face="Calibri">zgiles@gmail.com</font></u></a><font size="5" face="Calibri">></font><b><font size="5" face="Calibri"><br>Reply-To: </font></b><font size="5" face="Calibri">gpfsug main discussion list <</font><a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank"><u><font size="5" color="#0000FF" face="Calibri">gpfsug-discuss@spectrumscale.org</font></u></a><font size="5" face="Calibri">></font><b><font size="5" face="Calibri"><br>Date: </font></b><font size="5" face="Calibri">Friday, March 4, 2016 at 10:26 AM</font><b><font size="5" face="Calibri"><br><br>To: </font></b><font size="5" face="Calibri">gpfsug main discussion list <</font><a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank"><u><font size="5" color="#0000FF" face="Calibri">gpfsug-discuss@spectrumscale.org</font></u></a><font size="5" face="Calibri">></font><b><font size="5" face="Calibri"><br>Subject: </font></b><font size="5" face="Calibri">Re: [gpfsug-discuss] Small cluster</font><font size="4"><br></font><font size="4" face="Calibri"><br>You can do FPO for non-Hadoop workloads. It just alters the disks below the GPFS filesystem layer and looks like a normal GPFS system (mostly). I do think there were some restrictions on non-FPO nodes mounting FPO filesystems via multi-cluster.. not sure if those are still there.. any input on that from IBM? </font><font size="4"><br></font><font size="4" face="Calibri"><br>If small enough data, and with 3-way replication, it might just be wise to do internal storage and 3x rep. A 36TB 2U server is ~$10K (just common throwing out numbers), 3 of those per site would fit in your budget.</font><font size="4"><br></font><font size="4" face="Calibri"><br>Again.. depending on your requirements, stability balance between 'science experiment' vs production, GPFS knowledge level, etc etc...</font><font size="4"><br></font><font size="4" face="Calibri"><br>This is actually an interesting and somewhat missing space for small enterprises. If you just want 10-20TB active-active online everywhere, say, for VMware, or NFS, or something else, there arent all that many good solutions today that scale down far enough and are a decent price. It's easy with many many PB, but small.. idk. I think the above sounds good as anything without going SAN-crazy.</font><font size="4"><br><br><br></font><font size="4" face="Calibri"><br>On Fri, Mar 4, 2016 at 11:21 AM, </font><a href="mailto:Mark.Bush@siriuscom.com" target="_blank"><u><font size="4" color="#0000FF" face="Calibri">Mark.Bush@siriuscom.com</font></u></a><font size="4" face="Calibri"> <</font><a href="mailto:Mark.Bush@siriuscom.com" target="_blank"><u><font size="4" color="#0000FF" face="Calibri">Mark.Bush@siriuscom.com</font></u></a><font size="4" face="Calibri">> wrote:<br>I guess this is really my question. Budget is less than $50k per site and they need around 20TB storage. Two nodes with MD3 or something may work. But could it work (and be successful) with just servers and internal drives? Should I do FPO for non hadoop like workloads? I didn’t think I could get native raid except in the ESS (GSS no longer exists if I remember correctly). Do I just make replicas and call it good?</font><font size="4"><br><br></font><font size="4" face="Calibri"><br>Mark</font><font size="4"><br></font><b><font size="5" face="Calibri"><br>From: </font></b><font size="5" face="Calibri"><</font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank"><u><font size="5" color="#0000FF" face="Calibri">gpfsug-discuss-bounces@spectrumscale.org</font></u></a><font size="5" face="Calibri">> on behalf of Marc A Kaplan <</font><a href="mailto:makaplan@us.ibm.com" target="_blank"><u><font size="5" color="#0000FF" face="Calibri">makaplan@us.ibm.com</font></u></a><font size="5" face="Calibri">></font><b><font size="5" face="Calibri"><br>Reply-To: </font></b><font size="5" face="Calibri">gpfsug main discussion list <</font><a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank"><u><font size="5" color="#0000FF" face="Calibri">gpfsug-discuss@spectrumscale.org</font></u></a><font size="5" face="Calibri">></font><b><font size="5" face="Calibri"><br>Date: </font></b><font size="5" face="Calibri">Friday, March 4, 2016 at 10:09 AM</font><b><font size="5" face="Calibri"><br>To: </font></b><font size="5" face="Calibri">gpfsug main discussion list <</font><a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank"><u><font size="5" color="#0000FF" face="Calibri">gpfsug-discuss@spectrumscale.org</font></u></a><font size="5" face="Calibri">></font><b><font size="5" face="Calibri"><br>Subject: </font></b><font size="5" face="Calibri">Re: [gpfsug-discuss] Small cluster</font><font size="4"><br><br>Jon, I don't doubt your experience, but it's not quite fair or even sensible to make a decision today based on what was available in the GPFS 2.3 era.<br><br>We are now at GPFS 4.2 with support for 3 way replication and FPO. <br>Also we have Raid controllers, IB, and "Native Raid" and ESS, GSS solutions and more.<br><br>So more choices, more options, making finding an "optimal" solution more difficult.<br><br>To begin with, as with any provisioning problem, one should try to state: requirements, goals, budgets, constraints, failure/tolerance models/assumptions,<br>expected workloads, desired performance, etc, etc.</font><font size="4" face="Calibri"><br></font><p><font face="Cambria">This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. </font><p><a href="http://www.siriuscom.com/" target="_blank"><b><u><font size="4" color="#0000FF" face="Calibri">Sirius Computer Solutions</font></u></b></a><font size="4" face="Calibri"> <br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="http://spectrumscale.org/" target="_blank"><u><font size="4" color="#0000FF" face="Calibri">spectrumscale.org</font></u></a><u><font size="4" color="#0000FF"><br></font></u><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><u><font size="4" color="#0000FF" face="Calibri">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></u></a><font size="4"><br></font><font size="4" face="Calibri"><br></font><font size="4"><br><br></font><font size="4" face="Calibri"><br>-- <br>Zach Giles</font><u><font size="4" color="#0000FF"><br></font></u><a href="mailto:zgiles@gmail.com" target="_blank"><u><font size="4" color="#0000FF" face="Calibri">zgiles@gmail.com</font></u></a><font size="5"><br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="http://spectrumscale.org/" target="_blank"><u><font size="5" color="#0000FF">spectrumscale.org</font></u></a><u><font size="4" color="#0000FF"><br></font></u><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><u><font size="5" color="#0000FF">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></u></a></ul></ul><font size="4"><br><br></font><font size="5"><br>-- <br>Zach Giles</font><u><font size="4" color="#0000FF"><br></font></u><a href="mailto:zgiles@gmail.com" target="_blank"><u><font size="5" color="#0000FF">zgiles@gmail.com</font></u></a><tt><font size="4">_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font></tt><a href="http://spectrumscale.org/" target="_blank"><tt><u><font size="4" color="#0000FF">spectrumscale.org</font></u></tt></a><tt><u><font size="4" color="#0000FF"><br></font></u></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><tt><u><font size="4" color="#0000FF">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></u></tt></a><font size="4"><br><br><br></font><br><font size="4">_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="http://spectrumscale.org/" target="_blank"><u><font size="4" color="#0000FF">spectrumscale.org</font></u></a><u><font size="4" color="#0000FF"><br></font></u><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><u><font size="4" color="#0000FF">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></u></a><u><font size="4" color="#0000FF">[attachment "graycol.gif" deleted by Sven Oehme/Almaden/IBM] </font></u><br><br></ul><BR>
</body></html>