<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Depending on the size... I just quoted something both ways and DME (which is Advanced Edition equivalent) was about $400K cheaper than Standard Edition socket pricing for this particular customer and use case. It all depends.<div class=""><br class=""></div><div class="">Also, for the case where the OP wants to distribute the file system around on NVMe in *every* node, there is always the FPO license. The FPO license can share NSDs with other FPO licensed nodes and servers (just not clients).</div><div class=""><br class=""><div class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">-- <br class="">Stephen<br class=""><br class=""><br class=""></div>
</div>
<div><br class=""><blockquote type="cite" class=""><div class="">On Mar 14, 2018, at 1:33 PM, Sobey, Richard A <<a href="mailto:r.sobey@imperial.ac.uk" class="">r.sobey@imperial.ac.uk</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class=""><blockquote type="cite" class=""><blockquote type="cite" class="">2. Have data management edition and capacity license the amount of storage.<br class=""></blockquote></blockquote>There goes the budget 😉<br class=""><br class="">Richard<br class=""><br class="">-----Original Message-----<br class="">From: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" class="">gpfsug-discuss-bounces@spectrumscale.org</a> <<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" class="">gpfsug-discuss-bounces@spectrumscale.org</a>> On Behalf Of Simon Thompson (IT Research Support)<br class="">Sent: 14 March 2018 16:54<br class="">To: gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" class="">gpfsug-discuss@spectrumscale.org</a>><br class="">Subject: Re: [gpfsug-discuss] Preferred NSD<br class=""><br class="">Not always true.<br class=""><br class="">1. Use them with socket licenses as HAWC or LROC is OK on a client.<br class="">2. Have data management edition and capacity license the amount of storage.<br class=""><br class="">Simon<br class="">________________________________________<br class="">From: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" class="">gpfsug-discuss-bounces@spectrumscale.org</a> [<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" class="">gpfsug-discuss-bounces@spectrumscale.org</a>] on behalf of Jeffrey R. Lang [<a href="mailto:JRLang@uwyo.edu" class="">JRLang@uwyo.edu</a>]<br class="">Sent: 14 March 2018 14:11<br class="">To: gpfsug main discussion list<br class="">Subject: Re: [gpfsug-discuss] Preferred NSD<br class=""><br class="">Something I haven't heard in this discussion, it that of licensing of GPFS.<br class=""><br class="">I believe that once you export disks from a node it then becomes a server node and the license may need to be changed, from client to server. There goes the budget.<br class=""><br class=""><br class=""><br class="">-----Original Message-----<br class="">From: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" class="">gpfsug-discuss-bounces@spectrumscale.org</a> <<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" class="">gpfsug-discuss-bounces@spectrumscale.org</a>> On Behalf Of Lukas Hejtmanek<br class="">Sent: Wednesday, March 14, 2018 4:28 AM<br class="">To: gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" class="">gpfsug-discuss@spectrumscale.org</a>><br class="">Subject: Re: [gpfsug-discuss] Preferred NSD<br class=""><br class="">Hello,<br class=""><br class="">thank you for insight. Well, the point is, that I will get ~60 with 120 NVMe disks in it, each about 2TB size. It means that I will have 240TB in NVMe SSD that could build nice shared scratch. Moreover, I have no different HW or place to put these SSDs into. They have to be in the compute nodes.<br class=""><br class="">On Tue, Mar 13, 2018 at 10:48:21AM -0700, Alex Chekholko wrote:<br class=""><blockquote type="cite" class="">I would like to discourage you from building a large distributed <br class="">clustered filesystem made of many unreliable components. You will <br class="">need to overprovision your interconnect and will also spend a lot of <br class="">time in "healing" or "degraded" state.<br class=""><br class="">It is typically cheaper to centralize the storage into a subset of <br class="">nodes and configure those to be more highly available. E.g. of your<br class="">60 nodes, take 8 and put all the storage into those and make that a <br class="">dedicated GPFS cluster with no compute jobs on those nodes. Again, <br class="">you'll still need really beefy and reliable interconnect to make this work.<br class=""><br class="">Stepping back; what is the actual problem you're trying to solve? I <br class="">have certainly been in that situation before, where the problem is <br class="">more like: "I have a fixed hardware configuration that I can't change, <br class="">and I want to try to shoehorn a parallel filesystem onto that."<br class=""><br class="">I would recommend looking closer at your actual workloads. If this is <br class="">a "scratch" filesystem and file access is mostly from one node at a <br class="">time, it's not very useful to make two additional copies of that data <br class="">on other nodes, and it will only slow you down.<br class=""><br class="">Regards,<br class="">Alex<br class=""><br class="">On Tue, Mar 13, 2018 at 7:16 AM, Lukas Hejtmanek <br class=""><<a href="mailto:xhejtman@ics.muni.cz" class="">xhejtman@ics.muni.cz</a>><br class="">wrote:<br class=""><br class=""><blockquote type="cite" class="">On Tue, Mar 13, 2018 at 10:37:43AM +0000, John Hearns wrote:<br class=""><blockquote type="cite" class="">Lukas,<br class="">It looks like you are proposing a setup which uses your compute <br class="">servers<br class=""></blockquote>as storage servers also?<br class=""><br class="">yes, exactly. I would like to utilise NVMe SSDs that are in every <br class="">compute servers.. Using them as a shared scratch area with GPFS is <br class="">one of the options.<br class=""><br class=""><blockquote type="cite" class=""><br class=""> * I'm thinking about the following setup:<br class="">~ 60 nodes, each with two enterprise NVMe SSDs, FDR IB <br class="">interconnected<br class=""><br class="">There is nothing wrong with this concept, for instance see <br class=""><a href="https://www.beegfs.io/wiki/BeeOND" class="">https://www.beegfs.io/wiki/BeeOND</a><br class=""><br class="">I have an NVMe filesystem which uses 60 drives, but there are 10 servers.<br class="">You should look at "failure zones" also.<br class=""></blockquote><br class="">you still need the storage servers and local SSDs to use only for <br class="">caching, do I understand correctly?<br class=""><br class=""><blockquote type="cite" class=""><br class="">From: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" class="">gpfsug-discuss-bounces@spectrumscale.org</a><br class="">[mailto:gpfsug-discuss-<br class=""></blockquote><a href="mailto:bounces@spectrumscale.org" class="">bounces@spectrumscale.org</a>] On Behalf Of Knister, Aaron S.<br class="">(GSFC-606.2)[COMPUTER SCIENCE CORP]<br class=""><blockquote type="cite" class="">Sent: Monday, March 12, 2018 4:14 PM<br class="">To: gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" class="">gpfsug-discuss@spectrumscale.org</a>><br class="">Subject: Re: [gpfsug-discuss] Preferred NSD<br class=""><br class="">Hi Lukas,<br class=""><br class="">Check out FPO mode. That mimics Hadoop's data placement features.<br class="">You<br class=""></blockquote>can have up to 3 replicas both data and metadata but still the <br class="">downside, though, as you say is the wrong node failures will take your cluster down.<br class=""><blockquote type="cite" class=""><br class="">You might want to check out something like Excelero's NVMesh<br class="">(note: not<br class=""></blockquote>an endorsement since I can't give such things) which can create <br class="">logical volumes across all your NVMe drives. The product has erasure <br class="">coding on their roadmap. I'm not sure if they've released that <br class="">feature yet but in theory it will give better fault tolerance *and* <br class="">you'll get more efficient usage of your SSDs.<br class=""><blockquote type="cite" class=""><br class="">I'm sure there are other ways to skin this cat too.<br class=""><br class="">-Aaron<br class=""><br class=""><br class=""><br class="">On March 12, 2018 at 10:59:35 EDT, Lukas Hejtmanek <br class=""><<a href="mailto:xhejtman@ics.muni.cz" class="">xhejtman@ics.muni.cz</a><br class=""></blockquote><<a href="mailto:xhejtman@ics.muni.cz" class="">mailto:xhejtman@ics.muni.cz</a>>> wrote:<br class=""><blockquote type="cite" class="">Hello,<br class=""><br class="">I'm thinking about the following setup:<br class="">~ 60 nodes, each with two enterprise NVMe SSDs, FDR IB <br class="">interconnected<br class=""><br class="">I would like to setup shared scratch area using GPFS and those <br class="">NVMe<br class=""></blockquote>SSDs. Each<br class=""><blockquote type="cite" class="">SSDs as on NSD.<br class=""><br class="">I don't think like 5 or more data/metadata replicas are practical here.<br class=""></blockquote>On the<br class=""><blockquote type="cite" class="">other hand, multiple node failures is something really expected.<br class=""><br class="">Is there a way to instrument that local NSD is strongly preferred <br class="">to<br class=""></blockquote>store<br class=""><blockquote type="cite" class="">data? I.e. node failure most probably does not result in <br class="">unavailable<br class=""></blockquote>data for<br class=""><blockquote type="cite" class="">the other nodes?<br class=""><br class="">Or is there any other recommendation/solution to build shared <br class="">scratch<br class=""></blockquote>with<br class=""><blockquote type="cite" class="">GPFS in such setup? (Do not do it including.)<br class=""><br class="">--<br class="">Lukáš Hejtmánek<br class="">_______________________________________________<br class="">gpfsug-discuss mailing list<br class="">gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a> <br class=""><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="">-- The information contained in this communication and any <br class="">attachments<br class=""></blockquote>is confidential and may be privileged, and is for the sole use of <br class="">the intended recipient(s). Any unauthorized review, use, disclosure <br class="">or distribution is prohibited. Unless explicitly stated otherwise in <br class="">the body of this communication or the attachment thereto (if any), <br class="">the information is provided on an AS-IS basis without any express or <br class="">implied warranties or liabilities. To the extent you are relying on <br class="">this information, you are doing so at your own risk. If you are not <br class="">the intended recipient, please notify the sender immediately by <br class="">replying to this message and destroy all copies of this message and <br class="">any attachments. Neither the sender nor the company/group of <br class="">companies he or she represents shall be liable for the proper and <br class="">complete transmission of the information contained in this communication, or for any delay in its receipt.<br class=""><br class=""><blockquote type="cite" class="">_______________________________________________<br class="">gpfsug-discuss mailing list<br class="">gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a> <br class=""><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class=""></blockquote><br class=""><br class="">--<br class="">Lukáš Hejtmánek<br class=""><br class="">Linux Administrator only because<br class=""> Full Time Multitasking Ninja<br class=""> is not an official job title<br class="">_______________________________________________<br class="">gpfsug-discuss mailing list<br class="">gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a><br class=""><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class=""><br class=""></blockquote></blockquote><br class=""><blockquote type="cite" class="">_______________________________________________<br class="">gpfsug-discuss mailing list<br class="">gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a><br class=""><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class=""></blockquote><br class=""><br class="">--<br class="">Lukáš Hejtmánek<br class=""><br class="">Linux Administrator only because<br class=""> Full Time Multitasking Ninja<br class=""> is not an official job title<br class="">_______________________________________________<br class="">gpfsug-discuss mailing list<br class="">gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a><br class=""><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="">_______________________________________________<br class="">gpfsug-discuss mailing list<br class="">gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a><br class=""><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="">_______________________________________________<br class="">gpfsug-discuss mailing list<br class="">gpfsug-discuss at spectrumscale.org<br class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss<br class="">_______________________________________________<br class="">gpfsug-discuss mailing list<br class="">gpfsug-discuss at spectrumscale.org<br class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss<br class=""></div></div></blockquote></div><br class=""></div></body></html>