<font size=3 face="Arial">My understanding is that with Spectrum Scale
5.0 there is no longer a standard edition, only data management and advanced,
and the pricing is all done via storage not sockets. Now there
may be some grandfathering for those with existing socket licenses but
I really do not know. My point is that data management is not the
same as advanced edition. Again I could be wrong because I tend not
to concern myself with how the product is licensed.</font><br><br><font size=3 face="sans-serif">Fred<br>__________________________________________________<br>Fred Stock | IBM Pittsburgh Lab | 720-430-8821<br>stockf@us.ibm.com</font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Stephen Ulmer <ulmer@ulmer.org></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug main discussion
list <gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">03/14/2018 03:06 PM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
Preferred NSD</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><font size=3>Depending on the size... I just quoted something both
ways and DME (which is Advanced Edition equivalent) was about $400K cheaper
than Standard Edition socket pricing for this particular customer and use
case. It all depends.</font><br><br><font size=3>Also, for the case where the OP wants to distribute the
file system around on NVMe in *every* node, there is always the FPO license.
The FPO license can share NSDs with other FPO licensed nodes and servers
(just not clients).</font><br><br><font size=3>-- <br>Stephen<br><br></font><br><br><font size=3>On Mar 14, 2018, at 1:33 PM, Sobey, Richard A <</font><a href=mailto:r.sobey@imperial.ac.uk><font size=3 color=blue><u>r.sobey@imperial.ac.uk</u></font></a><font size=3>>
wrote:</font><br><br><font size=3>2. Have data management edition and capacity license the
amount of storage.</font><br><font size=3>There goes the budget 😉<br><br>Richard<br><br>-----Original Message-----<br>From: </font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><font size=3 color=blue><u>gpfsug-discuss-bounces@spectrumscale.org</u></font></a><font size=3><</font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><font size=3 color=blue><u>gpfsug-discuss-bounces@spectrumscale.org</u></font></a><font size=3>>
On Behalf Of Simon Thompson (IT Research Support)<br>Sent: 14 March 2018 16:54<br>To: gpfsug main discussion list <</font><a href="mailto:gpfsug-discuss@spectrumscale.org"><font size=3 color=blue><u>gpfsug-discuss@spectrumscale.org</u></font></a><font size=3>><br>Subject: Re: [gpfsug-discuss] Preferred NSD<br><br>Not always true.<br><br>1. Use them with socket licenses as HAWC or LROC is OK on a client.<br>2. Have data management edition and capacity license the amount of storage.<br><br>Simon<br>________________________________________<br>From: </font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><font size=3 color=blue><u>gpfsug-discuss-bounces@spectrumscale.org</u></font></a><font size=3>[</font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><font size=3 color=blue><u>gpfsug-discuss-bounces@spectrumscale.org</u></font></a><font size=3>]
on behalf of Jeffrey R. Lang [</font><a href=mailto:JRLang@uwyo.edu><font size=3 color=blue><u>JRLang@uwyo.edu</u></font></a><font size=3>]<br>Sent: 14 March 2018 14:11<br>To: gpfsug main discussion list<br>Subject: Re: [gpfsug-discuss] Preferred NSD<br><br>Something I haven't heard in this discussion, it that of licensing of GPFS.<br><br>I believe that once you export disks from a node it then becomes a server
node and the license may need to be changed, from client to server. There
goes the budget.<br><br><br><br>-----Original Message-----<br>From: </font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><font size=3 color=blue><u>gpfsug-discuss-bounces@spectrumscale.org</u></font></a><font size=3><</font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><font size=3 color=blue><u>gpfsug-discuss-bounces@spectrumscale.org</u></font></a><font size=3>>
On Behalf Of Lukas Hejtmanek<br>Sent: Wednesday, March 14, 2018 4:28 AM<br>To: gpfsug main discussion list <</font><a href="mailto:gpfsug-discuss@spectrumscale.org"><font size=3 color=blue><u>gpfsug-discuss@spectrumscale.org</u></font></a><font size=3>><br>Subject: Re: [gpfsug-discuss] Preferred NSD<br><br>Hello,<br><br>thank you for insight. Well, the point is, that I will get ~60 with 120
NVMe disks in it, each about 2TB size. It means that I will have 240TB
in NVMe SSD that could build nice shared scratch. Moreover, I have no different
HW or place to put these SSDs into. They have to be in the compute nodes.<br><br>On Tue, Mar 13, 2018 at 10:48:21AM -0700, Alex Chekholko wrote:</font><br><font size=3>I would like to discourage you from building a large distributed
<br>clustered filesystem made of many unreliable components. You will
<br>need to overprovision your interconnect and will also spend a lot of <br>time in "healing" or "degraded" state.<br><br>It is typically cheaper to centralize the storage into a subset of <br>nodes and configure those to be more highly available. E.g. of your<br>60 nodes, take 8 and put all the storage into those and make that a <br>dedicated GPFS cluster with no compute jobs on those nodes. Again,
<br>you'll still need really beefy and reliable interconnect to make this work.<br><br>Stepping back; what is the actual problem you're trying to solve? I
<br>have certainly been in that situation before, where the problem is <br>more like: "I have a fixed hardware configuration that I can't change,
<br>and I want to try to shoehorn a parallel filesystem onto that."<br><br>I would recommend looking closer at your actual workloads. If this
is <br>a "scratch" filesystem and file access is mostly from one node
at a <br>time, it's not very useful to make two additional copies of that data <br>on other nodes, and it will only slow you down.<br><br>Regards,<br>Alex<br><br>On Tue, Mar 13, 2018 at 7:16 AM, Lukas Hejtmanek <br><</font><a href=mailto:xhejtman@ics.muni.cz><font size=3 color=blue><u>xhejtman@ics.muni.cz</u></font></a><font size=3>><br>wrote:<br></font><br><font size=3>On Tue, Mar 13, 2018 at 10:37:43AM +0000, John Hearns
wrote:</font><br><font size=3>Lukas,<br>It looks like you are proposing a setup which uses your compute <br>servers</font><br><font size=3>as storage servers also?<br><br>yes, exactly. I would like to utilise NVMe SSDs that are in every <br>compute servers.. Using them as a shared scratch area with GPFS is <br>one of the options.<br></font><br><font size=3><br> * I'm thinking about the following setup:<br>~ 60 nodes, each with two enterprise NVMe SSDs, FDR IB <br>interconnected<br><br>There is nothing wrong with this concept, for instance see </font><font size=3 color=blue><u><br></u></font><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__www.beegfs.io_wiki_BeeOND&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=kB88vNQV9x5UFOu3tBxpRKmS3rSCi68KIBxOa_D5ji8&s=hkQw7geQgSpunsoZykLw_1bssNIeqzdGTV81d7nKfjk&e="><font size=3 color=blue><u>https://www.beegfs.io/wiki/BeeOND</u></font></a><font size=3><br><br>I have an NVMe filesystem which uses 60 drives, but there are 10 servers.<br>You should look at "failure zones" also.</font><br><font size=3><br>you still need the storage servers and local SSDs to use only for <br>caching, do I understand correctly?<br></font><br><font size=3><br>From: </font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><font size=3 color=blue><u>gpfsug-discuss-bounces@spectrumscale.org</u></font></a><font size=3><br>[</font><a href="mailto:gpfsug-discuss-"><font size=3>mailto:gpfsug-discuss-</font></a><br><a href=mailto:bounces@spectrumscale.org><font size=3 color=blue><u>bounces@spectrumscale.org</u></font></a><font size=3>]
On Behalf Of Knister, Aaron S.<br>(GSFC-606.2)[COMPUTER SCIENCE CORP]</font><br><font size=3>Sent: Monday, March 12, 2018 4:14 PM<br>To: gpfsug main discussion list <</font><a href="mailto:gpfsug-discuss@spectrumscale.org"><font size=3 color=blue><u>gpfsug-discuss@spectrumscale.org</u></font></a><font size=3>><br>Subject: Re: [gpfsug-discuss] Preferred NSD<br><br>Hi Lukas,<br><br>Check out FPO mode. That mimics Hadoop's data placement features.<br>You</font><br><font size=3>can have up to 3 replicas both data and metadata but still
the <br>downside, though, as you say is the wrong node failures will take your
cluster down.</font><br><font size=3><br>You might want to check out something like Excelero's NVMesh<br>(note: not</font><br><font size=3>an endorsement since I can't give such things) which can
create <br>logical volumes across all your NVMe drives. The product has erasure <br>coding on their roadmap. I'm not sure if they've released that <br>feature yet but in theory it will give better fault tolerance *and* <br>you'll get more efficient usage of your SSDs.</font><br><font size=3><br>I'm sure there are other ways to skin this cat too.<br><br>-Aaron<br><br><br><br>On March 12, 2018 at 10:59:35 EDT, Lukas Hejtmanek <br><</font><a href=mailto:xhejtman@ics.muni.cz><font size=3 color=blue><u>xhejtman@ics.muni.cz</u></font></a><br><font size=3><</font><a href=mailto:xhejtman@ics.muni.cz><font size=3 color=blue><u>mailto:xhejtman@ics.muni.cz</u></font></a><font size=3>>>
wrote:</font><br><font size=3>Hello,<br><br>I'm thinking about the following setup:<br>~ 60 nodes, each with two enterprise NVMe SSDs, FDR IB <br>interconnected<br><br>I would like to setup shared scratch area using GPFS and those <br>NVMe</font><br><font size=3>SSDs. Each</font><br><font size=3>SSDs as on NSD.<br><br>I don't think like 5 or more data/metadata replicas are practical here.</font><br><font size=3>On the</font><br><font size=3>other hand, multiple node failures is something really
expected.<br><br>Is there a way to instrument that local NSD is strongly preferred <br>to</font><br><font size=3>store</font><br><font size=3>data? I.e. node failure most probably does not result
in <br>unavailable</font><br><font size=3>data for</font><br><font size=3>the other nodes?<br><br>Or is there any other recommendation/solution to build shared <br>scratch</font><br><font size=3>with</font><br><font size=3>GPFS in such setup? (Do not do it including.)<br><br>--<br>Lukáš Hejtmánek<br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=kB88vNQV9x5UFOu3tBxpRKmS3rSCi68KIBxOa_D5ji8&s=xJpDE3UcdEsj91sv0G7HDhQdZKFoEHDQ4loeRZJlPF8&e="><font size=3 color=blue><u>spectrumscale.org</u></font></a><font size=3></font><font size=3 color=blue><u><br></u></font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=kB88vNQV9x5UFOu3tBxpRKmS3rSCi68KIBxOa_D5ji8&s=R9wxUL1IMkjtWZsFkSAXRUmuKi8uS1jpQRYVTvOYq3g&e="><font size=3 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></a><font size=3><br>-- The information contained in this communication and any <br>attachments</font><br><font size=3>is confidential and may be privileged, and is for the
sole use of <br>the intended recipient(s). Any unauthorized review, use, disclosure <br>or distribution is prohibited. Unless explicitly stated otherwise in <br>the body of this communication or the attachment thereto (if any), <br>the information is provided on an AS-IS basis without any express or <br>implied warranties or liabilities. To the extent you are relying on <br>this information, you are doing so at your own risk. If you are not <br>the intended recipient, please notify the sender immediately by <br>replying to this message and destroy all copies of this message and <br>any attachments. Neither the sender nor the company/group of <br>companies he or she represents shall be liable for the proper and <br>complete transmission of the information contained in this communication,
or for any delay in its receipt.<br></font><br><font size=3>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=kB88vNQV9x5UFOu3tBxpRKmS3rSCi68KIBxOa_D5ji8&s=xJpDE3UcdEsj91sv0G7HDhQdZKFoEHDQ4loeRZJlPF8&e="><font size=3 color=blue><u>spectrumscale.org</u></font></a><font size=3></font><font size=3 color=blue><u><br></u></font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=kB88vNQV9x5UFOu3tBxpRKmS3rSCi68KIBxOa_D5ji8&s=R9wxUL1IMkjtWZsFkSAXRUmuKi8uS1jpQRYVTvOYq3g&e="><font size=3 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></a><br><font size=3><br><br>--<br>Lukáš Hejtmánek<br><br>Linux Administrator only because<br> Full Time Multitasking Ninja<br> is not an official job title<br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=kB88vNQV9x5UFOu3tBxpRKmS3rSCi68KIBxOa_D5ji8&s=xJpDE3UcdEsj91sv0G7HDhQdZKFoEHDQ4loeRZJlPF8&e="><font size=3 color=blue><u>spectrumscale.org</u></font></a><font size=3 color=blue><u><br></u></font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=kB88vNQV9x5UFOu3tBxpRKmS3rSCi68KIBxOa_D5ji8&s=R9wxUL1IMkjtWZsFkSAXRUmuKi8uS1jpQRYVTvOYq3g&e="><font size=3 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></a><font size=3><br></font><br><br><font size=3>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=kB88vNQV9x5UFOu3tBxpRKmS3rSCi68KIBxOa_D5ji8&s=xJpDE3UcdEsj91sv0G7HDhQdZKFoEHDQ4loeRZJlPF8&e="><font size=3 color=blue><u>spectrumscale.org</u></font></a><font size=3 color=blue><u><br></u></font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=kB88vNQV9x5UFOu3tBxpRKmS3rSCi68KIBxOa_D5ji8&s=R9wxUL1IMkjtWZsFkSAXRUmuKi8uS1jpQRYVTvOYq3g&e="><font size=3 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></a><br><font size=3><br><br>--<br>Lukáš Hejtmánek<br><br>Linux Administrator only because<br> Full Time Multitasking Ninja<br> is not an official job title<br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=kB88vNQV9x5UFOu3tBxpRKmS3rSCi68KIBxOa_D5ji8&s=xJpDE3UcdEsj91sv0G7HDhQdZKFoEHDQ4loeRZJlPF8&e="><font size=3 color=blue><u>spectrumscale.org</u></font></a><font size=3 color=blue><u><br></u></font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=kB88vNQV9x5UFOu3tBxpRKmS3rSCi68KIBxOa_D5ji8&s=R9wxUL1IMkjtWZsFkSAXRUmuKi8uS1jpQRYVTvOYq3g&e="><font size=3 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></a><font size=3><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=kB88vNQV9x5UFOu3tBxpRKmS3rSCi68KIBxOa_D5ji8&s=xJpDE3UcdEsj91sv0G7HDhQdZKFoEHDQ4loeRZJlPF8&e="><font size=3 color=blue><u>spectrumscale.org</u></font></a><font size=3 color=blue><u><br></u></font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=kB88vNQV9x5UFOu3tBxpRKmS3rSCi68KIBxOa_D5ji8&s=R9wxUL1IMkjtWZsFkSAXRUmuKi8uS1jpQRYVTvOYq3g&e="><font size=3 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></a><font size=3><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><font size=3>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></a><font size=3><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><font size=3>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></a><br><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=kB88vNQV9x5UFOu3tBxpRKmS3rSCi68KIBxOa_D5ji8&s=R9wxUL1IMkjtWZsFkSAXRUmuKi8uS1jpQRYVTvOYq3g&e="><tt><font size=2>https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=kB88vNQV9x5UFOu3tBxpRKmS3rSCi68KIBxOa_D5ji8&s=R9wxUL1IMkjtWZsFkSAXRUmuKi8uS1jpQRYVTvOYq3g&e=</font></tt></a><tt><font size=2><br></font></tt><br><br><BR>