<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">OK, so the ECE recovery group is the four NSD servers with the System storage pool disks, and somehow I have to read the docs<div class="">and find out how to define pdisks that spread the replication across the four servers, but three disks at a time. </div><div class="">Three pdisks of 7 drives, three I can't do anything with, or are those for rebuilding space?</div><div class=""><br class=""></div><div class="">Can you provide me details of your six-node non-ECE configuration? Basically how the NSDs are defined...</div><div class=""><br class=""></div><div class="">The remainder of our new filesystem will have a fast pool of 12 nodes of excelero, and 2Pb of spinning disks, so another possibility</div><div class="">would be to license four more nodes and put the system pool under excelero. </div><div class=""><br class=""></div><div class=""> -- ddj</div><div class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Jul 30, 2019, at 8:19 AM, Sanchez, Paul <<a href="mailto:Paul.Sanchez@deshaw.com" class="">Paul.Sanchez@deshaw.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">Hi David,<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">In an ECE configuration, it would be typical to put all of the NVMe disks in all 4 of your servers into a single recovery group. So in your case, all 24 NVMe drives would be in one recovery group and the 4 servers would be “log group” servers in the recovery group, distributing the I/O load for the NSD/vdisks that are hosted on the RG. (The minimum disks for a single RG config is 12, and you meet that easily.)<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><a href="https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_plan_recommendations.htm" style="color: purple; text-decoration: underline;" class="">https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_plan_recommendations.htm</a><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""><o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">outlines the recommendations for raidCode protection. Your configuration (4 nodes) would use vdisks with 4+3P, which gives you a slightly better capacity yield than RAID10 would, but with much better recovery characteristics:<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 12pt; font-family: "Times New Roman", serif; text-indent: -0.25in;" class=""><span style="font-size: 11pt; font-family: Symbol; color: rgb(31, 73, 125);" class=""><span class="">·<span style="font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: "Times New Roman";" class=""> <span class="Apple-converted-space"> </span></span></span></span><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">No single failed node will result in a down system NSD.<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 12pt; font-family: "Times New Roman", serif; text-indent: -0.25in;" class=""><span style="font-size: 11pt; font-family: Symbol; color: rgb(31, 73, 125);" class=""><span class="">·<span style="font-style: normal; font-variant-caps: normal; font-weight: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: "Times New Roman";" class=""> <span class="Apple-converted-space"> </span></span></span></span><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">No single drive failure will require a critical priority rebuild, and can be handled in the background without killing performance.<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">So from that perspective, ECE is a win here and avoids a problem with the non-ECE, shared-nothing designs: the manual “mmchdisk <fsname> start -a” operation that is needed after any traditional shared-nothing metadata NSD goes offline to bring it back and protect against further failures.<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">Despite the operational challenges of the non-ECE design, it can sometimes survive two server failures (if replication factor is 3 and the filesystem descriptor quorum wasn’t lost by the two failures) which a 4 node ECE cluster cannot. Given that the world is complex and unexpected things can happen, I’d personally recommend<span class="Apple-converted-space"> </span><b class="">redistributing the 24 disks across 6 servers</b><span class="Apple-converted-space"> </span>if you can, so that the design could always survive 2 node failures. I’ve run this design and it’s fairly robust.<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">In any event, you should of course test the failure scenarios yourself before going into production to validate them and familiarize yourself with the process. And a special note on ECE: due to the cooperative nature at the pdisk level, the network between the servers in the RG should be as reliable as possible and any network redundancy should also be tested ahead of time.<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">-Paul<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""><o:p class=""> </o:p></span></div><div class=""><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(225, 225, 225); padding: 3pt 0in 0in;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><b class=""><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">From:</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class="Apple-converted-space"> </span><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" style="color: purple; text-decoration: underline;" class="">gpfsug-discuss-bounces@spectrumscale.org</a><span class="Apple-converted-space"> </span><<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" style="color: purple; text-decoration: underline;" class="">gpfsug-discuss-bounces@spectrumscale.org</a>><span class="Apple-converted-space"> </span><b class="">On Behalf Of<span class="Apple-converted-space"> </span></b>David Johnson<br class=""><b class="">Sent:</b><span class="Apple-converted-space"> </span>Tuesday, July 30, 2019 7:46 AM<br class=""><b class="">To:</b><span class="Apple-converted-space"> </span>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" style="color: purple; text-decoration: underline;" class="">gpfsug-discuss@spectrumscale.org</a>><br class=""><b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: [gpfsug-discuss] Building GPFS filesystem system data pool on shared nothing NVMe drives<o:p class=""></o:p></span></div></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div><p style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><span style="font-size: 9pt; font-family: Verdana, sans-serif; color: rgb(204, 0, 0);" class="">This message was sent by an external party.<o:p class=""></o:p></span></p><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">Can we confirm the requirement for disks per RG? I have 4 RG, but only 6 x 3TB NVMe drives per box.<o:p class=""></o:p></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><br class=""><br class=""><o:p class=""></o:p></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">On Jul 29, 2019, at 1:34 PM, Luis Bolinches <<a href="mailto:luis.bolinches@fi.ibm.com" style="color: purple; text-decoration: underline;" class="">luis.bolinches@fi.ibm.com</a>> wrote:<o:p class=""></o:p></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">Hi, from phone so sorry for typos. <span class="Apple-converted-space"> </span><o:p class=""></o:p></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">I really think you should look into Spectrum Scale Erasure Code Edition (ECE) for this. <o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">Sure you could do a RAID on each node as you mention here but that sounds like a lot of waste to me on storage capacity. Not to forget you get other goodies like end to end checksum and rapid rebuilds with ECE, among others. <o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">Four servers is the minimum requirement for ECE (4+3p) and from top of my head 12 disk per RG, you are fine with both requirements. <br class=""><br class="">There is a presentation on ECE on the user group web page from London May 2019 were we talk about ECE. <o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">And the ibm page of the product <a href="https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_intro.htm" style="color: purple; text-decoration: underline;" class="">https://www.ibm.com/support/knowledgecenter/STXKQY_ECE_5.0.3/com.ibm.spectrum.scale.ece.v5r03.doc/b1lece_intro.htm</a><o:p class=""></o:p></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">--<span class="Apple-converted-space"> </span><o:p class=""></o:p></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">Cheers<o:p class=""></o:p></div></div></div><div class=""><p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 12pt; font-family: "Times New Roman", serif;"><br class="">El 29 jul 2019, a las 19:06, David Johnson <<a href="mailto:david_johnson@brown.edu" style="color: purple; text-decoration: underline;" class="">david_johnson@brown.edu</a>> escribió:<o:p class=""></o:p></p></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 12pt; font-family: "Times New Roman", serif;">We are planning a 5.0.x upgrade onto new hardware to make use of the new 5.x GPFS features.<br class="">The goal is to use up to four NSD nodes for metadata, each one with 6 NVMe drives (to be determined<br class="">whether we use Intel VROC for raid 5 or raid 1, or just straight disks). <br class=""><br class="">So questions —<span class="Apple-converted-space"> </span><br class="">Has anyone done system pool on shared nothing cluster? How did you set it up?<br class="">With default metadata replication set at 3, can you make use of four NSD nodes effectively?<br class="">How would one design the location vectors and failure groups so that the system metadata is<br class="">spread evenly across the four servers?<br class=""><br class="">Thanks,<br class="">— ddj<br class="">Dave Johnson<br class="">_______________________________________________<br class="">gpfsug-discuss mailing list<br class="">gpfsug-discuss at<span class="Apple-converted-space"> </span><a href="http://spectrumscale.org/" style="color: purple; text-decoration: underline;" class="">spectrumscale.org</a><br class=""><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" style="color: purple; text-decoration: underline;" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><o:p class=""></o:p></p></div></blockquote></div><p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 12pt; font-family: "Times New Roman", serif;"><br class="">Ellei edellä ole toisin mainittu: / Unless stated otherwise above:<br class="">Oy IBM Finland Ab<br class="">PL 265, 00101 Helsinki, Finland<br class="">Business ID, Y-tunnus: 0195876-3<span class="Apple-converted-space"> </span><br class="">Registered in Finland<o:p class=""></o:p></p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class="">_______________________________________________<br class="">gpfsug-discuss mailing list<br class="">gpfsug-discuss at<span class="Apple-converted-space"> </span><a href="http://spectrumscale.org/" style="color: purple; text-decoration: underline;" class="">spectrumscale.org</a><br class=""><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" style="color: purple; text-decoration: underline;" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><o:p class=""></o:p></div></div></blockquote></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: "Times New Roman", serif;" class=""><o:p class=""> </o:p></div></div></div><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">_______________________________________________</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">gpfsug-discuss mailing list</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">gpfsug-discuss at<span class="Apple-converted-space"> </span></span><a href="http://spectrumscale.org/" style="color: purple; text-decoration: underline; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">spectrumscale.org</a><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" style="color: purple; text-decoration: underline; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></div></blockquote></div><br class=""></div></body></html>