<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div dir="ltr" >Ken Hill wrote:</div>
<div dir="ltr" >> <font size="2" face="sans-serif" >Hello Stephen,</font><br>><br><font size="2" face="sans-serif" >>There are three licensing models for Spectrum Scale | GPFS:</font><br>><br><font size="2" face="sans-serif" >>Server</font><br><font size="2" face="sans-serif" >>FPO</font><br><font size="2" face="sans-serif" >>Client</font><br>><br><font size="2" face="sans-serif" >>think the thing you might be missing is the associated cost per function. </font></div>
<div dir="ltr" ><br>What adds to this debate is that there is now (>=4.2.2) a new licensing model available that is capacity based rather than socket based.</div>
<div dir="ltr" > </div>
<div dir="ltr" >So previously Server:FPO:Client licenses where in the approximate ratio of 100:10:1 in Dollar cost list price.</div>
<div dir="ltr" >This heavily skewed the way people designed their storage subsystems : to minimise the number of relatively expensive server licenses.</div>
<div dir="ltr" >This was also complicated since FPO licenses although 10x cheaper that server licenses had the restriction that FPO nodes were not allowed to serve the GPFS filesystem to nodes only licensed as GPFS clients.</div>
<div dir="ltr" > </div>
<div dir="ltr" >The new volume based licensing option is I agree quite pricey per TB at first sight, but it could make some configuration choice, a lot cheaper than they used to be under the Client:FPO:Server model.</div>
<div dir="ltr" > </div>
<div dir="ltr" >So do other since significant changes in system design if using volume based licensing? eg adding some Protocol nodes for NFS./CIFS would be at zero extra software cost.<br> </div>
<div dir="ltr" ><div class="mail-signature-container" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family: Arial; font-size: 10.5pt;" ><div class="socmaildefaultfont" dir="ltr" style="font-family: Arial; font-size: 10.5pt;" ><div class="socmaildefaultfont" dir="ltr" style="font-family: Arial; font-size: 10.5pt;" ><div class="socmaildefaultfont" dir="ltr" style="font-family: Arial; font-size: 10.5pt;" ><div class="socmaildefaultfont" dir="ltr" style="font-family: Arial; font-size: 10.5pt;" ><div class="socmaildefaultfont" dir="ltr" style="font-family: Arial; font-size: 10.5pt;" ><div dir="ltr" style="margin-top: 20px;" ><div style="font-family: sans-serif; font-size: 8pt; margin-top: 10px;" ><div style="margin-bottom:0cm;margin-bottom:.0001pt;line-height:normal;" ><span style="font-size:1.143em;" ><span style="font-family: Verdana,Geneva,sans-serif;" >Daniel</span></span></div>
<div style="margin-bottom:0cm;margin-bottom:.0001pt;line-height:normal;" ><img alt="/spectrum_storage-banne" src="" src="http://ausgsa.ibm.com/projects/t/tivoli_visual_design/public/2015/Spectrum-Storage/Email-signatures/Storage/spectrum_storage-banner.png" style="width: 601px; height: 5px;" ></div>
<div style="margin-bottom:0cm;margin-bottom:.0001pt;line-height:normal;" ><br> </div>
<table cellpadding="0" cellspacing="0" border="0" > <tbody> <tr> <td style="width:201px;padding:0cm 0cm 0cm 0cm;" > <div style="margin-bottom:0cm;margin-bottom:.0001pt;line-height:normal;" ><img alt="Spectrum Scale Logo" src="" src="http://ausgsa.ibm.com/projects/t/tivoli_visual_design/public/2015/Spectrum-Storage/Email-signatures/Storage/spectrum_scale-logo.png" style="width: 75px; height: 120px; float: left;" ></div>
<div style="margin-bottom:0cm;margin-bottom:.0001pt;line-height:normal;" > </div> </td> <td style="width:21px;padding:0cm 0cm 0cm 0cm;" > </td> <td style="width:202px;padding:0cm 0cm 0cm 0cm;" > <div style="margin-bottom:0cm;margin-bottom:.0001pt;line-height:normal;" ><strong><span style="font-family:Arial, Helvetica, sans-serif" ><span style="font-size:10.0pt;" >Dr Daniel Kidger</span></span></strong><br> <span style="font-family:Arial, Helvetica, sans-serif" ><span style="font-size:7.5pt;" >IBM Technical Sales Specialist<br> Software Defined Solution Sales<br> <br> +</span></span><span style="color:#5F5F5F;" ><span style="font-family:Verdana, Geneva, sans-serif" ><span style="font-size:10.0pt;" >44-(0)7818 522 266 </span></span></span><br> <span style="color:#5F5F5F;" ><span style="font-family:Arial, Helvetica, sans-serif" ><span style="font-size:8.0pt;" >daniel.kidger@uk.ibm.com</span></span></span></div> </td> </tr> </tbody></table><font size="2" face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" > </font></div></div></div></div></div></div></div></div></div></div></div></div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Original message -----<br>From: Zachary Giles <zgiles@gmail.com><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Cc:<br>Subject: Re: [gpfsug-discuss] Strategies - servers with local SAS disks<br>Date: Thu, Dec 1, 2016 4:34 AM<br>
<div dir="ltr" >Aaron, Thanks for jumping onboard. It's nice to see others confirming this. Sometimes I feel alone on this topic.
<div>
<div>It's should also be possible to use ZFS with ZVOLs presented as block devices for a backing store for NSDs. I'm not claiming it's stable, nor a good idea, nor performant.. but should be possible. :) There are various reports about it. Might be at least worth looking in to compared to Linux "md raid" if one truly needs an all-software solution that already exists. Something to think about and test over.</div></div></div>
<div>
<div>On Wed, Nov 30, 2016 at 11:15 PM, Aaron Knister <span dir="ltr" ><<a href="mailto:aaron.s.knister@nasa.gov" target="_blank" >aaron.s.knister@nasa.gov</a>></span> wrote:
<blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex" >Thanks Zach, I was about to echo similar sentiments and you saved me a ton of typing :)<br><br>Bob, I know this doesn't help you today since I'm pretty sure its not yet available, but if one scours the interwebs they can find mention of something called Mestor.<br><br>There's very very limited information here:<br><br>- <a href="https://indico.cern.ch/event/531810/contributions/2306222/attachments/1357265/2053960/Spectrum_Scale-HEPIX_V1a.pdf" rel="noreferrer" target="_blank" >https://indico.cern.ch/event/5<wbr>31810/contributions/2306222/at<wbr>tachments/1357265/2053960/Spec<wbr>trum_Scale-HEPIX_V1a.pdf</a><br>- <a href="https://www.yumpu.com/en/document/view/5544551/ibm-system-x-gpfs-storage-server-stfc" rel="noreferrer" target="_blank" >https://www.yumpu.com/en/docum<wbr>ent/view/5544551/ibm-system-x-<wbr>gpfs-storage-server-stfc</a> (slide 20)<br><br>Sounds like if it were available it would fit this use case very well.<br><br>I also had preliminary success with using sheepdog (<a href="https://sheepdog.github.io/sheepdog/" rel="noreferrer" target="_blank" >https://sheepdog.github.io/sh<wbr>eepdog/</a>) as a backing store for GPFS in a similar situation. It's perhaps at a very high conceptually level similar to Mestor. You erasure code your data across the nodes w/ the SAS disks and then present those block devices to your NSD servers. I proved it could work but never tried to to much with it because the requirements changed.<br><br>My money would be on your first option-- creating local RAIDs and then replicating to give you availability in the event a node goes offline.<br><br>-Aaron<br><br><br><span>On 11/30/16 10:59 PM, Zachary Giles wrote:</span>
<blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex" ><span>Just remember that replication protects against data availability, not<br>integrity. GPFS still requires the underlying block device to return<br>good data.<br><br>If you're using it on plain disks (SAS or SSD), and the drive returns<br>corrupt data, GPFS won't know any better and just deliver it to the<br>client. Further, if you do a partial read followed by a write, both<br>replicas could be destroyed. There's also no efficient way to force use<br>of a second replica if you realize the first is bad, short of taking the<br>first entirely offline. In that case while migrating data, there's no<br>good way to prevent read-rewrite of other corrupt data on your drive<br>that has the "good copy" while restriping off a faulty drive.<br><br>Ideally RAID would have a goal of only returning data that passed the<br>RAID algorithm, so shouldn't be corrupt, or made good by recreating from<br>parity. However, as we all know RAID controllers are definitely prone to<br>failures as well for many reasons, but at least a drive can go bad in<br>various ways (bad sectors, slow, just dead, poor SSD cell wear, etc)<br>without (hopefully) silent corruption..<br><br>Just something to think about while considering replication ..<br><br><br><br>On Wed, Nov 30, 2016 at 11:28 AM, Uwe Falke <<a href="mailto:UWEFALKE@de.ibm.com" target="_blank" >UWEFALKE@de.ibm.com</a></span>
<div><div><mailto:<a href="mailto:UWEFALKE@de.ibm.com" target="_blank" >UWEFALKE@de.ibm.com</a>>> wrote:<br><br> I have once set up a small system with just a few SSDs in two NSD<br> servers,<br> providin a scratch file system in a computing cluster.<br> No RAID, two replica.<br> works, as long the admins do not do silly things (like rebooting servers<br> in sequence without checking for disks being up in between).<br> Going for RAIDs without GPFS replication protects you against single<br> disk<br> failures, but you're lost if just one of your NSD servers goes off.<br><br> FPO makes sense only sense IMHO if your NSD servers are also processing<br> the data (and then you need to control that somehow).<br><br> Other ideas? what else can you do with GPFS and local disks than<br> what you<br> considered? I suppose nothing reasonable ...<br><br><br> Mit freundlichen Grüßen / Kind regards<br><br><br> Dr. Uwe Falke<br><br> IT Specialist<br> High Performance Computing Services / Integrated Technology Services /<br> Data Center Services<br> ------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>-------------------<br> IBM Deutschland<br> Rathausstr. 7<br> 09111 Chemnitz</div></div> Phone: <a href="tel:%2B49%20371%206978%202165" target="_blank" value="+4937169782165" >+49 371 6978 2165</a> <tel:%2B49%20371%206978%202165<wbr>><br> Mobile: <a href="tel:%2B49%20175%20575%202877" target="_blank" value="+491755752877" >+49 175 575 2877</a> <tel:%2B49%20175%20575%202877><br> E-Mail: <a href="mailto:uwefalke@de.ibm.com" target="_blank" >uwefalke@de.ibm.com</a> <mailto:<a href="mailto:uwefalke@de.ibm.com" target="_blank" >uwefalke@de.ibm.com</a>><br><span> ------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>-------------------<br> IBM Deutschland Business & Technology Services GmbH / Geschäftsführung:<br> Frank Hammer, Thorsten Moehring<br> Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht<br> Stuttgart,<br> HRB 17122<br><br><br><br><br> From: "Oesterlin, Robert" <<a href="mailto:Robert.Oesterlin@nuance.com" target="_blank" >Robert.Oesterlin@nuance.com</a></span><br> <mailto:<a href="mailto:Robert.Oesterlin@nuance.com" target="_blank" >Robert.Oesterlin@nuanc<wbr>e.com</a>>><br><span> To: gpfsug main discussion list<br> <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank" >gpfsug-discuss@spectrumscale.<wbr>org</a></span><br> <mailto:<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank" >gpfsug-discuss@spectru<wbr>mscale.org</a>>><br><span> Date: 11/30/2016 03:34 PM<br> Subject: [gpfsug-discuss] Strategies - servers with local SAS<br> disks<br> Sent by: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank" >gpfsug-discuss-bounces@spectru<wbr>mscale.org</a></span><br> <mailto:<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank" >gpfsug-discuss-bounces<wbr>@spectrumscale.org</a>><br><br><br><br><span> Looking for feedback/strategies in setting up several GPFS servers with<br> local SAS. They would all be part of the same file system. The<br> systems are<br> all similar in configuration - 70 4TB drives.<br><br> Options I?m considering:<br><br> - Create RAID arrays of the disks on each server (worried about the RAID<br> rebuild time when a drive fails with 4, 6, 8TB drives)<br> - No RAID with 2 replicas, single drive per NSD. When a drive fails,<br> recreate the NSD ? but then I need to fix up the data replication via<br> restripe<br> - FPO ? with multiple failure groups - letting the system manage<br> replica<br> placement and then have GPFS due the restripe on disk failure<br> automatically<br><br> Comments or other ideas welcome.<br><br> Bob Oesterlin<br> Sr Principal Storage Engineer, Nuance</span><br> <a href="tel:507-269-0413" target="_blank" value="+15072690413" >507-269-0413</a> <tel:<a href="tel:507-269-0413" target="_blank" value="+15072690413" >507-269-0413</a>><br><br> _____________________________<wbr>__________________<br> gpfsug-discuss mailing list<br> gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank" >spectrumscale.org</a> <<a href="http://spectrumscale.org" target="_blank" >http://spectrumscale.org</a>><br> <a href="http://gpfsug.org/mailman/list" target="_blank" >http://gpfsug.org/mailman/list</a><wbr>info/gpfsug-discuss<br><span> <<a href="http://gpfsug.org/mailman/lis" target="_blank" >http://gpfsug.org/mailman/lis</a></span><wbr>tinfo/gpfsug-discuss><br><br><br><br><br> ______________________________<wbr>_________________<br> gpfsug-discuss mailing list<br> gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank" >spectrumscale.org</a> <<a href="http://spectrumscale.org" target="_blank" >http://spectrumscale.org</a>><br> <a href="http://gpfsug.org/mailman/list" target="_blank" >http://gpfsug.org/mailman/list</a><wbr>info/gpfsug-discuss<br><span> <<a href="http://gpfsug.org/mailman/lis" target="_blank" >http://gpfsug.org/mailman/lis</a></span><wbr>tinfo/gpfsug-discuss><br><br><br><br><br>--<br>Zach Giles<br><a href="mailto:zgiles@gmail.com" target="_blank" >zgiles@gmail.com</a> <mailto:<a href="mailto:zgiles@gmail.com" target="_blank" >zgiles@gmail.com</a>><br><br><br><span>______________________________<wbr>_________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank" >spectrumscale.org</a><br><a href="http://gpfsug.org/mailman/list" target="_blank" >http://gpfsug.org/mailman/list</a></span><wbr>info/gpfsug-discuss<br> </blockquote><br><span><font color="#888888" >--<br>Aaron Knister<br>NASA Center for Climate Simulation (Code 606.2)<br>Goddard Space Flight Center<br><a href="tel:%28301%29%20286-2776" target="_blank" value="+13012862776" >(301) 286-2776</a></font></span>
<div><div><br>______________________________<wbr>_________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank" >spectrumscale.org</a><br><a href="http://gpfsug.org/mailman/list" target="_blank" >http://gpfsug.org/mailman/list</a><wbr>info/gpfsug-discuss</div></div></blockquote></div>
<div> </div>--
<div data-smartmail="gmail_signature" >Zach Giles<br><a href="mailto:zgiles@gmail.com" target="_blank" >zgiles@gmail.com</a></div></div>
<div><font size="2" face="Default Monospace,Courier New,Courier,monospace" >_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank" >http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></div></blockquote>
<div dir="ltr" > </div></div>Unless stated otherwise above:<BR>
IBM United Kingdom Limited - Registered in England and Wales with number 741598. <BR>
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU<BR>
<BR>