<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" >Andi</div>
<div dir="ltr" > </div>
<div dir="ltr" >Daisy chained AFM caches are a bad idea -- while it might work -- when things go wrong they go really badly wrong.</div>
<div dir="ltr" > </div>
<div dir="ltr" >Based on the scenario your describing</div>
<div dir="ltr" > </div>
<div dir="ltr" >What I think your going to want to do is AFM-DR between Sites A and B</div>
<div dir="ltr" >and then look at a policy based copy (Scripted Rsync or somthing similar) from Site B to site C</div>
<div dir="ltr" > </div>
<div dir="ltr" >I don't believe at present we support an AFM-DR relationship between a cluster and a Cache which is doing AFM to its home --</div>
<div dir="ltr" >You could put in a request with IBM development to see if they would support such an architecture - but i'm not sure its ever been tested.</div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<div dir="ltr" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div dir="ltr" style="margin-top: 20px;" ><div style="font-size: 12pt; font-weight: bold; font-family: sans-serif; color: #7C7C5F;" >Andrew Beattie</div>
<div style="font-size: 10pt; font-weight: bold; font-family: sans-serif;" >File and Object Storage Technical Specialist - A/NZ</div>
<div style="font-size: 10pt; font-weight: bold; font-family: sans-serif;" >IBM Systems - Storage</div>
<div style="font-size: 8pt; font-family: sans-serif; margin-top: 10px;" ><div><span style="font-weight: bold; color: #336699;" >Phone: </span>614-2133-7927</div>
<div><span style="font-weight: bold; color: #336699;" >E-mail: </span><a href="mailto:abeattie@au1.ibm.com" style="color: #555">abeattie@au1.ibm.com</a></div></div></div></div></div></div></div></div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Original message -----<br>From: "Andi Nør Christiansen" <arc@b4restore.com><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Cc:<br>Subject: [EXTERNAL] Re: [gpfsug-discuss] Spectrum Scale mirrored filesets across sites.<br>Date: Wed, Dec 18, 2019 10:31 PM<br> <br><!--Notes ACF
<meta http-equiv="Content-Type" content="text/html; charset=utf8" >--> <!--[if !mso]><style></style><![endif]--><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit" >
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div><p style="margin: 0px;" ><span style="mso-fareast-language:EN-US" >Hi Andrew,<o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="mso-fareast-language:EN-US" ><o:p> </o:p></span></p>
<p style="margin: 0px;" ><span style="mso-fareast-language:EN-US" lang="EN-US" >Alright, that partly confirms that there is no automatically sweep of data at cache site, right? I mean data will not be deleted automatically after a while in the cache fileset, where it is only metadata that stays? If data is kept until a manual deletion of data is requested on the cache site then this is the way to go for us..!<o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="mso-fareast-language:EN-US" lang="EN-US" ><o:p> </o:p></span></p>
<p style="margin: 0px;" ><span style="mso-fareast-language:EN-US" lang="EN-US" >Also, Site A has no connection to Site C so it needs to be connected as A to B and B to C.. <o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="mso-fareast-language:EN-US" lang="EN-US" >This means: Site A holds live data from Site A, Site B holds live data from Site B and Replicated data from Site A, Site C holds replicated data from A and B. Does that make sense?<o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="mso-fareast-language:EN-US" lang="EN-US" ><o:p> </o:p></span></p>
<p style="margin: 0px;" ><span style="mso-fareast-language:EN-US" lang="EN-US" >The connection between A and B is LAN, about 500meters apart.. basically same site but different data centers and strictly separated because of security.. Site C is in another Country. Hence why we cant use GPFS AFM and also why we need to utilize WAN/NFS tunneled for AFM.<o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="mso-fareast-language:EN-US" lang="EN-US" ><o:p> </o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:10.0pt;color:#54534A" lang="EN-US" >Best Regards</span><span lang="EN-US" ><o:p></o:p></span></p>
<p style="margin: 0px;" ><span lang="EN-US" > <o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:10.0pt;color:#54534A" lang="EN-US" >Andi Nør Christiansen</span><span lang="EN-US" ><o:p></o:p></span></p>
<p style="margin: 0px;" ><b><span style="font-size:10.0pt;color:#F8981D" lang="EN-US" >B4Restore A/S</span></b><br><br><span style="font-size:10.0pt" lang="EN-US" ><span style="color:#54534A" >Phone +45 87 81 37 39</span><br><span style="color:#54534A" >Mobile +45 23 89 59 75</span></span><span lang="EN-US" ><o:p></o:p></span></p>
<p style="margin: 0px;" ><span lang="EN-US" > </span><span style="mso-fareast-language:EN-US" lang="EN-US" ><o:p></o:p></span></p>
<p style="margin: 0px;" ><b>Fra:</b> gpfsug-discuss-bounces@spectrumscale.org <gpfsug-discuss-bounces@spectrumscale.org> <b>På vegne af </b>Andrew Beattie<br><b>Sendt:</b> 18. december 2019 13:04<br><b>Til:</b> gpfsug-discuss@spectrumscale.org<br><b>Cc:</b> gpfsug-discuss@spectrumscale.org<br><b>Emne:</b> Re: [gpfsug-discuss] Spectrum Scale mirrored filesets across sites.<o:p></o:p></p>
<p style="margin: 0px;" ><o:p> </o:p></p>
<div><div><p style="margin: 0px;" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" >Andi,<o:p></o:p></span></p></div>
<div><p style="margin: 0px;" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" > <o:p></o:p></span></p></div>
<div><p style="margin: 0px;" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" >This is basic functionality that is part of Spectrum Scale there is no additional licensing or HSM costs required for this.<o:p></o:p></span></p></div>
<div><p style="margin: 0px;" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" > <o:p></o:p></span></p></div>
<div><p style="margin: 0px;" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" >Set Site C as your AFM Home, and have Site A and Site B both as Caches of Site C<o:p></o:p></span></p></div>
<div><p style="margin: 0px;" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" > <o:p></o:p></span></p></div>
<div><p style="margin: 0px;" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" >you can then Write Data in to Site A - have it stream to Site C, and call it on demand or Prefetch from Site C to Site B as required<o:p></o:p></span></p></div>
<div><p style="margin: 0px;" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" >the Same is true of Site B, you can write Data into Site B, have it Stream to Site C, and call it on demand to site A<o:p></o:p></span></p></div>
<div><p style="margin: 0px;" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" > <o:p></o:p></span></p></div>
<div><p style="margin: 0px;" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" > <o:p></o:p></span></p></div>
<div><p style="margin: 0px;" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" >if you want the data to be Multi Writer then you will need to make sure you use Independent writer as the AFM type<o:p></o:p></span></p></div>
<div><p style="margin: 0px;" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" > <o:p></o:p></span></p></div>
<div><p style="margin: 0px;" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><a href="https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/Active%20File%20Management%20(AFM" target="_blank">https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/Active%20File%20Management%20(AFM</a>)<o:p></o:p></span></p></div>
<div><p style="margin: 0px;" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" > <o:p></o:p></span></p></div>
<div><p style="margin: 0px;" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" >Regards,<o:p></o:p></span></p></div>
<div><div><div><div><div><div style="margin-top:15.0pt" ><div><p style="margin: 0px;" ><b><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#7C7C5F" >Andrew Beattie<o:p></o:p></span></b></p></div>
<div><p style="margin: 0px;" ><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif" >File and Object Storage Technical Specialist - A/NZ<o:p></o:p></span></b></p></div>
<div><p style="margin: 0px;" ><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif" >IBM Systems - Storage<o:p></o:p></span></b></p></div>
<div style="margin-top:7.5pt" ><div><p style="margin: 0px;" ><b><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#336699" >Phone: </span></b><span style="font-size:8.0pt;font-family:"Arial",sans-serif" >614-2133-7927<o:p></o:p></span></p></div>
<div><p style="margin: 0px;" ><b><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#336699" >E-mail: </span></b><span style="font-size:8.0pt;font-family:"Arial",sans-serif" ><a href="mailto:abeattie@au1.ibm.com" target="_blank"><span style="color:#555555" >abeattie@au1.ibm.com</span></a><o:p></o:p></span></p></div></div></div></div></div></div></div></div>
<div><p style="margin: 0px;" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" > <o:p></o:p></span></p></div>
<div><p style="margin: 0px;" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" > <o:p></o:p></span></p></div>
<blockquote style="border:none;border-left:solid #AAAAAA 1.5pt;padding:0cm 0cm 0cm 4.0pt;margin-left:3.75pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt" ><p style="margin: 0px;" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" >----- Original message -----<br>From: "Andi Nør Christiansen" <<a href="mailto:arc@b4restore.com" target="_blank">arc@b4restore.com</a>><br>Sent by: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a><br>To: gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a>><br>Cc:<br>Subject: [EXTERNAL] [gpfsug-discuss] Spectrum Scale mirrored filesets across sites.<br>Date: Wed, Dec 18, 2019 8:00 PM<br> <o:p></o:p></span></p>
<div><p style="margin:0cm;margin-bottom:.0001pt" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" lang="EN-US" >Hi,</span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" lang="EN-US" > </span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" lang="EN-US" >We are currently building a 3 site spectrum scale solution where data is going to be generated at two different sites (Site A and Site B, Site C is for archiving/backup) and then archived on site three. </span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" lang="EN-US" >I have however not worked with AFM much so I was wondering if there is someone who knows how to configure AFM to have all data generated in a file-set automatically being copied to an offsite. </span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" lang="EN-US" >GPFS AFM is not an option because of latency between sites so NFS AFM is going to be tunneled between the site via WAN. </span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" lang="EN-US" > </span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" lang="EN-US" >As of now we have tried to set up AFM but it only transfers data from home to cache when a prefetch is manually started or a file is being opened, we need all files from home to go to cache as soon as it is generated or at least after a little while. </span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" lang="EN-US" >It does not need to be synchronous it just need to be automatic. </span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" lang="EN-US" > </span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" lang="EN-US" >I’m not sure if attachments will be available in this thread but I have attached the concept of our design. </span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" lang="EN-US" > </span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" lang="EN-US" >Basically the setup is :</span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" lang="EN-US" > </span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" lang="EN-US" >Site A:</span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" lang="EN-US" >Owns “fileset A1” which needs to be replicated to Site B “fileset A2” the from Site B to Site C “fileset A3”.</span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" lang="EN-US" > </span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" lang="EN-US" >Site B:</span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" lang="EN-US" >Owns “fileset B1” which needs to be replicated to Site C “fileset B2”.</span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" lang="EN-US" > </span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" lang="EN-US" >Site C: </span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" lang="EN-US" >Holds all data from Site A and B “fileset A3 & B2”.</span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" lang="EN-US" > </span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" lang="EN-US" >We do not need any sites to have failover functionality only a copy of the data from the two first sites. </span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" lang="EN-US" > </span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" lang="EN-US" >If anyone knows how to accomplish this I would be glad to know how! </span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" lang="EN-US" > </span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" lang="EN-US" >We have been looking into switching the home and cache site so that data is generated at the cache sites which will trigger GPFS to transfer the files to home as soon as possible but as I have little to no experience with AFM I don’t know what happens to the cache site over time, does the cache site empty itself after a while or does data stay there until manually deleted?</span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" lang="EN-US" > </span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" lang="EN-US" >Thanks in advance!</span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" lang="EN-US" > </span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><o:p></o:p></span></p>
<div><p style="margin:0cm;margin-bottom:.0001pt" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><a href="https://www.linkedin.com/company/b4restore-a-s/" target="_blank" title="B4Restore on LinkedIn"><span style="color:#54534A;text-decoration:none" lang="EN-US" >Best Regards</span></a><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt" ><br><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><img id="_x0000_i1025" src="cid:image001.jpg@01D5B5A5.D4744A80" style="width:1.5416in;height:.2291in" width="148" height="22" border="0" ></span><br><br><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#54534A" lang="EN-US" >Andi Nør Christiansen</span></b><br><span style="font-size:10.0pt;font-family:"Arial",sans-serif" lang="EN-US" ><i><span style="color:#54534A" >IT Solution Specialist</span></i><br><br><span style="color:#54534A" >Phone +45 87 81 37 39</span><br><span style="color:#54534A" >Mobile +45 23 89 59 75<br>E-mail</span> </span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><a href="mailto:" target="_blank"><span style="color:#3C3C3C;text-decoration:none" lang="EN-US" >arc@b4restore.com</span></a></span><br><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#54534A" lang="EN-US" >Web</span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" lang="EN-US" > </span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><a href="http://www.b4restore.com" target="_blank" title="B4Restore Website"><span style="color:#3C3C3C;text-decoration:none" lang="EN-US" >www.b4restore.com</span></a><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" lang="EN-US" > </span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><a href="https://www.linkedin.com/company/b4restore-a-s/" target="_blank" title=""B4Restore on LinkedIn" "><span style="color:#F8981D;text-decoration:none" ><img alt="B4Restore on LinkedIn" id="_x0000_i1026" src="cid:image002.png@01D5B5A5.D4744A80" style="width:.3125in;height:.3125in" width="30" height="30" border="0" ></span></a> <a href="https://twitter.com/b4restore" target="_blank" title=""B4Restore on Twitter" "><span style="color:#F8981D;text-decoration:none" ><img alt="B4Restore on Facebook" id="_x0000_i1027" src="cid:image003.png@01D5B5A5.D4744A80" style="width:.3125in;height:.3125in" width="30" height="30" border="0" ></span></a> <a href="https://www.facebook.com/B4Restore" target="_blank" title=""B4Restore on Facebook" "><span style="color:#F8981D;text-decoration:none" ><img alt="B4Restore on Facebook" id="_x0000_i1028" src="cid:image004.png@01D5B5A5.D4744A80" style="width:.3125in;height:.3125in" width="30" height="30" border="0" ></span></a> <a href="https://www.b4restore.com/en/about-us/newsletter" target="_blank" title=""Sign up for our newsletter" "><span style="color:#F8981D;text-decoration:none" ><img alt="Sign up for our newsletter" id="_x0000_i1029" src="cid:image005.png@01D5B5A5.D4744A80" style="width:.3125in;height:.3125in" width="30" height="30" border="0" ></span></a><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><a href="https://www.b4restore.com/report" target="_blank" title="https://www.b4restore.com/report"><span style="color:#F8981D;text-decoration:none" ><img alt="Download Report" id="_x0000_i1030" src="cid:image006.png@01D5B5A5.D4744A80" style="width:4.6875in;height:1.2604in" width="450" height="121" border="0" ></span></a><o:p></o:p></span></p></div>
<div><p style="margin:0cm;margin-bottom:.0001pt" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" > <o:p></o:p></span></p></div></div>
<div id="MIMEAttachInfoDiv" ><p style="margin: 0px;" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" > <o:p></o:p></span></p></div>
<div><p style="margin: 0px;" ><span style="font-size:10.0pt;font-family:"Courier New"" >_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> </span><span style="font-size:10.0pt;font-family:"Arial",sans-serif" ><o:p></o:p></span></p></div></blockquote>
<div><p style="margin: 0px;" ><span style="font-size:10.0pt;font-family:"Arial",sans-serif" > <o:p></o:p></span></p></div></div>
<p style="margin: 0px;" ><o:p> </o:p></p></div>
<div><font size="2" face="Default Monospace,Courier New,Courier,monospace" >_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> </font></div></blockquote>
<div dir="ltr" > </div></div>
<BR>