<html><body bgcolor="#FFFFFF"><p>Hi Bob,<br><br>If you mean #4 with 2x data replication...then I would be very wary as the chance of data loss would be very high given local disk failure rates. So I think its really #4 with 3x replication vs #3 with 2x replication (and raid5/6 in node) (with maybe 3x for metadata). The space overhead is somewhat similar, but the rebuild times should be much faster for #3 given that a failed disk will not place any load on the storage network (as well there will be less data placed on network).<br><br>Dean <br><br><br><img width="16" height="16" src="cid:1__=07BB0AEFDFCD4E358f9e8a93df938690918c07B@" border="0" alt="Inactive hide details for "Oesterlin, Robert" ---12/01/2016 04:48:16 AM---Some interesting discussion here. Perhaps I should ha"><font color="#424282">"Oesterlin, Robert" ---12/01/2016 04:48:16 AM---Some interesting discussion here. Perhaps I should have been a bit clearer on what I’m looking at he</font><br><br><font size="2" color="#5F5F5F">From: </font><font size="2">"Oesterlin, Robert" <Robert.Oesterlin@nuance.com></font><br><font size="2" color="#5F5F5F">To: </font><font size="2">gpfsug main discussion list <gpfsug-discuss@spectrumscale.org></font><br><font size="2" color="#5F5F5F">Date: </font><font size="2">12/01/2016 04:48 AM</font><br><font size="2" color="#5F5F5F">Subject: </font><font size="2">Re: [gpfsug-discuss] Strategies - servers with local SAS disks</font><br><font size="2" color="#5F5F5F">Sent by: </font><font size="2">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br><br><br><font face="Calibri">Some interesting discussion here. Perhaps I should have been a bit clearer on what I’m looking at here:</font><br><font face="Calibri"> </font><br><font face="Calibri">I have 12 servers with 70*4TB drives each – so the hardware is free. What’s the best strategy for using these as GPFS NSD servers, given that I don’t want to relay on any “bleeding edge” technologies.</font><br><font face="Calibri"> </font><br><font face="Calibri">1) My first choice would be GNR on commodity hardware – if IBM would give that to us. :-) </font><br><font face="Calibri">2) Use standard RAID groups with no replication – downside is data availability of you lose an NSD and RAID group rebuild time with large disks</font><br><font face="Calibri">3) RAID groups with replication – but I lose a LOT of space (20% for RAID + 50% of what’s left for replication)</font><br><font face="Calibri">4) No raid groups, single NSD per disk, single failure group per servers, replication. Downside here is I need to restripe every time a disk fails to get the filesystem back to a good state. Might be OK using QoS to get the IO impact down</font><br><font face="Calibri">5) FPO doesn’t seem to by me anything, as these are straight NSD servers and no computation is going on these servers, and I still must live with the re-stripe.</font><br><font face="Calibri"> </font><br><font face="Calibri">Option (4) seems the best of the “no great options” I have in front of me.</font><br><font face="Calibri"> </font><br><font size="4" face="Calibri">Bob Oesterlin<br>Sr Principal Storage Engineer, Nuance<br></font><br><font size="4" face="Times New Roman"> </font><br><font face="Calibri"> </font><br><font face="Calibri"> </font><br><b><font size="4" face="Calibri">From: </font></b><font size="4" face="Calibri"><gpfsug-discuss-bounces@spectrumscale.org> on behalf of Zachary Giles <zgiles@gmail.com></font><b><font size="4" face="Calibri"><br>Reply-To: </font></b><font size="4" face="Calibri">gpfsug main discussion list <gpfsug-discuss@spectrumscale.org></font><b><font size="4" face="Calibri"><br>Date: </font></b><font size="4" face="Calibri">Wednesday, November 30, 2016 at 10:27 PM</font><b><font size="4" face="Calibri"><br>To: </font></b><font size="4" face="Calibri">gpfsug main discussion list <gpfsug-discuss@spectrumscale.org></font><b><font size="4" face="Calibri"><br>Subject: </font></b><font size="4" face="Calibri">[EXTERNAL] Re: [gpfsug-discuss] Strategies - servers with local SAS disks</font><br><font size="4" face="Times New Roman"> </font><br><font size="4" face="Times New Roman">Aaron, Thanks for jumping onboard. It's nice to see others confirming this. Sometimes I feel alone on this topic. </font><br><font size="4" face="Times New Roman"> </font><br><font size="4" face="Times New Roman">It's should also be possible to use ZFS with ZVOLs presented as block devices for a backing store for NSDs. I'm not claiming it's stable, nor a good idea, nor performant.. but should be possible. :) There are various reports about it. Might be at least worth looking in to compared to Linux "md raid" if one truly needs an all-software solution that already exists. Something to think about and test over.</font><br><font size="4" face="Times New Roman"> </font><br><font size="4" face="Times New Roman">On Wed, Nov 30, 2016 at 11:15 PM, Aaron Knister <</font><a href="mailto:aaron.s.knister@nasa.gov" target="_blank"><u><font size="4" color="#0000FF" face="Times New Roman">aaron.s.knister@nasa.gov</font></u></a><font size="4" face="Times New Roman">> wrote:</font><br><font size="4" face="Times New Roman">Thanks Zach, I was about to echo similar sentiments and you saved me a ton of typing :)<br><br>Bob, I know this doesn't help you today since I'm pretty sure its not yet available, but if one scours the interwebs they can find mention of something called Mestor.<br><br>There's very very limited information here:<br><br>- </font><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__indico.cern.ch_event_531810_contributions_2306222_attachments_1357265_2053960_Spectrum-5FScale-2DHEPIX-5FV1a.pdf&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=okpUZQo5UPYWvaBHBnDMMc_HKPdMppZvehm2-Wf24ms&e=" target="_blank"><u><font size="4" color="#0000FF" face="Times New Roman">https://indico.cern.ch/event/531810/contributions/2306222/attachments/1357265/2053960/Spectrum_Scale-HEPIX_V1a.pdf</font></u></a><font size="4" face="Times New Roman"><br>- </font><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__www.yumpu.com_en_document_view_5544551_ibm-2Dsystem-2Dx-2Dgpfs-2Dstorage-2Dserver-2Dstfc&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=UXlFxGKhufPyXf4aGSI3vVn_FOZ50Kibcgps8lgpzF4&e=" target="_blank"><u><font size="4" color="#0000FF" face="Times New Roman">https://www.yumpu.com/en/document/view/5544551/ibm-system-x-gpfs-storage-server-stfc</font></u></a><font size="4" face="Times New Roman"> (slide 20)<br><br>Sounds like if it were available it would fit this use case very well.<br><br>I also had preliminary success with using sheepdog (</font><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__sheepdog.github.io_sheepdog_&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=2TOkgjDy0toVMOUXditrShHtONR_T_twsOMfWVN4lFU&e=" target="_blank"><u><font size="4" color="#0000FF" face="Times New Roman">https://sheepdog.github.io/sheepdog/</font></u></a><font size="4" face="Times New Roman">) as a backing store for GPFS in a similar situation. It's perhaps at a very high conceptually level similar to Mestor. You erasure code your data across the nodes w/ the SAS disks and then present those block devices to your NSD servers. I proved it could work but never tried to to much with it because the requirements changed.<br><br>My money would be on your first option-- creating local RAIDs and then replicating to give you availability in the event a node goes offline.<br><br>-Aaron<br><br><br>On 11/30/16 10:59 PM, Zachary Giles wrote:</font><ul><font size="4" face="Times New Roman">Just remember that replication protects against data availability, not<br>integrity. GPFS still requires the underlying block device to return<br>good data.<br><br>If you're using it on plain disks (SAS or SSD), and the drive returns<br>corrupt data, GPFS won't know any better and just deliver it to the<br>client. Further, if you do a partial read followed by a write, both<br>replicas could be destroyed. There's also no efficient way to force use<br>of a second replica if you realize the first is bad, short of taking the<br>first entirely offline. In that case while migrating data, there's no<br>good way to prevent read-rewrite of other corrupt data on your drive<br>that has the "good copy" while restriping off a faulty drive.<br><br>Ideally RAID would have a goal of only returning data that passed the<br>RAID algorithm, so shouldn't be corrupt, or made good by recreating from<br>parity. However, as we all know RAID controllers are definitely prone to<br>failures as well for many reasons, but at least a drive can go bad in<br>various ways (bad sectors, slow, just dead, poor SSD cell wear, etc)<br>without (hopefully) silent corruption..<br><br>Just something to think about while considering replication ..<br><br><br><br>On Wed, Nov 30, 2016 at 11:28 AM, Uwe Falke <</font><a href="mailto:UWEFALKE@de.ibm.com" target="_blank"><u><font size="4" color="#0000FF" face="Times New Roman">UWEFALKE@de.ibm.com</font></u></a><br><font size="4" face="Times New Roman"><mailto:</font><a href="mailto:UWEFALKE@de.ibm.com" target="_blank"><u><font size="4" color="#0000FF" face="Times New Roman">UWEFALKE@de.ibm.com</font></u></a><font size="4" face="Times New Roman">>> wrote:<br><br> I have once set up a small system with just a few SSDs in two NSD<br> servers,<br> providin a scratch file system in a computing cluster.<br> No RAID, two replica.<br> works, as long the admins do not do silly things (like rebooting servers<br> in sequence without checking for disks being up in between).<br> Going for RAIDs without GPFS replication protects you against single<br> disk<br> failures, but you're lost if just one of your NSD servers goes off.<br><br> FPO makes sense only sense IMHO if your NSD servers are also processing<br> the data (and then you need to control that somehow).<br><br> Other ideas? what else can you do with GPFS and local disks than<br> what you<br> considered? I suppose nothing reasonable ...<br><br><br> Mit freundlichen Grüßen / Kind regards<br><br><br> Dr. Uwe Falke<br><br> IT Specialist<br> High Performance Computing Services / Integrated Technology Services /<br> Data Center Services<br> -------------------------------------------------------------------------------------------------------------------------------------------<br> IBM Deutschland<br> Rathausstr. 7<br> 09111 Chemnitz</font><br><font size="4" face="Times New Roman"> Phone: </font><a href="tel:%2B49%20371%206978%202165" target="_blank"><u><font size="4" color="#0000FF" face="Times New Roman">+49 371 6978 2165</font></u></a><font size="4" face="Times New Roman"> <tel:%2B49%20371%206978%202165><br> Mobile: </font><a href="tel:%2B49%20175%20575%202877" target="_blank"><u><font size="4" color="#0000FF" face="Times New Roman">+49 175 575 2877</font></u></a><font size="4" face="Times New Roman"> <tel:%2B49%20175%20575%202877><br> E-Mail: </font><a href="mailto:uwefalke@de.ibm.com" target="_blank"><u><font size="4" color="#0000FF" face="Times New Roman">uwefalke@de.ibm.com</font></u></a><font size="4" face="Times New Roman"> <mailto:</font><a href="mailto:uwefalke@de.ibm.com" target="_blank"><u><font size="4" color="#0000FF" face="Times New Roman">uwefalke@de.ibm.com</font></u></a><font size="4" face="Times New Roman">><br> -------------------------------------------------------------------------------------------------------------------------------------------<br> IBM Deutschland Business & Technology Services GmbH / Geschäftsführung:<br> Frank Hammer, Thorsten Moehring<br> Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht<br> Stuttgart,<br> HRB 17122<br><br><br><br><br> From: "Oesterlin, Robert" <</font><a href="mailto:Robert.Oesterlin@nuance.com" target="_blank"><u><font size="4" color="#0000FF" face="Times New Roman">Robert.Oesterlin@nuance.com</font></u></a><font size="4" face="Times New Roman"><br> <mailto:</font><a href="mailto:Robert.Oesterlin@nuance.com" target="_blank"><u><font size="4" color="#0000FF" face="Times New Roman">Robert.Oesterlin@nuance.com</font></u></a><font size="4" face="Times New Roman">>><br> To: gpfsug main discussion list<br> <</font><a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank"><u><font size="4" color="#0000FF" face="Times New Roman">gpfsug-discuss@spectrumscale.org</font></u></a><font size="4" face="Times New Roman"><br> <mailto:</font><a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank"><u><font size="4" color="#0000FF" face="Times New Roman">gpfsug-discuss@spectrumscale.org</font></u></a><font size="4" face="Times New Roman">>><br> Date: 11/30/2016 03:34 PM<br> Subject: [gpfsug-discuss] Strategies - servers with local SAS<br> disks<br> Sent by: </font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank"><u><font size="4" color="#0000FF" face="Times New Roman">gpfsug-discuss-bounces@spectrumscale.org</font></u></a><font size="4" face="Times New Roman"><br> <mailto:</font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank"><u><font size="4" color="#0000FF" face="Times New Roman">gpfsug-discuss-bounces@spectrumscale.org</font></u></a><font size="4" face="Times New Roman">><br><br><br><br> Looking for feedback/strategies in setting up several GPFS servers with<br> local SAS. They would all be part of the same file system. The<br> systems are<br> all similar in configuration - 70 4TB drives.<br><br> Options I?m considering:<br><br> - Create RAID arrays of the disks on each server (worried about the RAID<br> rebuild time when a drive fails with 4, 6, 8TB drives)<br> - No RAID with 2 replicas, single drive per NSD. When a drive fails,<br> recreate the NSD ? but then I need to fix up the data replication via<br> restripe<br> - FPO ? with multiple failure groups - letting the system manage<br> replica<br> placement and then have GPFS due the restripe on disk failure<br> automatically<br><br> Comments or other ideas welcome.<br><br> Bob Oesterlin<br> Sr Principal Storage Engineer, Nuance<br> </font><a href="tel:507-269-0413" target="_blank"><u><font size="4" color="#0000FF" face="Times New Roman">507-269-0413</font></u></a><font size="4" face="Times New Roman"> <tel:</font><a href="tel:507-269-0413" target="_blank"><u><font size="4" color="#0000FF" face="Times New Roman">507-269-0413</font></u></a><font size="4" face="Times New Roman">><br><br> _______________________________________________<br> gpfsug-discuss mailing list<br> gpfsug-discuss at </font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=5YbzvsgvARAfEma_-9ivfs2B950q-viphuylZrZc6Xo&e=" target="_blank"><u><font size="4" color="#0000FF" face="Times New Roman">spectrumscale.org</font></u></a><font size="4" face="Times New Roman"> <</font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=5YbzvsgvARAfEma_-9ivfs2B950q-viphuylZrZc6Xo&e=" target="_blank"><u><font size="4" color="#0000FF" face="Times New Roman">http://spectrumscale.org</font></u></a><font size="4" face="Times New Roman">><br> </font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=Mk9i5Ca4nkt7TzIPrrNMWMTcAQQDHGQ2uxhEtTsUKG0&e=" target="_blank"><u><font size="4" color="#0000FF" face="Times New Roman">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></u></a><font size="4" face="Times New Roman"><br> <</font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=Mk9i5Ca4nkt7TzIPrrNMWMTcAQQDHGQ2uxhEtTsUKG0&e=" target="_blank"><u><font size="4" color="#0000FF" face="Times New Roman">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></u></a><font size="4" face="Times New Roman">><br><br><br><br><br> _______________________________________________<br> gpfsug-discuss mailing list<br> gpfsug-discuss at </font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=5YbzvsgvARAfEma_-9ivfs2B950q-viphuylZrZc6Xo&e=" target="_blank"><u><font size="4" color="#0000FF" face="Times New Roman">spectrumscale.org</font></u></a><font size="4" face="Times New Roman"> <</font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=5YbzvsgvARAfEma_-9ivfs2B950q-viphuylZrZc6Xo&e=" target="_blank"><u><font size="4" color="#0000FF" face="Times New Roman">http://spectrumscale.org</font></u></a><font size="4" face="Times New Roman">><br> </font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=Mk9i5Ca4nkt7TzIPrrNMWMTcAQQDHGQ2uxhEtTsUKG0&e=" target="_blank"><u><font size="4" color="#0000FF" face="Times New Roman">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></u></a><font size="4" face="Times New Roman"><br> <</font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=Mk9i5Ca4nkt7TzIPrrNMWMTcAQQDHGQ2uxhEtTsUKG0&e=" target="_blank"><u><font size="4" color="#0000FF" face="Times New Roman">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></u></a><font size="4" face="Times New Roman">><br><br><br><br><br>--<br>Zach Giles</font><u><font size="4" color="#0000FF" face="Times New Roman"><br></font></u><a href="mailto:zgiles@gmail.com" target="_blank"><u><font size="4" color="#0000FF" face="Times New Roman">zgiles@gmail.com</font></u></a><font size="4" face="Times New Roman"> <mailto:</font><a href="mailto:zgiles@gmail.com" target="_blank"><u><font size="4" color="#0000FF" face="Times New Roman">zgiles@gmail.com</font></u></a><font size="4" face="Times New Roman">><br><br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=5YbzvsgvARAfEma_-9ivfs2B950q-viphuylZrZc6Xo&e=" target="_blank"><u><font size="4" color="#0000FF" face="Times New Roman">spectrumscale.org</font></u></a><u><font size="4" color="#0000FF" face="Times New Roman"><br></font></u><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=Mk9i5Ca4nkt7TzIPrrNMWMTcAQQDHGQ2uxhEtTsUKG0&e=" target="_blank"><u><font size="4" color="#0000FF" face="Times New Roman">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></u></a></ul><font size="4" color="#888888" face="Times New Roman"><br>-- <br>Aaron Knister<br>NASA Center for Climate Simulation (Code 606.2)<br>Goddard Space Flight Center</font><u><font size="4" color="#0000FF" face="Times New Roman"><br></font></u><a href="tel:%28301%29%20286-2776" target="_blank"><u><font size="4" color="#0000FF" face="Times New Roman">(301) 286-2776</font></u></a><font size="4" face="Times New Roman"> </font><br><font size="4" face="Times New Roman"><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=5YbzvsgvARAfEma_-9ivfs2B950q-viphuylZrZc6Xo&e=" target="_blank"><u><font size="4" color="#0000FF" face="Times New Roman">spectrumscale.org</font></u></a><u><font size="4" color="#0000FF" face="Times New Roman"><br></font></u><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=Mk9i5Ca4nkt7TzIPrrNMWMTcAQQDHGQ2uxhEtTsUKG0&e=" target="_blank"><u><font size="4" color="#0000FF" face="Times New Roman">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></u></a><br><font size="4" face="Times New Roman"><br></font><br><font size="4" face="Times New Roman"> </font><br><font size="4" face="Times New Roman">-- </font><br><font size="4" face="Times New Roman">Zach Giles</font><u><font size="4" color="#0000FF" face="Times New Roman"><br></font></u><a href="mailto:zgiles@gmail.com" target="_blank"><u><font size="4" color="#0000FF" face="Times New Roman">zgiles@gmail.com</font></u></a><tt>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></tt><tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></tt><tt><br></tt><br><br><BR>
</body></html>