<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.hoenzb
{mso-style-name:hoenzb;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:Calibri;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Yep, I should have added those requirements :-)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">1) Yes I care about the data. It’s not scratch but a permanent repository of older, less frequently accessed data.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2) Yes, it will be backed up<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">3) I expect it to grow over time<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">4) Data integrity requirement: high<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-family:Calibri">Bob Oesterlin<br>
Sr Principal Storage Engineer, Nuance<br>
<br>
<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-family:Calibri;color:black">From: </span>
</b><span style="font-family:Calibri;color:black"><gpfsug-discuss-bounces@spectrumscale.org> on behalf of Stephen Ulmer <ulmer@ulmer.org><br>
<b>Reply-To: </b>gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
<b>Date: </b>Thursday, December 1, 2016 at 7:13 AM<br>
<b>To: </b>gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
<b>Subject: </b>[EXTERNAL] Re: [gpfsug-discuss] Strategies - servers with local SAS disks<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">Just because I don’t think I’ve seen you state it: (How much) Do you care about the data?
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Is it scratch? Is it test data that exists elsewhere? Does it ever flow from this storage to any other storage? Will it be dubbed business critical two years after they swear to you that it’s not important at all? Is it just your movie
collection? Are you going to back it up? Is it going to grow? Is this temporary?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">That would inform us about the level of integrity required, which is one of the main differentiators for the options you’re considering.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Liberty,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">-- <br>
Stephen<br>
<br>
<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Dec 1, 2016, at 7:47 AM, Oesterlin, Robert <<a href="mailto:Robert.Oesterlin@nuance.com">Robert.Oesterlin@nuance.com</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;font-family:Calibri">Some interesting discussion here. Perhaps I should have been a bit clearer on what I’m looking at here:</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;font-family:Calibri"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;font-family:Calibri">I have 12 servers with 70*4TB drives each – so the hardware is free. What’s the best strategy for using these as GPFS NSD servers, given that I don’t want to relay
on any “bleeding edge” technologies.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;font-family:Calibri"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;font-family:Calibri">1) My first choice would be GNR on commodity hardware – if IBM would give that to us. :-)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;font-family:Calibri">2) Use standard RAID groups with no replication – downside is data availability of you lose an NSD and RAID group rebuild time with large disks</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;font-family:Calibri">3) RAID groups with replication – but I lose a LOT of space (20% for RAID + 50% of what’s left for replication)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;font-family:Calibri">4) No raid groups, single NSD per disk, single failure group per servers, replication. Downside here is I need to restripe every time a disk fails to get the filesystem
back to a good state. Might be OK using QoS to get the IO impact down</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;font-family:Calibri">5) FPO doesn’t seem to by me anything, as these are straight NSD servers and no computation is going on these servers, and I still must live with the re-stripe.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;font-family:Calibri"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;font-family:Calibri">Option (4) seems the best of the “no great options” I have in front of me.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;font-family:Calibri"> </span><o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri">Bob Oesterlin<br>
Sr Principal Storage Engineer, Nuance<br>
<br>
<br>
</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="background:white"> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;font-family:Calibri"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:11.0pt;font-family:Calibri"> </span><o:p></o:p></p>
</div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<div>
<p class="MsoNormal" style="background:white"><b><span style="font-family:Calibri">From:<span class="apple-converted-space"> </span></span></b><span style="font-family:Calibri"><<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><span style="color:purple">gpfsug-discuss-bounces@spectrumscale.org</span></a>>
on behalf of Zachary Giles <<a href="mailto:zgiles@gmail.com"><span style="color:purple">zgiles@gmail.com</span></a>><br>
<b>Reply-To:<span class="apple-converted-space"> </span></b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org"><span style="color:purple">gpfsug-discuss@spectrumscale.org</span></a>><br>
<b>Date:<span class="apple-converted-space"> </span></b>Wednesday, November 30, 2016 at 10:27 PM<br>
<b>To:<span class="apple-converted-space"> </span></b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org"><span style="color:purple">gpfsug-discuss@spectrumscale.org</span></a>><br>
<b>Subject:<span class="apple-converted-space"> </span></b>[EXTERNAL] Re: [gpfsug-discuss] Strategies - servers with local SAS disks</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="background:white"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="background:white">Aaron, Thanks for jumping onboard. It's nice to see others confirming this. Sometimes I feel alone on this topic. <o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="background:white"> <o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="background:white">It's should also be possible to use ZFS with ZVOLs presented as block devices for a backing store for NSDs. I'm not claiming it's stable, nor a good idea, nor performant.. but should be possible. :) There are various
reports about it. Might be at least worth looking in to compared to Linux "md raid" if one truly needs an all-software solution that already exists. Something to think about and test over.<o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="background:white"> <o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="background:white">On Wed, Nov 30, 2016 at 11:15 PM, Aaron Knister <<a href="mailto:aaron.s.knister@nasa.gov" target="_blank"><span style="color:purple">aaron.s.knister@nasa.gov</span></a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="background:white">Thanks Zach, I was about to echo similar sentiments and you saved me a ton of typing :)<br>
<br>
Bob, I know this doesn't help you today since I'm pretty sure its not yet available, but if one scours the interwebs they can find mention of something called Mestor.<br>
<br>
There's very very limited information here:<br>
<br>
-<span class="apple-converted-space"> </span><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__indico.cern.ch_event_531810_contributions_2306222_attachments_1357265_2053960_Spectrum-5FScale-2DHEPIX-5FV1a.pdf&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=okpUZQo5UPYWvaBHBnDMMc_HKPdMppZvehm2-Wf24ms&e=" target="_blank"><span style="color:purple">https://indico.cern.ch/event/531810/contributions/2306222/attachments/1357265/2053960/Spectrum_Scale-HEPIX_V1a.pdf</span></a><br>
-<span class="apple-converted-space"> </span><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__www.yumpu.com_en_document_view_5544551_ibm-2Dsystem-2Dx-2Dgpfs-2Dstorage-2Dserver-2Dstfc&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=UXlFxGKhufPyXf4aGSI3vVn_FOZ50Kibcgps8lgpzF4&e=" target="_blank"><span style="color:purple">https://www.yumpu.com/en/document/view/5544551/ibm-system-x-gpfs-storage-server-stfc</span></a><span class="apple-converted-space"> </span>(slide
20)<br>
<br>
Sounds like if it were available it would fit this use case very well.<br>
<br>
I also had preliminary success with using sheepdog (<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__sheepdog.github.io_sheepdog_&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=2TOkgjDy0toVMOUXditrShHtONR_T_twsOMfWVN4lFU&e=" target="_blank"><span style="color:purple">https://sheepdog.github.io/sheepdog/</span></a>)
as a backing store for GPFS in a similar situation. It's perhaps at a very high conceptually level similar to Mestor. You erasure code your data across the nodes w/ the SAS disks and then present those block devices to your NSD servers. I proved it could work
but never tried to to much with it because the requirements changed.<br>
<br>
My money would be on your first option-- creating local RAIDs and then replicating to give you availability in the event a node goes offline.<br>
<br>
-Aaron<br>
<br>
<br>
On 11/30/16 10:59 PM, Zachary Giles wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="background:white">Just remember that replication protects against data availability, not<br>
integrity. GPFS still requires the underlying block device to return<br>
good data.<br>
<br>
If you're using it on plain disks (SAS or SSD), and the drive returns<br>
corrupt data, GPFS won't know any better and just deliver it to the<br>
client. Further, if you do a partial read followed by a write, both<br>
replicas could be destroyed. There's also no efficient way to force use<br>
of a second replica if you realize the first is bad, short of taking the<br>
first entirely offline. In that case while migrating data, there's no<br>
good way to prevent read-rewrite of other corrupt data on your drive<br>
that has the "good copy" while restriping off a faulty drive.<br>
<br>
Ideally RAID would have a goal of only returning data that passed the<br>
RAID algorithm, so shouldn't be corrupt, or made good by recreating from<br>
parity. However, as we all know RAID controllers are definitely prone to<br>
failures as well for many reasons, but at least a drive can go bad in<br>
various ways (bad sectors, slow, just dead, poor SSD cell wear, etc)<br>
without (hopefully) silent corruption..<br>
<br>
Just something to think about while considering replication ..<br>
<br>
<br>
<br>
On Wed, Nov 30, 2016 at 11:28 AM, Uwe Falke <<a href="mailto:UWEFALKE@de.ibm.com" target="_blank"><span style="color:purple">UWEFALKE@de.ibm.com</span></a><o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="background:white"><mailto:<a href="mailto:UWEFALKE@de.ibm.com" target="_blank"><span style="color:purple">UWEFALKE@de.ibm.com</span></a>>> wrote:<br>
<br>
I have once set up a small system with just a few SSDs in two NSD<br>
servers,<br>
providin a scratch file system in a computing cluster.<br>
No RAID, two replica.<br>
works, as long the admins do not do silly things (like rebooting servers<br>
in sequence without checking for disks being up in between).<br>
Going for RAIDs without GPFS replication protects you against single<br>
disk<br>
failures, but you're lost if just one of your NSD servers goes off.<br>
<br>
FPO makes sense only sense IMHO if your NSD servers are also processing<br>
the data (and then you need to control that somehow).<br>
<br>
Other ideas? what else can you do with GPFS and local disks than<br>
what you<br>
considered? I suppose nothing reasonable ...<br>
<br>
<br>
Mit freundlichen Grüßen / Kind regards<br>
<br>
<br>
Dr. Uwe Falke<br>
<br>
IT Specialist<br>
High Performance Computing Services / Integrated Technology Services /<br>
Data Center Services<br>
-------------------------------------------------------------------------------------------------------------------------------------------<br>
IBM Deutschland<br>
Rathausstr. 7<br>
09111 Chemnitz<o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt;background:white"> Phone:<span class="apple-converted-space"> </span><a href="tel:%2B49%20371%206978%202165" target="_blank"><span style="color:purple">+49 371 6978 2165</span></a><span class="apple-converted-space"> </span><<a href="tel:%2B49%20371%206978%202165"><span style="color:purple">tel:%2B49%20371%206978%202165</span></a>><br>
Mobile:<span class="apple-converted-space"> </span><a href="tel:%2B49%20175%20575%202877" target="_blank"><span style="color:purple">+49 175 575 2877</span></a><span class="apple-converted-space"> </span><<a href="tel:%2B49%20175%20575%202877"><span style="color:purple">tel:%2B49%20175%20575%202877</span></a>><br>
E-Mail:<span class="apple-converted-space"> </span><a href="mailto:uwefalke@de.ibm.com" target="_blank"><span style="color:purple">uwefalke@de.ibm.com</span></a><span class="apple-converted-space"> </span><mailto:<a href="mailto:uwefalke@de.ibm.com" target="_blank"><span style="color:purple">uwefalke@de.ibm.com</span></a>><br>
-------------------------------------------------------------------------------------------------------------------------------------------<br>
IBM Deutschland Business & Technology Services GmbH / Geschäftsführung:<br>
Frank Hammer, Thorsten Moehring<br>
Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht<br>
Stuttgart,<br>
HRB 17122<br>
<br>
<br>
<br>
<br>
From: "Oesterlin, Robert" <<a href="mailto:Robert.Oesterlin@nuance.com" target="_blank"><span style="color:purple">Robert.Oesterlin@nuance.com</span></a><br>
<mailto:<a href="mailto:Robert.Oesterlin@nuance.com" target="_blank"><span style="color:purple">Robert.Oesterlin@nuance.com</span></a>>><br>
To: gpfsug main discussion list<br>
<<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank"><span style="color:purple">gpfsug-discuss@spectrumscale.org</span></a><br>
<mailto:<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank"><span style="color:purple">gpfsug-discuss@spectrumscale.org</span></a>>><br>
Date: 11/30/2016 03:34 PM<br>
Subject: [gpfsug-discuss] Strategies - servers with local SAS<br>
disks<br>
Sent by: <span class="apple-converted-space"> </span><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank"><span style="color:purple">gpfsug-discuss-bounces@spectrumscale.org</span></a><br>
<mailto:<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank"><span style="color:purple">gpfsug-discuss-bounces@spectrumscale.org</span></a>><br>
<br>
<br>
<br>
Looking for feedback/strategies in setting up several GPFS servers with<br>
local SAS. They would all be part of the same file system. The<br>
systems are<br>
all similar in configuration - 70 4TB drives.<br>
<br>
Options I?m considering:<br>
<br>
- Create RAID arrays of the disks on each server (worried about the RAID<br>
rebuild time when a drive fails with 4, 6, 8TB drives)<br>
- No RAID with 2 replicas, single drive per NSD. When a drive fails,<br>
recreate the NSD ? but then I need to fix up the data replication via<br>
restripe<br>
- FPO ? with multiple failure groups - letting the system manage<br>
replica<br>
placement and then have GPFS due the restripe on disk failure<br>
automatically<br>
<br>
Comments or other ideas welcome.<br>
<br>
Bob Oesterlin<br>
Sr Principal Storage Engineer, Nuance<br>
<span class="apple-converted-space"> </span><a href="tel:507-269-0413" target="_blank"><span style="color:purple">507-269-0413</span></a><span class="apple-converted-space"> </span><tel:<a href="tel:507-269-0413" target="_blank"><span style="color:purple">507-269-0413</span></a>><br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at<span class="apple-converted-space"> </span><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=5YbzvsgvARAfEma_-9ivfs2B950q-viphuylZrZc6Xo&e=" target="_blank"><span style="color:purple">spectrumscale.org</span></a><span class="apple-converted-space"> </span><<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=5YbzvsgvARAfEma_-9ivfs2B950q-viphuylZrZc6Xo&e=" target="_blank"><span style="color:purple">http://spectrumscale.org</span></a>><br>
<span class="apple-converted-space"> </span><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=Mk9i5Ca4nkt7TzIPrrNMWMTcAQQDHGQ2uxhEtTsUKG0&e=" target="_blank"><span style="color:purple">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></a><br>
<<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=Mk9i5Ca4nkt7TzIPrrNMWMTcAQQDHGQ2uxhEtTsUKG0&e=" target="_blank"><span style="color:purple">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></a>><br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at<span class="apple-converted-space"> </span><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=5YbzvsgvARAfEma_-9ivfs2B950q-viphuylZrZc6Xo&e=" target="_blank"><span style="color:purple">spectrumscale.org</span></a><span class="apple-converted-space"> </span><<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=5YbzvsgvARAfEma_-9ivfs2B950q-viphuylZrZc6Xo&e=" target="_blank"><span style="color:purple">http://spectrumscale.org</span></a>><br>
<span class="apple-converted-space"> </span><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=Mk9i5Ca4nkt7TzIPrrNMWMTcAQQDHGQ2uxhEtTsUKG0&e=" target="_blank"><span style="color:purple">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></a><br>
<<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=Mk9i5Ca4nkt7TzIPrrNMWMTcAQQDHGQ2uxhEtTsUKG0&e=" target="_blank"><span style="color:purple">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></a>><br>
<br>
<br>
<br>
<br>
--<br>
Zach Giles<br>
<a href="mailto:zgiles@gmail.com" target="_blank"><span style="color:purple">zgiles@gmail.com</span></a><span class="apple-converted-space"> </span><mailto:<a href="mailto:zgiles@gmail.com" target="_blank"><span style="color:purple">zgiles@gmail.com</span></a>><br>
<br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at<span class="apple-converted-space"> </span><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=5YbzvsgvARAfEma_-9ivfs2B950q-viphuylZrZc6Xo&e=" target="_blank"><span style="color:purple">spectrumscale.org</span></a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=Mk9i5Ca4nkt7TzIPrrNMWMTcAQQDHGQ2uxhEtTsUKG0&e=" target="_blank"><span style="color:purple">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></a><o:p></o:p></p>
</blockquote>
<div>
<p class="MsoNormal" style="background:white"><span style="color:#888888"><br>
<span class="hoenzb">--</span><span class="apple-converted-space"> </span><br>
<span class="hoenzb">Aaron Knister</span><br>
<span class="hoenzb">NASA Center for Climate Simulation (Code 606.2)</span><br>
<span class="hoenzb">Goddard Space Flight Center</span><br>
<span class="hoenzb"><a href="tel:%28301%29%20286-2776" target="_blank"><span style="color:purple">(301) 286-2776</span></a></span></span><o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="background:white"><br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at<span class="apple-converted-space"> </span><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=5YbzvsgvARAfEma_-9ivfs2B950q-viphuylZrZc6Xo&e=" target="_blank"><span style="color:purple">spectrumscale.org</span></a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=Mk9i5Ca4nkt7TzIPrrNMWMTcAQQDHGQ2uxhEtTsUKG0&e=" target="_blank"><span style="color:purple">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></a><o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
</div>
<div>
<p class="MsoNormal" style="background:white"><br>
<br clear="all">
<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="background:white"> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="background:white">--<span class="apple-converted-space"> </span><o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="background:white">Zach Giles<br>
<a href="mailto:zgiles@gmail.com" target="_blank"><span style="color:purple">zgiles@gmail.com</span></a><o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Helvetica;background:white">_______________________________________________</span><span style="font-size:9.0pt;font-family:Helvetica"><br>
<span style="background:white">gpfsug-discuss mailing list</span><br>
<span style="background:white">gpfsug-discuss at<span class="apple-converted-space"> </span></span></span><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org_&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=REMqVSMAx6oXrE83z1k_NJv7fbllu4dFDK_zMRUcJVg&s=M3DNuff2lj4tWIFZdH5nhSTvLNRKgrV44jrrMf4lZac&e="><span style="font-size:9.0pt;font-family:Helvetica;color:purple;background:white">spectrumscale.org</span></a><span style="font-size:9.0pt;font-family:Helvetica"><br>
</span><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=REMqVSMAx6oXrE83z1k_NJv7fbllu4dFDK_zMRUcJVg&s=Mj93FfL8zSOajEvg1N48TZyUALmqsDqtfSJL8nB_PSU&e="><span style="font-size:9.0pt;font-family:Helvetica;color:purple;background:white">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></a><o:p></o:p></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>