<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
span.hoenzb
{mso-style-name:hoenzb;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:Calibri;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:53893230;
mso-list-type:hybrid;
mso-list-template-ids:735846728 67698705 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-text:"%1\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Some interesting discussion here. Perhaps I should have been a bit clearer on what I’m looking at here:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">I have 12 servers with 70*4TB drives each – so the hardware is free. What’s the best strategy for using these as GPFS NSD servers, given that I don’t want to relay on any “bleeding edge”
technologies.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">1) My first choice would be GNR on commodity hardware – if IBM would give that to us. :-)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">2) Use standard RAID groups with no replication – downside is data availability of you lose an NSD and RAID group rebuild time with large disks<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">3) RAID groups with replication – but I lose a LOT of space (20% for RAID + 50% of what’s left for replication)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">4) No raid groups, single NSD per disk, single failure group per servers, replication. Downside here is I need to restripe every time a disk fails to get the filesystem back to a good state.
Might be OK using QoS to get the IO impact down<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">5) FPO doesn’t seem to by me anything, as these are straight NSD servers and no computation is going on these servers, and I still must live with the re-stripe.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Option (4) seems the best of the “no great options” I have in front of me.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-family:Calibri">Bob Oesterlin<br>
Sr Principal Storage Engineer, Nuance<br>
<br>
<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-family:Calibri;color:black">From: </span>
</b><span style="font-family:Calibri;color:black"><gpfsug-discuss-bounces@spectrumscale.org> on behalf of Zachary Giles <zgiles@gmail.com><br>
<b>Reply-To: </b>gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
<b>Date: </b>Wednesday, November 30, 2016 at 10:27 PM<br>
<b>To: </b>gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
<b>Subject: </b>[EXTERNAL] Re: [gpfsug-discuss] Strategies - servers with local SAS disks<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Aaron, Thanks for jumping onboard. It's nice to see others confirming this. Sometimes I feel alone on this topic.
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">It's should also be possible to use ZFS with ZVOLs presented as block devices for a backing store for NSDs. I'm not claiming it's stable, nor a good idea, nor performant.. but should be possible. :) There are various reports about it. Might
be at least worth looking in to compared to Linux "md raid" if one truly needs an all-software solution that already exists. Something to think about and test over.<o:p></o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Wed, Nov 30, 2016 at 11:15 PM, Aaron Knister <<a href="mailto:aaron.s.knister@nasa.gov" target="_blank">aaron.s.knister@nasa.gov</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal">Thanks Zach, I was about to echo similar sentiments and you saved me a ton of typing :)<br>
<br>
Bob, I know this doesn't help you today since I'm pretty sure its not yet available, but if one scours the interwebs they can find mention of something called Mestor.<br>
<br>
There's very very limited information here:<br>
<br>
- <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__indico.cern.ch_event_531810_contributions_2306222_attachments_1357265_2053960_Spectrum-5FScale-2DHEPIX-5FV1a.pdf&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=okpUZQo5UPYWvaBHBnDMMc_HKPdMppZvehm2-Wf24ms&e=" target="_blank">
https://indico.cern.ch/event/531810/contributions/2306222/attachments/1357265/2053960/Spectrum_Scale-HEPIX_V1a.pdf</a><br>
- <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__www.yumpu.com_en_document_view_5544551_ibm-2Dsystem-2Dx-2Dgpfs-2Dstorage-2Dserver-2Dstfc&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=UXlFxGKhufPyXf4aGSI3vVn_FOZ50Kibcgps8lgpzF4&e=" target="_blank">
https://www.yumpu.com/en/document/view/5544551/ibm-system-x-gpfs-storage-server-stfc</a> (slide 20)<br>
<br>
Sounds like if it were available it would fit this use case very well.<br>
<br>
I also had preliminary success with using sheepdog (<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__sheepdog.github.io_sheepdog_&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=2TOkgjDy0toVMOUXditrShHtONR_T_twsOMfWVN4lFU&e=" target="_blank">https://sheepdog.github.io/sheepdog/</a>)
as a backing store for GPFS in a similar situation. It's perhaps at a very high conceptually level similar to Mestor. You erasure code your data across the nodes w/ the SAS disks and then present those block devices to your NSD servers. I proved it could work
but never tried to to much with it because the requirements changed.<br>
<br>
My money would be on your first option-- creating local RAIDs and then replicating to give you availability in the event a node goes offline.<br>
<br>
-Aaron<br>
<br>
<br>
On 11/30/16 10:59 PM, Zachary Giles wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal">Just remember that replication protects against data availability, not<br>
integrity. GPFS still requires the underlying block device to return<br>
good data.<br>
<br>
If you're using it on plain disks (SAS or SSD), and the drive returns<br>
corrupt data, GPFS won't know any better and just deliver it to the<br>
client. Further, if you do a partial read followed by a write, both<br>
replicas could be destroyed. There's also no efficient way to force use<br>
of a second replica if you realize the first is bad, short of taking the<br>
first entirely offline. In that case while migrating data, there's no<br>
good way to prevent read-rewrite of other corrupt data on your drive<br>
that has the "good copy" while restriping off a faulty drive.<br>
<br>
Ideally RAID would have a goal of only returning data that passed the<br>
RAID algorithm, so shouldn't be corrupt, or made good by recreating from<br>
parity. However, as we all know RAID controllers are definitely prone to<br>
failures as well for many reasons, but at least a drive can go bad in<br>
various ways (bad sectors, slow, just dead, poor SSD cell wear, etc)<br>
without (hopefully) silent corruption..<br>
<br>
Just something to think about while considering replication ..<br>
<br>
<br>
<br>
On Wed, Nov 30, 2016 at 11:28 AM, Uwe Falke <<a href="mailto:UWEFALKE@de.ibm.com" target="_blank">UWEFALKE@de.ibm.com</a><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><mailto:<a href="mailto:UWEFALKE@de.ibm.com" target="_blank">UWEFALKE@de.ibm.com</a>>> wrote:<br>
<br>
I have once set up a small system with just a few SSDs in two NSD<br>
servers,<br>
providin a scratch file system in a computing cluster.<br>
No RAID, two replica.<br>
works, as long the admins do not do silly things (like rebooting servers<br>
in sequence without checking for disks being up in between).<br>
Going for RAIDs without GPFS replication protects you against single<br>
disk<br>
failures, but you're lost if just one of your NSD servers goes off.<br>
<br>
FPO makes sense only sense IMHO if your NSD servers are also processing<br>
the data (and then you need to control that somehow).<br>
<br>
Other ideas? what else can you do with GPFS and local disks than<br>
what you<br>
considered? I suppose nothing reasonable ...<br>
<br>
<br>
Mit freundlichen Grüßen / Kind regards<br>
<br>
<br>
Dr. Uwe Falke<br>
<br>
IT Specialist<br>
High Performance Computing Services / Integrated Technology Services /<br>
Data Center Services<br>
-------------------------------------------------------------------------------------------------------------------------------------------<br>
IBM Deutschland<br>
Rathausstr. 7<br>
09111 Chemnitz<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"> Phone: <a href="tel:%2B49%20371%206978%202165" target="_blank">
+49 371 6978 2165</a> <tel:%2B49%20371%206978%202165><br>
Mobile: <a href="tel:%2B49%20175%20575%202877" target="_blank">+49 175 575 2877</a> <tel:%2B49%20175%20575%202877><br>
E-Mail: <a href="mailto:uwefalke@de.ibm.com" target="_blank">uwefalke@de.ibm.com</a> <mailto:<a href="mailto:uwefalke@de.ibm.com" target="_blank">uwefalke@de.ibm.com</a>><br>
-------------------------------------------------------------------------------------------------------------------------------------------<br>
IBM Deutschland Business & Technology Services GmbH / Geschäftsführung:<br>
Frank Hammer, Thorsten Moehring<br>
Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht<br>
Stuttgart,<br>
HRB 17122<br>
<br>
<br>
<br>
<br>
From: "Oesterlin, Robert" <<a href="mailto:Robert.Oesterlin@nuance.com" target="_blank">Robert.Oesterlin@nuance.com</a><br>
<mailto:<a href="mailto:Robert.Oesterlin@nuance.com" target="_blank">Robert.Oesterlin@nuance.com</a>>><br>
To: gpfsug main discussion list<br>
<<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a><br>
<mailto:<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a>>><br>
Date: 11/30/2016 03:34 PM<br>
Subject: [gpfsug-discuss] Strategies - servers with local SAS<br>
disks<br>
Sent by: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">
gpfsug-discuss-bounces@spectrumscale.org</a><br>
<mailto:<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a>><br>
<br>
<br>
<br>
Looking for feedback/strategies in setting up several GPFS servers with<br>
local SAS. They would all be part of the same file system. The<br>
systems are<br>
all similar in configuration - 70 4TB drives.<br>
<br>
Options I?m considering:<br>
<br>
- Create RAID arrays of the disks on each server (worried about the RAID<br>
rebuild time when a drive fails with 4, 6, 8TB drives)<br>
- No RAID with 2 replicas, single drive per NSD. When a drive fails,<br>
recreate the NSD ? but then I need to fix up the data replication via<br>
restripe<br>
- FPO ? with multiple failure groups - letting the system manage<br>
replica<br>
placement and then have GPFS due the restripe on disk failure<br>
automatically<br>
<br>
Comments or other ideas welcome.<br>
<br>
Bob Oesterlin<br>
Sr Principal Storage Engineer, Nuance<br>
<a href="tel:507-269-0413" target="_blank">507-269-0413</a> <tel:<a href="tel:507-269-0413" target="_blank">507-269-0413</a>><br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=5YbzvsgvARAfEma_-9ivfs2B950q-viphuylZrZc6Xo&e=" target="_blank">
spectrumscale.org</a> <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=5YbzvsgvARAfEma_-9ivfs2B950q-viphuylZrZc6Xo&e=" target="_blank">http://spectrumscale.org</a>><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=Mk9i5Ca4nkt7TzIPrrNMWMTcAQQDHGQ2uxhEtTsUKG0&e=" target="_blank">
http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
<<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=Mk9i5Ca4nkt7TzIPrrNMWMTcAQQDHGQ2uxhEtTsUKG0&e=" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a>><br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=5YbzvsgvARAfEma_-9ivfs2B950q-viphuylZrZc6Xo&e=" target="_blank">
spectrumscale.org</a> <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=5YbzvsgvARAfEma_-9ivfs2B950q-viphuylZrZc6Xo&e=" target="_blank">http://spectrumscale.org</a>><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=Mk9i5Ca4nkt7TzIPrrNMWMTcAQQDHGQ2uxhEtTsUKG0&e=" target="_blank">
http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
<<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=Mk9i5Ca4nkt7TzIPrrNMWMTcAQQDHGQ2uxhEtTsUKG0&e=" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a>><br>
<br>
<br>
<br>
<br>
--<br>
Zach Giles<br>
<a href="mailto:zgiles@gmail.com" target="_blank">zgiles@gmail.com</a> <mailto:<a href="mailto:zgiles@gmail.com" target="_blank">zgiles@gmail.com</a>><br>
<br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=5YbzvsgvARAfEma_-9ivfs2B950q-viphuylZrZc6Xo&e=" target="_blank">
spectrumscale.org</a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=Mk9i5Ca4nkt7TzIPrrNMWMTcAQQDHGQ2uxhEtTsUKG0&e=" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span style="color:#888888"><br>
<span class="hoenzb">-- </span><br>
<span class="hoenzb">Aaron Knister</span><br>
<span class="hoenzb">NASA Center for Climate Simulation (Code 606.2)</span><br>
<span class="hoenzb">Goddard Space Flight Center</span><br>
<span class="hoenzb"><a href="tel:%28301%29%20286-2776" target="_blank">(301) 286-2776</a></span></span>
<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=5YbzvsgvARAfEma_-9ivfs2B950q-viphuylZrZc6Xo&e=" target="_blank">
spectrumscale.org</a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=oKF5vN9YFYVDqLO7RZfZ7WPfriUSAhGlwbWrwi5fMy0&s=Mk9i5Ca4nkt7TzIPrrNMWMTcAQQDHGQ2uxhEtTsUKG0&e=" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<p class="MsoNormal">Zach Giles<br>
<a href="mailto:zgiles@gmail.com" target="_blank">zgiles@gmail.com</a><o:p></o:p></p>
</div>
</div>
</div>
</body>
</html>