<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Hi Brian,
<div class=""><br class="">
</div>
<div class="">We’re probably an outlier on this (Bob’s case is probably much more typical) but we can get away with doing weekly migrations based on file atime. Some thoughts:</div>
<div class=""><br class="">
</div>
<div class="">1. absolutely use QOS! It’s one of the best things IBM has ever added to GPFS. </div>
<div class="">2. personally, I limit even my capacity pool to no more than 98% capacity. I just don’t think it’s a good idea to 100% fill anything.</div>
<div class="">3. if you do use anything like atime or mtime as your criteria, don’t forget to have a rule to move stuff back from the capacity pool if it’s now being used.</div>
<div class="">4. we also help manage a DDN device and there they do also implement a rule to move stuff if the “fast” pool exceeds a certain threshold … but they use file size as the weight. Not saying that’s right or wrong, it’s just another approach.</div>
<div class=""><br class="">
</div>
<div class="">HTHAL…</div>
<div class=""><br class="">
</div>
<div class="">Kevin</div>
<div class=""><br class="">
</div>
<div class="">
<div>
<blockquote type="cite" class="">
<div class="">On Dec 19, 2016, at 9:25 AM, Oesterlin, Robert <<a href="mailto:Robert.Oesterlin@nuance.com" class="">Robert.Oesterlin@nuance.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="WordSection1" style="page: WordSection1; font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
<span style="font-size: 11pt; font-family: Calibri;" class="">I tend to do migration based on “file heat”, moving the least active files to HDD and more active to SSD. Something simple like this:<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
<span style="font-size: 11pt; font-family: Calibri;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
<span style="font-size: 11pt; font-family: Calibri;" class="">rule grpdef GROUP POOL gpool IS ssd LIMIT(75) THEN disk<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
<span style="font-size: 11pt; font-family: Calibri;" class="">rule repack<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
<span style="font-size: 11pt; font-family: Calibri;" class=""> MIGRATE FROM POOL gpool TO POOL gpool<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
<span style="font-size: 11pt; font-family: Calibri;" class=""> WEIGHT(FILE_HEAT)<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
<span style="font-size: 11pt; font-family: Calibri;" class=""><o:p class=""> </o:p></span></div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
<span style="font-family: Calibri;" class="">Bob Oesterlin<br class="">
Sr Principal Storage Engineer, Nuance<br class="">
<br class="">
<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
<o:p class=""> </o:p></div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
<span style="font-size: 11pt; font-family: Calibri;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
<span style="font-size: 11pt; font-family: Calibri;" class=""><o:p class=""> </o:p></span></div>
<div style="border-style: solid none none; border-top-color: rgb(181, 196, 223); border-top-width: 1pt; padding: 3pt 0in 0in;" class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
<b class=""><span style="font-family: Calibri;" class="">From:<span class="Apple-converted-space"> </span></span></b><span style="font-family: Calibri;" class=""><<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" style="color: purple; text-decoration: underline;" class="">gpfsug-discuss-bounces@spectrumscale.org</a>>
on behalf of Brian Marshall <<a href="mailto:mimarsh2@vt.edu" style="color: purple; text-decoration: underline;" class="">mimarsh2@vt.edu</a>><br class="">
<b class="">Reply-To:<span class="Apple-converted-space"> </span></b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" style="color: purple; text-decoration: underline;" class="">gpfsug-discuss@spectrumscale.org</a>><br class="">
<b class="">Date:<span class="Apple-converted-space"> </span></b>Monday, December 19, 2016 at 9:15 AM<br class="">
<b class="">To:<span class="Apple-converted-space"> </span></b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" style="color: purple; text-decoration: underline;" class="">gpfsug-discuss@spectrumscale.org</a>><br class="">
<b class="">Subject:<span class="Apple-converted-space"> </span></b>[EXTERNAL] Re: [gpfsug-discuss] Tiers<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
We are in very similar situation. VT - ARC has a layer of SSD for metadata only, another layer of SSD for "hot" data, and a layer of 8TB HDDs for capacity. We just now in the process of getting it all into production.<o:p class=""></o:p></div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
On this topic:<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
What is everyone's favorite migration policy to move data from SSD to HDD (and vice versa)?<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
Do you nightly move large/old files to HDD or wait until the fast tier hit some capacity limit?<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
Do you use QOS to limit the migration from SSD to HDD i.e. try not to kill the file system with migration work?<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
Thanks,<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
Brian Marshall<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
<o:p class=""> </o:p></div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
On Thu, Dec 15, 2016 at 4:25 PM, Buterbaugh, Kevin L <<a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" target="_blank" style="color: purple; text-decoration: underline;" class="">Kevin.Buterbaugh@vanderbilt.edu</a>> wrote:<o:p class=""></o:p></div>
<blockquote style="border-style: none none none solid; border-left-color: rgb(204, 204, 204); border-left-width: 1pt; padding: 0in 0in 0in 6pt; margin-left: 4.8pt; margin-right: 0in;" class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
Hi Mark,<span class="Apple-converted-space"> </span><o:p class=""></o:p></div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
We just use an 8 Gb FC SAN. For the data pool we typically have a dual active-active controller storage array fronting two big RAID 6 LUNs and 1 RAID 1 (for /home). For the capacity pool, it might be the same exact model of controller, but the two controllers
are now fronting that whole 60-bay array.<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
But our users tend to have more modest performance needs than most…<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
<span style="color: rgb(136, 136, 136);" class=""><o:p class=""> </o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
<span style="color: rgb(136, 136, 136);" class="">Kevin<o:p class=""></o:p></span></div>
</div>
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
<o:p class=""> </o:p></div>
<div class="">
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
On Dec 15, 2016, at 3:19 PM,<span class="Apple-converted-space"> </span><a href="mailto:Mark.Bush@siriuscom.com" target="_blank" style="color: purple; text-decoration: underline;" class="">Mark.Bush@siriuscom.com</a><span class="Apple-converted-space"> </span>wrote:<o:p class=""></o:p></div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
<o:p class=""> </o:p></div>
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
<span style="font-size: 11pt; font-family: Calibri;" class="">Kevin, out of curiosity, what type of disk does your data pool use? SAS or just some SAN attached system?</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
<span style="font-size: 11pt; font-family: Calibri;" class=""> </span><o:p class=""></o:p></div>
</div>
<div style="border-style: solid none none; border-top-color: rgb(181, 196, 223); border-top-width: 1pt; padding: 3pt 0in 0in;" class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
<b class=""><span style="font-family: Calibri;" class="">From:<span class="m-4095158022254147304apple-converted-space"> </span></span></b><span style="font-family: Calibri;" class=""><<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">gpfsug-discuss-bounces@spectrumscale.org</span></a>>
on behalf of "Buterbaugh, Kevin L" <<a href="mailto:Kevin.Buterbaugh@Vanderbilt.Edu" target="_blank" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">Kevin.Buterbaugh@Vanderbilt.Edu</span></a>><br class="">
<b class="">Reply-To:<span class="m-4095158022254147304apple-converted-space"> </span></b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">gpfsug-discuss@spectrumscale.org</span></a>><br class="">
<b class="">Date:<span class="m-4095158022254147304apple-converted-space"> </span></b>Thursday, December 15, 2016 at 2:47 PM<br class="">
<b class="">To:<span class="m-4095158022254147304apple-converted-space"> </span></b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">gpfsug-discuss@spectrumscale.org</span></a>><br class="">
<b class="">Subject:<span class="m-4095158022254147304apple-converted-space"> </span></b>Re: [gpfsug-discuss] Tiers</span><o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
Hi Mark,<span class="m-4095158022254147304apple-converted-space"> </span><o:p class=""></o:p></div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
We’re a “traditional” university HPC center with a very untraditional policy on our scratch filesystem … we don’t purge it and we sell quota there. Ultimately, a lot of that disk space is taken up by stuff that, let’s just say, isn’t exactly in active use.<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
So what we’ve done, for example, is buy a 60-bay storage array and stuff it with 8 TB drives. It wouldn’t offer good enough performance for actively used files, but we use GPFS policies to migrate files to the “capacity” pool based on file atime. So we have
3 pools:<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
1. the system pool with metadata only (on SSDs)<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
2. the data pool, which is where actively used files are stored and which offers decent performance<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
3. the capacity pool, for data which hasn’t been accessed “recently”, and which is on slower storage<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
I would imagine others do similar things. HTHAL…<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
Kevin<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
<o:p class=""></o:p></div>
</div>
<div class="">
<div class="">
<div class="">
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
On Dec 15, 2016, at 2:32 PM,<span class="m-4095158022254147304apple-converted-space"> </span><a href="mailto:Mark.Bush@siriuscom.com" target="_blank" style="color: purple; text-decoration: underline;" class=""><span style="color: purple;" class="">Mark.Bush@siriuscom.com</span></a><span class="m-4095158022254147304apple-converted-space"> </span>wrote:<o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
<o:p class=""></o:p></div>
</div>
<div class="">
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
<span style="font-size: 11pt; font-family: Calibri;" class="">Just curious how many of you out there deploy SS with various tiers? It seems like a lot are doing the system pool with SSD’s but do you routinely have clusters that have more than system pool and
one more tier? <span class="m-4095158022254147304apple-converted-space"> </span></span><o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
<span style="font-size: 11pt; font-family: Calibri;" class=""> </span><o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
<span style="font-size: 11pt; font-family: Calibri;" class="">I know if you are doing Archive in connection that’s an obvious choice for another tier but I’m struggling with knowing why someone needs more than two tiers really.</span><o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
<span style="font-size: 11pt; font-family: Calibri;" class=""> </span><o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
<span style="font-size: 11pt; font-family: Calibri;" class="">I’ve read all the fine manuals as to how to do such a thing and some of the marketing as to maybe why. I’m still scratching my head on this though. In fact, my understanding is in the ESS there
isn’t any different pools (tiers) as it’s all NL-SAS or SSD (DF150, etc).<span class="m-4095158022254147304apple-converted-space"> </span></span><o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
<span style="font-size: 11pt; font-family: Calibri;" class=""> </span><o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
<span style="font-size: 11pt; font-family: Calibri;" class="">It does make sense to me know with TCT and I could create an ILM policy to get some of my data into the cloud.</span><o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
<span style="font-size: 11pt; font-family: Calibri;" class=""> </span><o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
<span style="font-size: 11pt; font-family: Calibri;" class="">But in the real world I would like to know what yall do in this regard.</span><o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
<span style="font-size: 11pt; font-family: Calibri;" class=""> </span><o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
<span style="font-size: 11pt; font-family: Calibri;" class=""> </span><o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
<span style="font-size: 11pt; font-family: Calibri;" class="">Thanks</span><o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
<span style="font-size: 11pt; font-family: Calibri;" class=""> </span><o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
<span style="font-size: 11pt; font-family: Calibri;" class="">Mark<span class="m-4095158022254147304apple-converted-space"> </span></span><o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
<span style="font-size: 11pt; font-family: Calibri;" class=""> </span><o:p class=""></o:p></div>
</div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; line-height: 10pt; background-color: white;" class="">
<span style="font-size: 8pt; font-family: Cambria;" class="">This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged,
confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by
parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately
and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you.</span><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
<b class=""><span style="font-size: 10pt; font-family: Calibri; color: rgb(204, 0, 0); background-color: white; background-position: initial initial; background-repeat: initial initial;" class=""><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.siriuscom.com_&d=DgMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=2V9MbsY4SgTRmE8kIq6GQAq0owTDl_XMhRmx6pH61Os&s=ApfdK36fjx8EPle4P0_HHozWlQgTEFSkvigVGHY-94U&e=" target="_blank" style="color: purple; text-decoration: underline;" class=""><span style="color: rgb(149, 79, 114);" class="">Sirius
Computer Solutions</span></a></span></b><o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
<span style="font-size: 10.5pt; font-family: Helvetica; background-color: white; background-position: initial initial; background-repeat: initial initial;" class="">_______________________________________________</span><span style="font-size: 10.5pt; font-family: Helvetica;" class=""><br class="">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial;" class="">gpfsug-discuss mailing list</span><br class="">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial;" class="">gpfsug-discuss at<span class="m-4095158022254147304apple-converted-space"> </span></span></span><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org_&d=DgMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=2V9MbsY4SgTRmE8kIq6GQAq0owTDl_XMhRmx6pH61Os&s=rtkufzJTlSLEaFc-2qGxOm33u9-5xYdReXP2nuL6KLM&e=" target="_blank" style="color: purple; text-decoration: underline;" class=""><span style="font-size: 10.5pt; font-family: Helvetica; color: rgb(149, 79, 114); background-color: white; background-position: initial initial; background-repeat: initial initial;" class="">spectrumscale.org</span></a><span style="font-size: 10.5pt; font-family: Helvetica;" class=""><br class="">
</span><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DgMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=2V9MbsY4SgTRmE8kIq6GQAq0owTDl_XMhRmx6pH61Os&s=bU09EKmlGWP0q6ENn-SjisUJ3b-BFCitjbUPWfLMhUc&e=" target="_blank" style="color: purple; text-decoration: underline;" class=""><span style="font-size: 10.5pt; font-family: Helvetica; color: rgb(149, 79, 114); background-color: white; background-position: initial initial; background-repeat: initial initial;" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></a><o:p class=""></o:p></div>
</div>
</div>
</blockquote>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman'; background-color: white;" class="">
<o:p class=""></o:p></div>
</div>
</div>
</div>
</div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
<span style="font-size: 10.5pt; font-family: Helvetica; background-color: white; background-position: initial initial; background-repeat: initial initial;" class="">_______________________________________________</span><span style="font-size: 10.5pt; font-family: Helvetica;" class=""><br class="">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial;" class="">gpfsug-discuss mailing list</span><br class="">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial;" class="">gpfsug-discuss at<span class="m-4095158022254147304apple-converted-space"> </span></span></span><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org_&d=DgMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=2V9MbsY4SgTRmE8kIq6GQAq0owTDl_XMhRmx6pH61Os&s=rtkufzJTlSLEaFc-2qGxOm33u9-5xYdReXP2nuL6KLM&e=" target="_blank" style="color: purple; text-decoration: underline;" class=""><span style="font-size: 10.5pt; font-family: Helvetica; color: purple; background-color: white; background-position: initial initial; background-repeat: initial initial;" class="">spectrumscale.org</span></a><span style="font-size: 10.5pt; font-family: Helvetica;" class=""><br class="">
</span><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DgMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=2V9MbsY4SgTRmE8kIq6GQAq0owTDl_XMhRmx6pH61Os&s=bU09EKmlGWP0q6ENn-SjisUJ3b-BFCitjbUPWfLMhUc&e=" target="_blank" style="color: purple; text-decoration: underline;" class=""><span style="font-size: 10.5pt; font-family: Helvetica; color: purple; background-color: white; background-position: initial initial; background-repeat: initial initial;" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></a><o:p class=""></o:p></div>
</div>
</blockquote>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
<o:p class=""> </o:p></div>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 12pt; font-family: 'Times New Roman';">
<br class="">
_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at<span class="Apple-converted-space"> </span><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DgMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=2V9MbsY4SgTRmE8kIq6GQAq0owTDl_XMhRmx6pH61Os&s=bZPT0Z1zNEbNctrugxAQ_G6wpNfuJFzNpfawDym_G9U&e=" target="_blank" style="color: purple; text-decoration: underline;" class="">spectrumscale.org</a><br class="">
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DgMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=2V9MbsY4SgTRmE8kIq6GQAq0owTDl_XMhRmx6pH61Os&s=bU09EKmlGWP0q6ENn-SjisUJ3b-BFCitjbUPWfLMhUc&e=" target="_blank" style="color: purple; text-decoration: underline;" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><o:p class=""></o:p></p>
</blockquote>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman';" class="">
<o:p class=""> </o:p></div>
</div>
</div>
<span style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<span style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;" class="">gpfsug-discuss
mailing list</span><br style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<span style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;" class="">gpfsug-discuss
at<span class="Apple-converted-space"> </span></span><a href="http://spectrumscale.org/" style="color: purple; text-decoration: underline; font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">spectrumscale.org</a><br style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" style="color: purple; text-decoration: underline; font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br style="font-family: Helvetica; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>