<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body>
<div dir="ltr">Thanks Sven! That makes sense to me and is what I thought was the case which is why I was confused when I saw the reply to the thread that said the >32 subblocks code had no performance impact.
<div dir="ltr"><br>
</div>
<div dir="ltr">A couple more question for you— in your presentation there’s a benchmark that shows the file create performance without the zero padding. Since you mention this is done for security reasons was that feature ever promoted to a GA Scale release?
I’m also wondering if you could explain the performance difference between the no zero padding code and the > 32 subblock code since given your the example of 32K files and 16MB block size I figure both cases ought to write the same amount to disk. </div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Thanks!</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">-Aaron</div>
</div>
<span id="draft-break"></span><br>
<br>
<br>
<span id="draft-break"></span><br>
<br>
<div>
<div class="null" dir="auto">On December 15, 2017 at 18:07:23 EST, Sven Oehme <oehmes@gmail.com> wrote:<br class="null">
</div>
<blockquote type="cite" style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;" class="null">
<div class="null" dir="auto">
<div class="null">
<div class="null">
<div dir="ltr" class="null">i thought i answered that already, but maybe i just thought about answering it and then forgot about it :-D
<div class="null"><br class="null">
</div>
<div class="null">so yes more than 32 subblocks per block significant increase the performance of filesystems with small files, for the sake of the argument let's say 32k in a large block filesystem again for sake of argument say 16MB. </div>
<div class="null"><br class="null">
</div>
<div class="null">you probably ask why ? </div>
<div class="null"><br class="null">
</div>
<div class="null">if you create a file and write 32k into it in a pre 5.0.0 Version 16 MB filesystem your client actually doesn't write 32k to the NSD Server, it writes 512k, because thats the subblock size and we need to write the full subblock (for security
reasons). so first you waste significant memory on the client to cache that zero padding, you waste network bandwidth and you waste NSD Server cache because you store it there too. this means you overrun the cache more quickly, means you start doing read/modify
writes earlier on all your nice large raid tracks... i guess you get the story by now. </div>
<div class="null"><br class="null">
</div>
<div class="null">in fact, if you have a good raid code that can drive really a lot of bandwidth out of individual drives like a GNR system you get more performance for small file writes as larger your blocksize is, because we can 'pack' more files into larger
i/os and therefore turn a small file create workload into a bandwidth workload, essentially exactly what we did and i demonstrated in the CORAL presentation . </div>
<div class="null"><br class="null">
</div>
<div class="null">hope that makes this crystal clear now . </div>
<div class="null"><br class="null">
</div>
<div class="null">sven</div>
<div class="null"><br class="null">
</div>
<div class="null"><br class="null">
</div>
</div>
<br class="null">
<div nop="gmail_quote" class="null">
<div dir="ltr" class="null">On Fri, Dec 15, 2017 at 10:47 PM Aaron Knister <<a href="mailto:aaron.s.knister@nasa.gov" class="null">aaron.s.knister@nasa.gov</a>> wrote:<br class="null">
</div>
<div class="null" ref="4332">
<div id="bx-quote-4332" class="null"><span class="null"></span></div>
</div>
<blockquote nop="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex" class="null">
Thanks, Alex. I'm all too familiar with the trade offs between large<br class="null">
blocks and small files and we do use pretty robust SSD storage for our<br class="null">
metadata. We support a wide range of workloads and we have some folks<br class="null">
with many small (<1M) files and other folks with many large (>256MB) files.<br class="null">
<br class="null">
My point in this thread is that IBM has said over and over again in<br class="null">
presentations that there is a significant performance gain with the >32<br class="null">
subblocks code on filesystems with large block sizes (although to your<br class="null">
point I'm not clear on exactly what large means since I didn't define<br class="null">
large in this context). Therefore given that the >32 subblock code gives<br class="null">
a significant performance gain one could reasonably assume that having a<br class="null">
filesystem with >32 subblocks is required to see this gain (rather than<br class="null">
just running the >32 subblocks code on an fs w/o > 32 subblocks).<br class="null">
<br class="null">
This lead me to ask about a migration tool because in my mind if there's<br class="null">
a performance gain from having >32 subblocks on the FS I'd like that<br class="null">
feature and having to manually copy 10's of PB to new hardware to get<br class="null">
this performance boost is unacceptable. However, IBM can't seem to make<br class="null">
up their mind about whether or not the >32 subblocks code *actually*<br class="null">
provides a performance increase. This seems like a pretty<br class="null">
straightforward question.<br class="null">
<br class="null">
-Aaron<br class="null">
<br class="null">
On 12/15/17 3:48 PM, Alex Chekholko wrote:<br class="null">
> Hey Aaron,<br class="null">
><br class="null">
> Can you define your sizes for "large blocks" and "small files"? If you<br class="null">
> dial one up and the other down, your performance will be worse. And in<br class="null">
> any case it's a pathological corner case so it shouldn't matter much for<br class="null">
> your workflow, unless you've designed your system with the wrong values.<br class="null">
><br class="null">
> For example, for bioinformatics workloads, I prefer to use 256KB<br class="null">
> filesystem block size, and I'd consider 4MB+ to be "large block size",<br class="null">
> which would make the filesystem obviously unsuitable for processing<br class="null">
> millions of 8KB files.<br class="null">
><br class="null">
> You can make a histogram of file sizes in your existing filesystems and<br class="null">
> then make your subblock size (1/32 of block size) on the smaller end of<br class="null">
> that. Also definitely use the "small file in inode" feature and put<br class="null">
> your metadata on SSD.<br class="null">
><br class="null">
> Regards,<br class="null">
> Alex<br class="null">
><br class="null">
> On Fri, Dec 15, 2017 at 11:49 AM, Aaron Knister<br class="null">
> <<a href="mailto:aaron.s.knister@nasa.gov" target="_blank" class="null">aaron.s.knister@nasa.gov</a> <mailto:<a href="mailto:aaron.s.knister@nasa.gov" target="_blank" class="null">aaron.s.knister@nasa.gov</a>>> wrote:<br class="null">
><br class="null">
> Thanks, Bill.<br class="null">
><br class="null">
> I still don't feel like I've got an clear answer from IBM and frankly<br class="null">
> the core issue of a lack of migration tool was totally dodged.<br class="null">
><br class="null">
> Again in Sven's presentation from SSUG @ SC17<br class="null">
> (<a href="http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf" rel="noreferrer" target="_blank" class="null">http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf</a> <<a href="http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf" rel="noreferrer" target="_blank" class="null">http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf</a>>)<br class="null">
> he mentions "It has a significant performance penalty for small files in<br class="null">
> large block size filesystems" and the demonstrates that with several<br class="null">
> mdtest runs (which show the effect with and without the >32<br class="null">
> subblocks code):<br class="null">
><br class="null">
><br class="null">
> 4.2.1 base code - SUMMARY: (of 3 iterations)<br class="null">
> File creation : Mean = 2237.644<br class="null">
><br class="null">
> zero-end-of-file-padding (4.2.2 + ifdef for zero padding): SUMMARY: (of<br class="null">
> 3 iterations)<br class="null">
> File creation : Mean = 12866.842<br class="null">
><br class="null">
> more sub blocks per block (4.2.2 + morethan32subblock code):<br class="null">
> File creation : Mean = 40316.721<br class="null">
><br class="null">
> Can someone (ideally Sven) give me a straight answer as to whether or<br class="null">
> not the > 32 subblock code actually makes a performance difference for<br class="null">
> small files in large block filesystems? And if not, help me understand<br class="null">
> why his slides and provided benchmark data have consistently indicated<br class="null">
> it does?<br class="null">
><br class="null">
> -Aaron<br class="null">
><br class="null">
> On 12/1/17 11:44 AM, Bill Hartner wrote:<br class="null">
> > ESS GL4 4u106 w/ 10 TB drives - same HW Sven reported some of the<br class="null">
> > results @ user group meeting.<br class="null">
> ><br class="null">
> > -Bill<br class="null">
> ><br class="null">
> > Bill Hartner<br class="null">
> > IBM Systems<br class="null">
> > Scalable I/O Development<br class="null">
> > Austin, Texas<br class="null">
> > <a href="mailto:bhartner@us.ibm.com" target="_blank" class="null">bhartner@us.ibm.com</a> <mailto:<a href="mailto:bhartner@us.ibm.com" target="_blank" class="null">bhartner@us.ibm.com</a>><br class="null">
> > home office <a href="tel:(512)%20784-0980" value="+15127840980" target="_blank" class="null">
512-784-0980</a> <tel:<a href="tel:(512)%20784-0980" value="+15127840980" target="_blank" class="null">512-784-0980</a>><br class="null">
> ><br class="null">
> ><br class="null">
> > Inactive hide details for Jan-Frode Myklebust ---12/01/2017 06:53:44<br class="null">
> > AM---Bill, could you say something about what the metadataJan-Frode<br class="null">
> > Myklebust ---12/01/2017 06:53:44 AM---Bill, could you say something<br class="null">
> > about what the metadata-storage here was? ESS/NL-SAS/3way replication?<br class="null">
> ><br class="null">
> > From: Jan-Frode Myklebust <<a href="mailto:janfrode@tanso.net" target="_blank" class="null">janfrode@tanso.net</a> <mailto:<a href="mailto:janfrode@tanso.net" target="_blank" class="null">janfrode@tanso.net</a>>><br class="null">
> > To: gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank" class="null">gpfsug-discuss@spectrumscale.org</a><br class="null">
> <mailto:<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank" class="null">gpfsug-discuss@spectrumscale.org</a>>><br class="null">
> > Date: 12/01/2017 06:53 AM<br class="null">
> > Subject: Re: [gpfsug-discuss] Online data migration tool<br class="null">
> > Sent by: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank" class="null">
gpfsug-discuss-bounces@spectrumscale.org</a><br class="null">
> <mailto:<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank" class="null">gpfsug-discuss-bounces@spectrumscale.org</a>><br class="null">
> ><br class="null">
> ><br class="null">
> ------------------------------------------------------------------------<br class="null">
> ><br class="null">
> ><br class="null">
> ><br class="null">
> > Bill, could you say something about what the metadata-storage here was?<br class="null">
> > ESS/NL-SAS/3way replication?<br class="null">
> ><br class="null">
> > I just asked about this in the internal slack channel #scale-help today..<br class="null">
> ><br class="null">
> ><br class="null">
> ><br class="null">
> > -jf<br class="null">
> ><br class="null">
> > fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner <_bhartner@us.ibm.com_<br class="null">
> > <mailto:<a href="mailto:bhartner@us.ibm.com" target="_blank" class="null">bhartner@us.ibm.com</a> <mailto:<a href="mailto:bhartner@us.ibm.com" target="_blank" class="null">bhartner@us.ibm.com</a>>>>:<br class="null">
> ><br class="null">
> > > "It has a significant performance penalty for small files in<br class="null">
> large<br class="null">
> > > block size filesystems"<br class="null">
> ><br class="null">
> > Aaron,<br class="null">
> ><br class="null">
> > Below are mdtest results for a test we ran for CORAL - file<br class="null">
> size was<br class="null">
> > 32k.<br class="null">
> ><br class="null">
> > We have not gone back and ran the test on a file system formatted<br class="null">
> > without > 32 subblocks. We'll do that at some point...<br class="null">
> ><br class="null">
> > -Bill<br class="null">
> ><br class="null">
> > -- started at 10/28/2017 17:51:38 --<br class="null">
> ><br class="null">
> > mdtest-1.9.3 was launched with 228 total task(s) on 12 node(s)<br class="null">
> > Command line used: /tmp/mdtest-binary-dir/mdtest -d<br class="null">
> > /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w 32768 -C -F -r<br class="null">
> -p 360<br class="null">
> > -u -y<br class="null">
> > Path: /ibm/fs2-16m-10<br class="null">
> > FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used Inodes: 0.0%<br class="null">
> ><br class="null">
> > 228 tasks, 67239936 files<br class="null">
> ><br class="null">
> > SUMMARY: (of 3 iterations)<br class="null">
> > Operation Max Min Mean Std Dev<br class="null">
> > --------- --- --- ---- -------<br class="null">
> > File creation : 51953.498 50558.517 51423.221 616.643<br class="null">
> > File stat : 0.000 0.000 0.000 0.000<br class="null">
> > File read : 0.000 0.000 0.000 0.000<br class="null">
> > File removal : 96746.376 92149.535 94658.774 1900.187<br class="null">
> > Tree creation : 1.588 0.070 0.599 0.700<br class="null">
> > Tree removal : 0.213 0.034 0.097 0.082<br class="null">
> ><br class="null">
> > -- finished at 10/28/2017 19:51:54 --<br class="null">
> ><br class="null">
> > Bill Hartner<br class="null">
> > IBM Systems<br class="null">
> > Scalable I/O Development<br class="null">
> > Austin, Texas_<br class="null">
> > __bhartner@us.ibm.com_ <mailto:<a href="mailto:bhartner@us.ibm.com" target="_blank" class="null">bhartner@us.ibm.com</a><br class="null">
> <mailto:<a href="mailto:bhartner@us.ibm.com" target="_blank" class="null">bhartner@us.ibm.com</a>>><br class="null">
> > home office <a href="tel:(512)%20784-0980" value="+15127840980" target="_blank" class="null">
512-784-0980</a> <tel:<a href="tel:(512)%20784-0980" value="+15127840980" target="_blank" class="null">512-784-0980</a>><br class="null">
> ><br class="null">
> > _<br class="null">
> > __gpfsug-discuss-bounces@spectrumscale.org_<br class="null">
> > <mailto:<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank" class="null">gpfsug-discuss-bounces@spectrumscale.org</a><br class="null">
> <mailto:<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank" class="null">gpfsug-discuss-bounces@spectrumscale.org</a>>> wrote on<br class="null">
> > 11/29/2017 04:41:48 PM:<br class="null">
> ><br class="null">
> > > From: Aaron Knister <_aaron.knister@gmail.com_<br class="null">
> > <mailto:<a href="mailto:aaron.knister@gmail.com" target="_blank" class="null">aaron.knister@gmail.com</a> <mailto:<a href="mailto:aaron.knister@gmail.com" target="_blank" class="null">aaron.knister@gmail.com</a>>>><br class="null">
> ><br class="null">
> ><br class="null">
> > > To: gpfsug main discussion list<br class="null">
> > <_gpfsug-discuss@spectrumscale.org_<br class="null">
> > <mailto:<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank" class="null">gpfsug-discuss@spectrumscale.org</a><br class="null">
> <mailto:<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank" class="null">gpfsug-discuss@spectrumscale.org</a>>>><br class="null">
> ><br class="null">
> > > Date: 11/29/2017 04:42 PM<br class="null">
> ><br class="null">
> ><br class="null">
> > > Subject: Re: [gpfsug-discuss] Online data migration tool<br class="null">
> > > Sent by: _gpfsug-discuss-bounces@spectrumscale.org_<br class="null">
> > <mailto:<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank" class="null">gpfsug-discuss-bounces@spectrumscale.org</a><br class="null">
> <mailto:<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank" class="null">gpfsug-discuss-bounces@spectrumscale.org</a>>><br class="null">
> ><br class="null">
> > ><br class="null">
> ><br class="null">
> > > Thanks, Nikhil. Most of that was consistent with my understnading,<br class="null">
> > > however I was under the impression that the >32 subblocks code is<br class="null">
> > > required to achieve the touted 50k file creates/second that Sven has<br class="null">
> > > talked about a bunch of times:<br class="null">
> > ><br class="null">
> > ><br class="null">
> > <br class="null">
> _<a href="http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_" rel="noreferrer" target="_blank" class="null">http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_</a><br class="null">
> <<a href="http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_" rel="noreferrer" target="_blank" class="null">http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_</a>><br class="null">
> > <br class="null">
> <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=" rel="noreferrer" target="_blank" class="null">https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=</a><br class="null">
> <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=" rel="noreferrer" target="_blank" class="null">https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=</a>>><br class="null">
> > ><br class="null">
> > <br class="null">
> _<a href="http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_" rel="noreferrer" target="_blank" class="null">http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_</a><br class="null">
> <<a href="http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_" rel="noreferrer" target="_blank" class="null">http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_</a>><br class="null">
> <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=" rel="noreferrer" target="_blank" class="null">https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=</a><br class="null">
> <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=" rel="noreferrer" target="_blank" class="null">https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=</a>>><br class="null">
> > > _Sven_Oehme_-_News_from_Research.pdf<br class="null">
> > > _<a href="http://files.gpfsug.org/presentations/2016/SC16/12_-_" rel="noreferrer" target="_blank" class="null">http://files.gpfsug.org/presentations/2016/SC16/12_-_</a><br class="null">
> <<a href="http://files.gpfsug.org/presentations/2016/SC16/12_-_" rel="noreferrer" target="_blank" class="null">http://files.gpfsug.org/presentations/2016/SC16/12_-_</a>><br class="null">
> > <br class="null">
> <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=" rel="noreferrer" target="_blank" class="null">https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=</a><br class="null">
> <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=" rel="noreferrer" target="_blank" class="null">https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=</a>>><br class="null">
> > > _Sven_Oehme_Dean_Hildebrand_-_News_from_IBM_Research.pdf<br class="null">
> ><br class="null">
> ><br class="null">
> > > from those presentations regarding 32 subblocks:<br class="null">
> > ><br class="null">
> > > "It has a significant performance penalty for small files in large<br class="null">
> > > block size filesystems"<br class="null">
> ><br class="null">
> > > although I'm not clear on the specific definition of "large". Many<br class="null">
> > > filesystems I encounter only have a 1M block size so it may not<br class="null">
> > > matter there, although that same presentation clearly shows the<br class="null">
> > > benefit of larger block sizes which is yet *another* thing for which<br class="null">
> > > a migration tool would be helpful.<br class="null">
> ><br class="null">
> > > -Aaron<br class="null">
> > ><br class="null">
> > > On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal<br class="null">
> > <_nikhilk@us.ibm.com_ <mailto:<a href="mailto:nikhilk@us.ibm.com" target="_blank" class="null">nikhilk@us.ibm.com</a><br class="null">
> <mailto:<a href="mailto:nikhilk@us.ibm.com" target="_blank" class="null">nikhilk@us.ibm.com</a>>>> wrote:<br class="null">
> ><br class="null">
> > > Hi,<br class="null">
> > ><br class="null">
> > > I would like to clarify migration path to 5.0.0 from 4.X.X<br class="null">
> clusters.<br class="null">
> > > For all Spectrum Scale clusters that are currently at 4.X.X,<br class="null">
> it is<br class="null">
> > > possible to migrate to 5.0.0 with no offline data migration<br class="null">
> and no<br class="null">
> > > need to move data. Once these clusters are at 5.0.0, they will<br class="null">
> > > benefit from the performance improvements, new features (such as<br class="null">
> > > file audit logging), and various enhancements that are<br class="null">
> included in<br class="null">
> > 5.0.0.<br class="null">
> > ><br class="null">
> > > That being said, there is one enhancement that will not be<br class="null">
> applied<br class="null">
> > > to these clusters, and that is the increased number of<br class="null">
> sub-blocks<br class="null">
> > > per block for small file allocation. This means that for file<br class="null">
> > > systems with a large block size and a lot of small files, the<br class="null">
> > > overall space utilization will be the same it currently is<br class="null">
> in 4.X.X.<br class="null">
> > > Since file systems created at 4.X.X and earlier used a block<br class="null">
> size<br class="null">
> > > that kept this allocation in mind, there should be very little<br class="null">
> > > impact on existing file systems.<br class="null">
> > ><br class="null">
> > > Outside of that one particular function, the remainder of the<br class="null">
> > > performance improvements, metadata improvements, updated<br class="null">
> > > compatibility, new functionality, and all of the other<br class="null">
> enhancements<br class="null">
> > > will be immediately available to you once you complete the<br class="null">
> upgrade<br class="null">
> > > to 5.0.0 -- with no need to reformat, move data, or take<br class="null">
> your data<br class="null">
> > offline.<br class="null">
> > ><br class="null">
> > > I hope that clarifies things a little and makes the upgrade path<br class="null">
> > > more accessible.<br class="null">
> > ><br class="null">
> > > Please let me know if there are any other questions or concerns.<br class="null">
> > ><br class="null">
> > > Thank you,<br class="null">
> > > Nikhil Khandelwal<br class="null">
> > > Spectrum Scale Development<br class="null">
> > > Client Adoption<br class="null">
> > ><br class="null">
> > > _______________________________________________<br class="null">
> > > gpfsug-discuss mailing list<br class="null">
> > > gpfsug-discuss at _spectrumscale.org_<br class="null">
> > <br class="null">
> <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=" rel="noreferrer" target="_blank" class="null">https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=</a><br class="null">
> <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=" rel="noreferrer" target="_blank" class="null">https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=</a>>><br class="null">
> > > _<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss_" rel="noreferrer" target="_blank" class="null">http://gpfsug.org/mailman/listinfo/gpfsug-discuss_</a><br class="null">
> <<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss_" rel="noreferrer" target="_blank" class="null">http://gpfsug.org/mailman/listinfo/gpfsug-discuss_</a>><br class="null">
> > <br class="null">
> <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=" rel="noreferrer" target="_blank" class="null">https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=</a><br class="null">
> <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=" rel="noreferrer" target="_blank" class="null">https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=</a>>><br class="null">
> ><br class="null">
> > > _______________________________________________<br class="null">
> > > gpfsug-discuss mailing list<br class="null">
> > > gpfsug-discuss at _spectrumscale.org_<br class="null">
> > <br class="null">
> <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=" rel="noreferrer" target="_blank" class="null">https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=</a><br class="null">
> <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=" rel="noreferrer" target="_blank" class="null">https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=</a>>><br class="null">
> ><br class="null">
> > > _<a href="https://urldefense.proofpoint.com/v2/url?_" rel="noreferrer" target="_blank" class="null">https://urldefense.proofpoint.com/v2/url?_</a><br class="null">
> <<a href="https://urldefense.proofpoint.com/v2/url?_" rel="noreferrer" target="_blank" class="null">https://urldefense.proofpoint.com/v2/url?_</a>><br class="null">
> > ><br class="null">
> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-<br class="null">
> > ><br class="null">
> > siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=DHoqgBeMFgcM0LpXEI0VCYvvb8ollct5aSYUDln2t68&s=iOxGm-853L_W0XkB3jGsGzCTVlSYUvANOTSewcR_Ue8&e=<br class="null">
> ><br class="null">
> > _______________________________________________<br class="null">
> > gpfsug-discuss mailing list<br class="null">
> > gpfsug-discuss at _spectrumscale.org_<br class="null">
> > <br class="null">
> <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=" rel="noreferrer" target="_blank" class="null">https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=</a><br class="null">
> <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=" rel="noreferrer" target="_blank" class="null">https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=</a>>>_<br class="null">
> > __<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss_" rel="noreferrer" target="_blank" class="null">http://gpfsug.org/mailman/listinfo/gpfsug-discuss_</a><br class="null">
> <<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss_" rel="noreferrer" target="_blank" class="null">http://gpfsug.org/mailman/listinfo/gpfsug-discuss_</a>><br class="null">
> > <br class="null">
> <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=" rel="noreferrer" target="_blank" class="null">https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=</a><br class="null">
> <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=" rel="noreferrer" target="_blank" class="null">https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=</a>>>_______________________________________________<br class="null">
> > gpfsug-discuss mailing list<br class="null">
> > gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank" class="null">
spectrumscale.org</a> <<a href="http://spectrumscale.org" rel="noreferrer" target="_blank" class="null">http://spectrumscale.org</a>><br class="null">
> > <br class="null">
> <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=" rel="noreferrer" target="_blank" class="null">https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=</a><br class="null">
> <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=" rel="noreferrer" target="_blank" class="null">https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=</a>><br class="null">
> ><br class="null">
> ><br class="null">
> ><br class="null">
> ><br class="null">
> > _______________________________________________<br class="null">
> > gpfsug-discuss mailing list<br class="null">
> > gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank" class="null">
spectrumscale.org</a> <<a href="http://spectrumscale.org" rel="noreferrer" target="_blank" class="null">http://spectrumscale.org</a>><br class="null">
> > <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank" class="null">
http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="null">
> <<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank" class="null">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a>><br class="null">
> ><br class="null">
><br class="null">
> --<br class="null">
> Aaron Knister<br class="null">
> NASA Center for Climate Simulation (Code 606.2)<br class="null">
> Goddard Space Flight Center<br class="null">
> <a href="tel:(301)%20286-2776" value="+13012862776" target="_blank" class="null">(301) 286-2776</a> <tel:%28301%29%20286-2776><br class="null">
> _______________________________________________<br class="null">
> gpfsug-discuss mailing list<br class="null">
> gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank" class="null">
spectrumscale.org</a> <<a href="http://spectrumscale.org" rel="noreferrer" target="_blank" class="null">http://spectrumscale.org</a>><br class="null">
> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank" class="null">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="null">
> <<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank" class="null">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a>><br class="null">
><br class="null">
><br class="null">
><br class="null">
><br class="null">
> _______________________________________________<br class="null">
> gpfsug-discuss mailing list<br class="null">
> gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank" class="null">
spectrumscale.org</a><br class="null">
> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank" class="null">
http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="null">
><br class="null">
<br class="null">
--<br class="null">
Aaron Knister<br class="null">
NASA Center for Climate Simulation (Code 606.2)<br class="null">
Goddard Space Flight Center<br class="null">
<a href="tel:(301)%20286-2776" value="+13012862776" target="_blank" class="null">(301) 286-2776</a><br class="null">
_______________________________________________<br class="null">
gpfsug-discuss mailing list<br class="null">
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank" class="null">
spectrumscale.org</a><br class="null">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank" class="null">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="null">
</blockquote>
<span id="bx-quote-end-4332" class="null"></span></div>
</div>
</div>
</div>
</blockquote>
</div>
</body>
</html>