<div dir="ltr">Hey Aaron,<div><br></div><div>Can you define your sizes for "large blocks" and "small files"? If you dial one up and the other down, your performance will be worse. And in any case it's a pathological corner case so it shouldn't matter much for your workflow, unless you've designed your system with the wrong values.</div><div><br></div><div>For example, for bioinformatics workloads, I prefer to use 256KB filesystem block size, and I'd consider 4MB+ to be "large block size", which would make the filesystem obviously unsuitable for processing millions of 8KB files.</div><div><br></div><div>You can make a histogram of file sizes in your existing filesystems and then make your subblock size (1/32 of block size) on the smaller end of that. Also definitely use the "small file in inode" feature and put your metadata on SSD.</div><div><br></div><div>Regards,</div><div>Alex</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Dec 15, 2017 at 11:49 AM, Aaron Knister <span dir="ltr"><<a href="mailto:aaron.s.knister@nasa.gov" target="_blank">aaron.s.knister@nasa.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Thanks, Bill.<br>
<br>
I still don't feel like I've got an clear answer from IBM and frankly<br>
the core issue of a lack of migration tool was totally dodged.<br>
<br>
Again in Sven's presentation from SSUG @ SC17<br>
(<a href="http://files.gpfsug.org/presentations/2017/SC17/SC17-UG-CORAL_V3.pdf" rel="noreferrer" target="_blank">http://files.gpfsug.org/<wbr>presentations/2017/SC17/SC17-<wbr>UG-CORAL_V3.pdf</a>)<br>
he mentions "It has a significant performance penalty for small files in<br>
large block size filesystems" and the demonstrates that with several<br>
mdtest runs (which show the effect with and without the >32 subblocks code):<br>
<br>
<br>
4.2.1 base code - SUMMARY: (of 3 iterations)<br>
File creation : Mean = 2237.644<br>
<br>
zero-end-of-file-padding (4.2.2 + ifdef for zero padding): SUMMARY: (of<br>
3 iterations)<br>
File creation : Mean = 12866.842<br>
<br>
more sub blocks per block (4.2.2 + morethan32subblock code):<br>
File creation : Mean = 40316.721<br>
<br>
Can someone (ideally Sven) give me a straight answer as to whether or<br>
not the > 32 subblock code actually makes a performance difference for<br>
small files in large block filesystems? And if not, help me understand<br>
why his slides and provided benchmark data have consistently indicated<br>
it does?<br>
<br>
-Aaron<br>
<span class=""><br>
On 12/1/17 11:44 AM, Bill Hartner wrote:<br>
> ESS GL4 4u106 w/ 10 TB drives - same HW Sven reported some of the<br>
> results @ user group meeting.<br>
><br>
> -Bill<br>
><br>
> Bill Hartner<br>
> IBM Systems<br>
> Scalable I/O Development<br>
> Austin, Texas<br>
> <a href="mailto:bhartner@us.ibm.com">bhartner@us.ibm.com</a><br>
> home office <a href="tel:512-784-0980" value="+15127840980">512-784-0980</a><br>
><br>
><br>
</span><span class="">> Inactive hide details for Jan-Frode Myklebust ---12/01/2017 06:53:44<br>
</span>> AM---Bill, could you say something about what the metadataJan-Frode<br>
<span class="">> Myklebust ---12/01/2017 06:53:44 AM---Bill, could you say something<br>
> about what the metadata-storage here was? ESS/NL-SAS/3way replication?<br>
><br>
> From: Jan-Frode Myklebust <<a href="mailto:janfrode@tanso.net">janfrode@tanso.net</a>><br>
> To: gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.<wbr>org</a>><br>
> Date: 12/01/2017 06:53 AM<br>
> Subject: Re: [gpfsug-discuss] Online data migration tool<br>
> Sent by: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">gpfsug-discuss-bounces@<wbr>spectrumscale.org</a><br>
><br>
</span>> ------------------------------<wbr>------------------------------<wbr>------------<br>
<span class="">><br>
><br>
><br>
> Bill, could you say something about what the metadata-storage here was?<br>
> ESS/NL-SAS/3way replication?<br>
><br>
> I just asked about this in the internal slack channel #scale-help today..<br>
><br>
><br>
><br>
> -jf<br>
><br>
</span>> fre. 1. des. 2017 kl. 13:44 skrev Bill Hartner <_bhartner@us.ibm.com_<br>
> <mailto:<a href="mailto:bhartner@us.ibm.com">bhartner@us.ibm.com</a>>>:<br>
<div><div class="h5">><br>
> > "It has a significant performance penalty for small files in large<br>
> > block size filesystems"<br>
><br>
> Aaron,<br>
><br>
> Below are mdtest results for a test we ran for CORAL - file size was<br>
> 32k.<br>
><br>
> We have not gone back and ran the test on a file system formatted<br>
> without > 32 subblocks. We'll do that at some point...<br>
><br>
> -Bill<br>
><br>
> -- started at 10/28/2017 17:51:38 --<br>
><br>
> mdtest-1.9.3 was launched with 228 total task(s) on 12 node(s)<br>
> Command line used: /tmp/mdtest-binary-dir/mdtest -d<br>
> /ibm/fs2-16m-10/mdtest-60000 -i 3 -n 294912 -w 32768 -C -F -r -p 360<br>
> -u -y<br>
> Path: /ibm/fs2-16m-10<br>
> FS: 128.1 TiB Used FS: 0.3% Inodes: 476.8 Mi Used Inodes: 0.0%<br>
><br>
> 228 tasks, 67239936 files<br>
><br>
> SUMMARY: (of 3 iterations)<br>
> Operation Max Min Mean Std Dev<br>
> --------- --- --- ---- -------<br>
> File creation : 51953.498 50558.517 51423.221 616.643<br>
> File stat : 0.000 0.000 0.000 0.000<br>
> File read : 0.000 0.000 0.000 0.000<br>
> File removal : 96746.376 92149.535 94658.774 1900.187<br>
> Tree creation : 1.588 0.070 0.599 0.700<br>
> Tree removal : 0.213 0.034 0.097 0.082<br>
><br>
> -- finished at 10/28/2017 19:51:54 --<br>
><br>
> Bill Hartner<br>
> IBM Systems<br>
> Scalable I/O Development<br>
</div></div>> Austin, Texas_<br>
> __bhartner@us.ibm.com_ <mailto:<a href="mailto:bhartner@us.ibm.com">bhartner@us.ibm.com</a>><br>
> home office <a href="tel:512-784-0980" value="+15127840980">512-784-0980</a><br>
><br>
> _<br>
> __gpfsug-discuss-bounces@<wbr>spectrumscale.org_<br>
> <mailto:<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">gpfsug-discuss-<wbr>bounces@spectrumscale.org</a>> <wbr>wrote on<br>
<span class="">> 11/29/2017 04:41:48 PM:<br>
><br>
</span>> > From: Aaron Knister <_aaron.knister@gmail.com_<br>
> <mailto:<a href="mailto:aaron.knister@gmail.com">aaron.knister@gmail.<wbr>com</a>>><br>
<span class="">><br>
><br>
> > To: gpfsug main discussion list<br>
</span>> <_gpfsug-discuss@<wbr>spectrumscale.org_<br>
> <mailto:<a href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@<wbr>spectrumscale.org</a>>><br>
<span class="">><br>
> > Date: 11/29/2017 04:42 PM<br>
><br>
><br>
> > Subject: Re: [gpfsug-discuss] Online data migration tool<br>
</span>> > Sent by: _gpfsug-discuss-bounces@<wbr>spectrumscale.org_<br>
> <mailto:<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">gpfsug-discuss-<wbr>bounces@spectrumscale.org</a>><br>
<span class="">><br>
> ><br>
><br>
> > Thanks, Nikhil. Most of that was consistent with my understnading,<br>
> > however I was under the impression that the >32 subblocks code is<br>
> > required to achieve the touted 50k file creates/second that Sven has<br>
> > talked about a bunch of times:<br>
> ><br>
> ><br>
</span>> _<a href="http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf_" rel="noreferrer" target="_blank">http://files.gpfsug.org/<wbr>presentations/2017/Manchester/<wbr>08_Research_Topics.pdf_</a><br>
> <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=UGLr4Z6sa2yWvKL99g7SuQdgwxnoZwhVmDuIbYsLqYY&e=" rel="noreferrer" target="_blank">https://urldefense.<wbr>proofpoint.com/v2/url?u=http-<wbr>3A__files.gpfsug.org_<wbr>presentations_2017_Manchester_<wbr>08-5FResearch-5FTopics.pdf&d=<wbr>DwMFaQ&c=jf_iaSHvJObTbx-<wbr>siA1ZOg&r=<wbr>Ew59QH6nxuyx6oTs7a8AYX7kKG3gaW<wbr>UGDGo5ZZr3wQ4&m=<wbr>KLv9eH4GG8WlXC5ENj_<wbr>jXnzCpm60QSNAADfp6s94oa4&s=<wbr>UGLr4Z6sa2yWvKL99g7SuQdgwxnoZw<wbr>hVmDuIbYsLqYY&e=</a>><br>
> ><br>
> _<a href="http://files.gpfsug.org/presentations/2017/Ehningen/31_-_SSUG17DE_-_" rel="noreferrer" target="_blank">http://files.gpfsug.org/<wbr>presentations/2017/Ehningen/<wbr>31_-_SSUG17DE_-_</a> <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Ehningen_31-5F-2D-5FSSUG17DE-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Il2rMx4AtGwjVRzX89kobZ0W25vW8TGm0KJevLd7KQ8&e=" rel="noreferrer" target="_blank">https://urldefense.<wbr>proofpoint.com/v2/url?u=http-<wbr>3A__files.gpfsug.org_<wbr>presentations_2017_Ehningen_<wbr>31-5F-2D-5FSSUG17DE-5F-2D&d=<wbr>DwMFaQ&c=jf_iaSHvJObTbx-<wbr>siA1ZOg&r=<wbr>Ew59QH6nxuyx6oTs7a8AYX7kKG3gaW<wbr>UGDGo5ZZr3wQ4&m=<wbr>KLv9eH4GG8WlXC5ENj_<wbr>jXnzCpm60QSNAADfp6s94oa4&s=<wbr>Il2rMx4AtGwjVRzX89kobZ0W25vW8T<wbr>Gm0KJevLd7KQ8&e=</a>><br>
> > _Sven_Oehme_-_News_from_<wbr>Research.pdf<br>
> > _<a href="http://files.gpfsug.org/presentations/2016/SC16/12_-_" rel="noreferrer" target="_blank">http://files.gpfsug.org/<wbr>presentations/2016/SC16/12_-_</a><br>
> <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2016_SC16_12-5F-2D&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=u_qcvB--uvtByHp9H471EowagobMpPLXYT_FFzMkQiw&e=" rel="noreferrer" target="_blank">https://urldefense.<wbr>proofpoint.com/v2/url?u=http-<wbr>3A__files.gpfsug.org_<wbr>presentations_2016_SC16_12-5F-<wbr>2D&d=DwMFaQ&c=jf_iaSHvJObTbx-<wbr>siA1ZOg&r=<wbr>Ew59QH6nxuyx6oTs7a8AYX7kKG3gaW<wbr>UGDGo5ZZr3wQ4&m=<wbr>KLv9eH4GG8WlXC5ENj_<wbr>jXnzCpm60QSNAADfp6s94oa4&s=u_<wbr>qcvB--<wbr>uvtByHp9H471EowagobMpPLXYT_<wbr>FFzMkQiw&e=</a>><br>
<span class="">> > _Sven_Oehme_Dean_Hildebrand_-_<wbr>News_from_IBM_Research.pdf<br>
><br>
><br>
> > from those presentations regarding 32 subblocks:<br>
> ><br>
> > "It has a significant performance penalty for small files in large<br>
> > block size filesystems"<br>
><br>
> > although I'm not clear on the specific definition of "large". Many<br>
> > filesystems I encounter only have a 1M block size so it may not<br>
> > matter there, although that same presentation clearly shows the<br>
> > benefit of larger block sizes which is yet *another* thing for which<br>
> > a migration tool would be helpful.<br>
><br>
> > -Aaron<br>
> ><br>
> > On Wed, Nov 29, 2017 at 2:08 PM, Nikhil Khandelwal<br>
</span><div><div class="h5">> <_nikhilk@us.ibm.com_ <mailto:<a href="mailto:nikhilk@us.ibm.com">nikhilk@us.ibm.com</a>>> wrote:<br>
><br>
> > Hi,<br>
> ><br>
> > I would like to clarify migration path to 5.0.0 from 4.X.X clusters.<br>
> > For all Spectrum Scale clusters that are currently at 4.X.X, it is<br>
> > possible to migrate to 5.0.0 with no offline data migration and no<br>
> > need to move data. Once these clusters are at 5.0.0, they will<br>
> > benefit from the performance improvements, new features (such as<br>
> > file audit logging), and various enhancements that are included in<br>
> 5.0.0.<br>
> ><br>
> > That being said, there is one enhancement that will not be applied<br>
> > to these clusters, and that is the increased number of sub-blocks<br>
> > per block for small file allocation. This means that for file<br>
> > systems with a large block size and a lot of small files, the<br>
> > overall space utilization will be the same it currently is in 4.X.X.<br>
> > Since file systems created at 4.X.X and earlier used a block size<br>
> > that kept this allocation in mind, there should be very little<br>
> > impact on existing file systems.<br>
> ><br>
> > Outside of that one particular function, the remainder of the<br>
> > performance improvements, metadata improvements, updated<br>
> > compatibility, new functionality, and all of the other enhancements<br>
> > will be immediately available to you once you complete the upgrade<br>
> > to 5.0.0 -- with no need to reformat, move data, or take your data<br>
> offline.<br>
> ><br>
> > I hope that clarifies things a little and makes the upgrade path<br>
> > more accessible.<br>
> ><br>
> > Please let me know if there are any other questions or concerns.<br>
> ><br>
> > Thank you,<br>
> > Nikhil Khandelwal<br>
> > Spectrum Scale Development<br>
> > Client Adoption<br>
> ><br>
> > ______________________________<wbr>_________________<br>
> > gpfsug-discuss mailing list<br>
</div></div>> > gpfsug-discuss at _spectrumscale.org_<br>
> <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=" rel="noreferrer" target="_blank">https://urldefense.<wbr>proofpoint.com/v2/url?u=http-<wbr>3A__spectrumscale.org&d=<wbr>DwMFaQ&c=jf_iaSHvJObTbx-<wbr>siA1ZOg&r=<wbr>Ew59QH6nxuyx6oTs7a8AYX7kKG3gaW<wbr>UGDGo5ZZr3wQ4&m=<wbr>KLv9eH4GG8WlXC5ENj_<wbr>jXnzCpm60QSNAADfp6s94oa4&s=Q-<wbr>P8kRqnjsWB7ePz6YtA3U0xguo7-<wbr>lVWKmb_zyZPndE&e=</a>><br>
> > _<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss_" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss_</a><br>
> <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=" rel="noreferrer" target="_blank">https://urldefense.<wbr>proofpoint.com/v2/url?u=http-<wbr>3A__gpfsug.org_mailman_<wbr>listinfo_gpfsug-2Ddiscuss&d=<wbr>DwMFaQ&c=jf_iaSHvJObTbx-<wbr>siA1ZOg&r=<wbr>Ew59QH6nxuyx6oTs7a8AYX7kKG3gaW<wbr>UGDGo5ZZr3wQ4&m=<wbr>KLv9eH4GG8WlXC5ENj_<wbr>jXnzCpm60QSNAADfp6s94oa4&s=<wbr>WolSBY_<wbr>TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_<wbr>Y5xORkE&e=</a>><br>
><br>
> > ______________________________<wbr>_________________<br>
> > gpfsug-discuss mailing list<br>
> > gpfsug-discuss at _spectrumscale.org_<br>
> <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=" rel="noreferrer" target="_blank">https://urldefense.<wbr>proofpoint.com/v2/url?u=http-<wbr>3A__spectrumscale.org&d=<wbr>DwMFaQ&c=jf_iaSHvJObTbx-<wbr>siA1ZOg&r=<wbr>Ew59QH6nxuyx6oTs7a8AYX7kKG3gaW<wbr>UGDGo5ZZr3wQ4&m=<wbr>KLv9eH4GG8WlXC5ENj_<wbr>jXnzCpm60QSNAADfp6s94oa4&s=Q-<wbr>P8kRqnjsWB7ePz6YtA3U0xguo7-<wbr>lVWKmb_zyZPndE&e=</a>><br>
><br>
> > _<a href="https://urldefense.proofpoint.com/v2/url?_" rel="noreferrer" target="_blank">https://urldefense.<wbr>proofpoint.com/v2/url?_</a><br>
<span class="">> ><br>
> u=http-3A__gpfsug.org_mailman_<wbr>listinfo_gpfsug-2Ddiscuss&d=<wbr>DwICAg&c=jf_iaSHvJObTbx-<br>
> ><br>
> siA1ZOg&r=<wbr>Ew59QH6nxuyx6oTs7a8AYX7kKG3gaW<wbr>UGDGo5ZZr3wQ4&m=<wbr>DHoqgBeMFgcM0LpXEI0VCYvvb8ollc<wbr>t5aSYUDln2t68&s=iOxGm-853L_<wbr>W0XkB3jGsGzCTVlSYUvANOTSewcR_<wbr>Ue8&e=<br>
><br>
> ______________________________<wbr>_________________<br>
> gpfsug-discuss mailing list<br>
</span>> gpfsug-discuss at _spectrumscale.org_<br>
> <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=Q-P8kRqnjsWB7ePz6YtA3U0xguo7-lVWKmb_zyZPndE&e=" rel="noreferrer" target="_blank">https://urldefense.<wbr>proofpoint.com/v2/url?u=http-<wbr>3A__spectrumscale.org&d=<wbr>DwMFaQ&c=jf_iaSHvJObTbx-<wbr>siA1ZOg&r=<wbr>Ew59QH6nxuyx6oTs7a8AYX7kKG3gaW<wbr>UGDGo5ZZr3wQ4&m=<wbr>KLv9eH4GG8WlXC5ENj_<wbr>jXnzCpm60QSNAADfp6s94oa4&s=Q-<wbr>P8kRqnjsWB7ePz6YtA3U0xguo7-<wbr>lVWKmb_zyZPndE&e=</a>>_<br>
> __<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss_" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss_</a><br>
> <<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=" rel="noreferrer" target="_blank">https://urldefense.<wbr>proofpoint.com/v2/url?u=http-<wbr>3A__gpfsug.org_mailman_<wbr>listinfo_gpfsug-2Ddiscuss&d=<wbr>DwMFaQ&c=jf_iaSHvJObTbx-<wbr>siA1ZOg&r=<wbr>Ew59QH6nxuyx6oTs7a8AYX7kKG3gaW<wbr>UGDGo5ZZr3wQ4&m=<wbr>KLv9eH4GG8WlXC5ENj_<wbr>jXnzCpm60QSNAADfp6s94oa4&s=<wbr>WolSBY_<wbr>TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_<wbr>Y5xORkE&e=</a>>___________________<wbr>____________________________<br>
<span class="">> gpfsug-discuss mailing list<br>
> gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
</span>> <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Ew59QH6nxuyx6oTs7a8AYX7kKG3gaWUGDGo5ZZr3wQ4&m=KLv9eH4GG8WlXC5ENj_jXnzCpm60QSNAADfp6s94oa4&s=WolSBY_TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_Y5xORkE&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint.<wbr>com/v2/url?u=http-3A__gpfsug.<wbr>org_mailman_listinfo_gpfsug-<wbr>2Ddiscuss&d=DwICAg&c=jf_<wbr>iaSHvJObTbx-siA1ZOg&r=<wbr>Ew59QH6nxuyx6oTs7a8AYX7kKG3gaW<wbr>UGDGo5ZZr3wQ4&m=<wbr>KLv9eH4GG8WlXC5ENj_<wbr>jXnzCpm60QSNAADfp6s94oa4&s=<wbr>WolSBY_<wbr>TPJVJVPj5WEZ6JAbDZQK3j7oqn8u_<wbr>Y5xORkE&e=</a><br>
<span class="im HOEnZb">><br>
><br>
><br>
><br>
> ______________________________<wbr>_________________<br>
> gpfsug-discuss mailing list<br>
> gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</a><br>
><br>
<br>
</span><span class="HOEnZb"><font color="#888888">--<br>
Aaron Knister<br>
NASA Center for Climate Simulation (Code 606.2)<br>
Goddard Space Flight Center<br>
<a href="tel:%28301%29%20286-2776" value="+13012862776">(301) 286-2776</a><br>
</font></span><div class="HOEnZb"><div class="h5">______________________________<wbr>_________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</a><br>
</div></div></blockquote></div><br></div>