<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
All,
<div class=""><br class="">
</div>
<div class="">Sorry for the 2nd e-mail but I realize that 4 MB is 4 times 1 MB … so does this go back to what Marc is saying that there’s really only one sub blocks per block parameter? If so, is there any way to get what I want as described below?</div>
<div class=""><br class="">
</div>
<div class="">Thanks…</div>
<div class=""><br class="">
</div>
<div class="">Kevin</div>
<div class=""><br class="">
<div class="">
<div class="">—</div>
<div class="">Kevin Buterbaugh - Senior System Administrator</div>
<div class="">Vanderbilt University - Advanced Computing Center for Research and Education</div>
<div class=""><a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" class="">Kevin.Buterbaugh@vanderbilt.edu</a> - (615)875-9633</div>
<div class=""><br class="">
</div>
</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Aug 1, 2018, at 1:47 PM, Buterbaugh, Kevin L <<a href="mailto:Kevin.Buterbaugh@Vanderbilt.Edu" class="">Kevin.Buterbaugh@Vanderbilt.Edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi Sven,
<div class=""><br class="">
</div>
<div class="">OK … but why? I mean, that’s not what the man page says. Where does that “4 x” come from?</div>
<div class=""><br class="">
</div>
<div class="">And, most importantly … that’s not what I want. I want a smaller block size for the system pool since it’s metadata only and on RAID 1 mirrors (HD’s on the test cluster but SSD’s on the production cluster). So … side question … is 1 MB OK there?</div>
<div class=""><br class="">
</div>
<div class="">But I want a 4 MB block size for data with an 8 KB sub block … I want good performance for the sane people using our cluster without unduly punishing the … ahem … fine folks whose apps want to create a bazillion tiny files!</div>
<div class=""><br class="">
</div>
<div class="">So how do I do that?</div>
<div class=""><br class="">
</div>
<div class="">Thanks!</div>
<div class=""><br class="">
<div class="">
<div class="">—</div>
<div class="">Kevin Buterbaugh - Senior System Administrator</div>
<div class="">Vanderbilt University - Advanced Computing Center for Research and Education</div>
<div class=""><a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" class="">Kevin.Buterbaugh@vanderbilt.edu</a> - (615)875-9633</div>
<div class=""><br class="">
</div>
</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Aug 1, 2018, at 1:41 PM, Sven Oehme <<a href="mailto:oehmes@gmail.com" class="">oehmes@gmail.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">the number of subblocks is derived by the smallest blocksize in any pool of a given filesystem. so if you pick a metadata blocksize of 1M it will be 8k in the metadata pool, but 4 x of that in the data pool if your data pool is 4M.
<div class=""><br class="">
</div>
<div class="">sven</div>
<div class=""><br class="">
<br class="">
<div class="gmail_quote">
<div dir="ltr" class="">On Wed, Aug 1, 2018 at 11:21 AM Felipe Knop <<a href="mailto:knop@us.ibm.com" class="">knop@us.ibm.com</a>> wrote:<br class="">
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="">
<p class=""><font size="2" class="">Marc, Kevin,</font><br class="">
<br class="">
<font size="2" class="">We'll be looking into this issue, since at least at a first glance, it does look odd. A 4MB block size should have resulted in an 8KB subblock size. I suspect that, somehow, the
</font><b class=""><font size="2" class="">--metadata-block-size</font></b><b class=""><font size="2" class=""> 1M</font></b><font size="2" class=""> may have resulted in
</font><br class="">
</p>
</div>
<div class="">
<p class=""><br class="">
32768 Minimum fragment (subblock) size in bytes (other pools)<br class="">
<br class="">
</p>
</div>
<div class="">
<p class=""><font size="2" class="">but I do not yet understand how.</font><br class="">
<br class="">
<font size="2" class="">The </font><b class=""><font size="2" class="">subblocks-per-full-block</font></b><font size="2" class=""> parameter is not supported with
</font><b class=""><font size="2" class="">mmcrfs </font></b><font size="2" class="">.</font><br class="">
<br class="">
<font size="2" class="">Felipe</font><br class="">
<br class="">
<font size="2" class="">----<br class="">
Felipe Knop <a href="mailto:knop@us.ibm.com" target="_blank" class="">knop@us.ibm.com</a><br class="">
GPFS Development and Security<br class="">
IBM Systems<br class="">
IBM Building 008<br class="">
2455 South Rd, Poughkeepsie, NY 12601<br class="">
<a href="tel:(845)%20433-9314" value="+18454339314" target="_blank" class="">(845) 433-9314</a> T/L 293-9314<br class="">
<br class="">
</font><br class="">
<br class="">
</p>
<span id="cid:164f6c93466308fca931" class=""><graycol.gif></span><font size="2" color="#424282" class="">"Marc A Kaplan" ---08/01/2018 01:21:23 PM---I haven't looked into all the details but here's a clue -- notice there is only one "subblocks-per-</font><br class="">
<br class="">
<font size="2" color="#5F5F5F" class="">From: </font><font size="2" class="">"Marc A Kaplan" <<a href="mailto:makaplan@us.ibm.com" target="_blank" class="">makaplan@us.ibm.com</a>></font>
<div class=""><br class="webkit-block-placeholder">
</div>
</div>
<div class="">
<p class=""><br class="">
<font size="2" color="#5F5F5F" class="">To: </font><font size="2" class="">gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank" class="">gpfsug-discuss@spectrumscale.org</a>></font><br class="">
</p>
</div>
<div class="">
<p class=""><font size="2" color="#5F5F5F" class="">Date: </font><font size="2" class="">08/01/2018 01:21 PM</font><br class="">
<font size="2" color="#5F5F5F" class="">Subject: </font><font size="2" class="">Re: [gpfsug-discuss] Sub-block size wrong on GPFS 5 filesystem?</font></p>
</div>
<div class="">
<p class=""><br class="">
<font size="2" color="#5F5F5F" class="">Sent by: </font><font size="2" class=""><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank" class="">gpfsug-discuss-bounces@spectrumscale.org</a></font><br class="">
</p>
<hr width="100%" size="2" align="left" noshade="" style="color:#8091a5" class="">
<br class="">
<br class="">
<br class="">
<font size="2" class="">I haven't looked into all the details but here's a clue -- notice there is only one "subblocks-per-full-block" parameter.
</font><br class="">
<font size="2" class=""><br class="">
And it is the same for both metadata blocks and datadata blocks.<br class="">
<br class="">
So maybe (MAYBE) that is a constraint somewhere...</font><br class="">
<font size="2" class=""><br class="">
Certainly, in the currently supported code, that's what you get.</font><br class="">
<br class="">
<br class="">
<br class="">
<font size="2" color="#5F5F5F" class=""><br class="">
From: </font><font size="2" class="">"Buterbaugh, Kevin L" <<a href="mailto:Kevin.Buterbaugh@Vanderbilt.Edu" class="">Kevin.Buterbaugh@Vanderbilt.Edu</a>></font><font size="2" color="#5F5F5F" class=""><br class="">
To: </font><font size="2" class="">gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank" class="">gpfsug-discuss@spectrumscale.org</a>></font><font size="2" color="#5F5F5F" class=""><br class="">
Date: </font><font size="2" class="">08/01/2018 12:55 PM</font><font size="2" color="#5F5F5F" class=""><br class="">
Subject: </font><font size="2" class="">[gpfsug-discuss] Sub-block size wrong on GPFS 5 filesystem?</font><font size="2" color="#5F5F5F" class=""><br class="">
Sent by: </font><font size="2" class=""><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank" class="">gpfsug-discuss-bounces@spectrumscale.org</a></font><br class="">
<hr width="100%" size="2" align="left" noshade="" class="">
<br class="">
<br class="">
<br class="">
Hi All, <br class="">
<br class="">
Our production cluster is still on GPFS 4.2.3.x, but in preparation for moving to GPFS 5 I have upgraded our small (7 node) test cluster to GPFS 5.0.1-1. I am setting up a new filesystem there using hardware that we recently life-cycled out of our production
environment.<br class="">
<br class="">
I “successfully” created a filesystem but I believe the sub-block size is wrong. I’m using a 4 MB filesystem block size, so according to the mmcrfs man page the sub-block size should be 8K:<br class="">
<br class="">
Table 1. Block sizes and subblock sizes<br class="">
<br class="">
+‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+<br class="">
| Block size | Subblock size |<br class="">
+‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+<br class="">
| 64 KiB | 2 KiB |<br class="">
+‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+<br class="">
| 128 KiB | 4 KiB |<br class="">
+‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+<br class="">
| 256 KiB, 512 KiB, 1 MiB, 2 | 8 KiB |<br class="">
| MiB, 4 MiB | |<br class="">
+‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+<br class="">
| 8 MiB, 16 MiB | 16 KiB |<br class="">
+‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+<br class="">
<br class="">
However, it appears that it’s 8K for the system pool but 32K for the other pools:<br class="">
<br class="">
flag value description<br class="">
------------------- ------------------------ -----------------------------------<br class="">
-f 8192 Minimum fragment (subblock) size in bytes (system pool)<br class="">
32768 Minimum fragment (subblock) size in bytes (other pools)<br class="">
-i 4096 Inode size in bytes<br class="">
-I 32768 Indirect block size in bytes<br class="">
-m 2 Default number of metadata replicas<br class="">
-M 3 Maximum number of metadata replicas<br class="">
-r 1 Default number of data replicas<br class="">
-R 3 Maximum number of data replicas<br class="">
-j scatter Block allocation type<br class="">
-D nfs4 File locking semantics in effect<br class="">
-k all ACL semantics in effect<br class="">
-n 32 Estimated number of nodes that will mount file system<br class="">
-B 1048576 Block size (system pool)<br class="">
4194304 Block size (other pools)<br class="">
-Q user;group;fileset Quotas accounting enabled<br class="">
user;group;fileset Quotas enforced<br class="">
none Default quotas enabled<br class="">
--perfileset-quota No Per-fileset quota enforcement<br class="">
--filesetdf No Fileset df enabled?<br class="">
-V 19.01 (5.0.1.0) File system version<br class="">
--create-time Wed Aug 1 11:39:39 2018 File system creation time<br class="">
-z No Is DMAPI enabled?<br class="">
-L 33554432 Logfile size<br class="">
-E Yes Exact mtime mount option<br class="">
-S relatime Suppress atime mount option<br class="">
-K whenpossible Strict replica allocation option<br class="">
--fastea Yes Fast external attributes enabled?<br class="">
--encryption No Encryption enabled?<br class="">
--inode-limit 101095424 Maximum number of inodes<br class="">
--log-replicas 0 Number of log replicas<br class="">
--is4KAligned Yes is4KAligned?<br class="">
--rapid-repair Yes rapidRepair enabled?<br class="">
--write-cache-threshold 0 HAWC Threshold (max 65536)<br class="">
--subblocks-per-full-block 128 Number of subblocks per full block<br class="">
-P system;raid1;raid6 Disk storage pools in file system<br class="">
--file-audit-log No File Audit Logging enabled?<br class="">
--maintenance-mode No Maintenance Mode enabled?<br class="">
-d test21A3nsd;test21A4nsd;test21B3nsd;test21B4nsd;test23Ansd;test23Bnsd;test23Cnsd;test24Ansd;test24Bnsd;test24Cnsd;test25Ansd;test25Bnsd;test25Cnsd Disks in file system<br class="">
-A yes Automatic mount option<br class="">
-o none Additional mount options<br class="">
-T /gpfs5 Default mount point<br class="">
--mount-priority 0 Mount priority<br class="">
<br class="">
Output of mmcrfs:<br class="">
<br class="">
mmcrfs gpfs5 -F ~/gpfs/gpfs5.stanza -A yes -B 4M -E yes -i 4096 -j scatter -k all -K whenpossible -m 2 -M 3 -n 32 -Q yes -r 1 -R 3 -T /gpfs5 -v yes --nofilesetdf --metadata-block-size 1M<br class="">
<br class="">
The following disks of gpfs5 will be formatted on node testnsd3:<br class="">
test21A3nsd: size 953609 MB<br class="">
test21A4nsd: size 953609 MB<br class="">
test21B3nsd: size 953609 MB<br class="">
test21B4nsd: size 953609 MB<br class="">
test23Ansd: size 15259744 MB<br class="">
test23Bnsd: size 15259744 MB<br class="">
test23Cnsd: size 1907468 MB<br class="">
test24Ansd: size 15259744 MB<br class="">
test24Bnsd: size 15259744 MB<br class="">
test24Cnsd: size 1907468 MB<br class="">
test25Ansd: size 15259744 MB<br class="">
test25Bnsd: size 15259744 MB<br class="">
test25Cnsd: size 1907468 MB<br class="">
Formatting file system ...<br class="">
Disks up to size 8.29 TB can be added to storage pool system.<br class="">
Disks up to size 16.60 TB can be added to storage pool raid1.<br class="">
Disks up to size 132.62 TB can be added to storage pool raid6.<br class="">
Creating Inode File<br class="">
8 % complete on Wed Aug 1 11:39:19 2018<br class="">
18 % complete on Wed Aug 1 11:39:24 2018<br class="">
27 % complete on Wed Aug 1 11:39:29 2018<br class="">
37 % complete on Wed Aug 1 11:39:34 2018<br class="">
48 % complete on Wed Aug 1 11:39:39 2018<br class="">
60 % complete on Wed Aug 1 11:39:44 2018<br class="">
72 % complete on Wed Aug 1 11:39:49 2018<br class="">
83 % complete on Wed Aug 1 11:39:54 2018<br class="">
95 % complete on Wed Aug 1 11:39:59 2018<br class="">
100 % complete on Wed Aug 1 11:40:01 2018<br class="">
Creating Allocation Maps<br class="">
Creating Log Files<br class="">
3 % complete on Wed Aug 1 11:40:07 2018<br class="">
28 % complete on Wed Aug 1 11:40:14 2018<br class="">
53 % complete on Wed Aug 1 11:40:19 2018<br class="">
78 % complete on Wed Aug 1 11:40:24 2018<br class="">
100 % complete on Wed Aug 1 11:40:25 2018<br class="">
Clearing Inode Allocation Map<br class="">
Clearing Block Allocation Map<br class="">
Formatting Allocation Map for storage pool system<br class="">
85 % complete on Wed Aug 1 11:40:32 2018<br class="">
100 % complete on Wed Aug 1 11:40:33 2018<br class="">
Formatting Allocation Map for storage pool raid1<br class="">
53 % complete on Wed Aug 1 11:40:38 2018<br class="">
100 % complete on Wed Aug 1 11:40:42 2018<br class="">
Formatting Allocation Map for storage pool raid6<br class="">
20 % complete on Wed Aug 1 11:40:47 2018<br class="">
39 % complete on Wed Aug 1 11:40:52 2018<br class="">
60 % complete on Wed Aug 1 11:40:57 2018<br class="">
79 % complete on Wed Aug 1 11:41:02 2018<br class="">
100 % complete on Wed Aug 1 11:41:08 2018<br class="">
Completed creation of file system /dev/gpfs5.<br class="">
mmcrfs: Propagating the cluster configuration data to all<br class="">
affected nodes. This is an asynchronous process.<br class="">
<br class="">
And contents of stanza file:<br class="">
<br class="">
%nsd:<br class="">
nsd=test21A3nsd<br class="">
usage=metadataOnly<br class="">
failureGroup=210<br class="">
pool=system<br class="">
servers=testnsd3,testnsd1,testnsd2<br class="">
device=dm-15<br class="">
<br class="">
%nsd:<br class="">
nsd=test21A4nsd<br class="">
usage=metadataOnly<br class="">
failureGroup=210<br class="">
pool=system<br class="">
servers=testnsd1,testnsd2,testnsd3<br class="">
device=dm-14<br class="">
<br class="">
%nsd:<br class="">
nsd=test21B3nsd<br class="">
usage=metadataOnly<br class="">
failureGroup=211<br class="">
pool=system<br class="">
servers=testnsd1,testnsd2,testnsd3<br class="">
device=dm-17<br class="">
<br class="">
%nsd:<br class="">
nsd=test21B4nsd<br class="">
usage=metadataOnly<br class="">
failureGroup=211<br class="">
pool=system<br class="">
servers=testnsd2,testnsd3,testnsd1<br class="">
device=dm-16<br class="">
<br class="">
%nsd:<br class="">
nsd=test23Ansd<br class="">
usage=dataOnly<br class="">
failureGroup=23<br class="">
pool=raid6<br class="">
servers=testnsd2,testnsd3,testnsd1<br class="">
device=dm-10<br class="">
<br class="">
%nsd:<br class="">
nsd=test23Bnsd<br class="">
usage=dataOnly<br class="">
failureGroup=23<br class="">
pool=raid6<br class="">
servers=testnsd3,testnsd1,testnsd2<br class="">
device=dm-9<br class="">
<br class="">
%nsd:<br class="">
nsd=test23Cnsd<br class="">
usage=dataOnly<br class="">
failureGroup=23<br class="">
pool=raid1<br class="">
servers=testnsd1,testnsd2,testnsd3<br class="">
device=dm-5<br class="">
<br class="">
%nsd:<br class="">
nsd=test24Ansd<br class="">
usage=dataOnly<br class="">
failureGroup=24<br class="">
pool=raid6<br class="">
servers=testnsd3,testnsd1,testnsd2<br class="">
device=dm-6<br class="">
<br class="">
%nsd:<br class="">
nsd=test24Bnsd<br class="">
usage=dataOnly<br class="">
failureGroup=24<br class="">
pool=raid6<br class="">
servers=testnsd1,testnsd2,testnsd3<br class="">
device=dm-0<br class="">
<br class="">
%nsd:<br class="">
nsd=test24Cnsd<br class="">
usage=dataOnly<br class="">
failureGroup=24<br class="">
pool=raid1<br class="">
servers=testnsd2,testnsd3,testnsd1<br class="">
device=dm-2<br class="">
<br class="">
%nsd:<br class="">
nsd=test25Ansd<br class="">
usage=dataOnly<br class="">
failureGroup=25<br class="">
pool=raid6<br class="">
servers=testnsd1,testnsd2,testnsd3<br class="">
device=dm-6<br class="">
<br class="">
%nsd:<br class="">
nsd=test25Bnsd<br class="">
usage=dataOnly<br class="">
failureGroup=25<br class="">
pool=raid6<br class="">
servers=testnsd2,testnsd3,testnsd1<br class="">
device=dm-6<br class="">
<br class="">
%nsd:<br class="">
nsd=test25Cnsd<br class="">
usage=dataOnly<br class="">
failureGroup=25<br class="">
pool=raid1<br class="">
servers=testnsd3,testnsd1,testnsd2<br class="">
device=dm-3<br class="">
<br class="">
%pool:<br class="">
pool=system<br class="">
blockSize=1M<br class="">
usage=metadataOnly<br class="">
layoutMap=scatter<br class="">
allowWriteAffinity=no<br class="">
<br class="">
%pool:<br class="">
pool=raid6<br class="">
blockSize=4M<br class="">
usage=dataOnly<br class="">
layoutMap=scatter<br class="">
allowWriteAffinity=no<br class="">
<br class="">
%pool:<br class="">
pool=raid1<br class="">
blockSize=4M<br class="">
usage=dataOnly<br class="">
layoutMap=scatter<br class="">
allowWriteAffinity=no<br class="">
<br class="">
What am I missing or what have I done wrong? Thanks…<br class="">
<br class="">
Kevin<br class="">
—<br class="">
Kevin Buterbaugh - Senior System Administrator<br class="">
Vanderbilt University - Advanced Computing Center for Research and Education<u class=""><font color="#0000FF" class=""><br class="">
</font></u><a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" target="_blank" class=""><u class=""><font color="#0000FF" class="">Kevin.Buterbaugh@vanderbilt.edu</font></u></a>-
<a href="tel:(615)%20875-9633" value="+16158759633" target="_blank" class="">(615)875-9633</a><br class="">
<br class="">
<tt class=""><font size="2" class=""><br class="">
_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C8a00ac1e037d45913c8708d5f7de60ac%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636687456834171332&sdata=sFB5TXhhOddzDjupY8G04%2FUb%2BWKO6UDsaS0lWcBsAVE%3D&reserved=0" originalsrc="http://spectrumscale.org" shash="Ok1IHIxwDaRIDGhf1qwq7nKfOIG1JE1mK9Dq/fFDZqnuNHKHxmgtr/Vs6eWzyZD0RKfTB4CEkIpLqRigBz0wNM00V+Nfz+N7EAlj1blhfxRVHb57XbHJgFGwTjd0NgTGsEiSnct2MKMi1zUL68TpluHlsbFEeOfG+Wax7wL4Qv0=" target="_blank" class="">
spectrumscale.org</a></font></tt><u class=""><font color="#0000FF" class=""><br class="">
</font></u><a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C8a00ac1e037d45913c8708d5f7de60ac%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636687456834181344&sdata=iyZVZSpq2Z3e6xzMKa2nACI8GATEqkGOaqrZyuvZMjc%3D&reserved=0" originalsrc="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" shash="yANLdOahZCwVdRd12xI7q/wZWTOxJM+VkOtAaQlXhpjQf13efYKRxqQJkkVFNfjcREnQLrzGxxktthHjT1shDv3/ppueerccgR5lcVepn2kOMhAQt85FZPXiQ6gj+DlmQkZfVeUxd/bUdfXXLWBxjek6fnj9hrHbjIbLet0mZJQ=" target="_blank" class=""><tt class=""><u class=""><font size="2" color="#0000FF" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></u></tt></a><br class="">
<br class="">
<br class="">
<tt class=""><font size="2" class="">_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C8a00ac1e037d45913c8708d5f7de60ac%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636687456834191353&sdata=AGpbm%2BxjIycToPKKP9Amtzzl6jAn59e3d3kr9R7Setc%3D&reserved=0" originalsrc="http://spectrumscale.org" shash="B9A5q/0L6X2Bz4CJS58V+px60mUlUJ7QiZ0t+RGkuOfqfGDJvDwxu4bZ1N9xTPfPryaLa65samtNIgYP9zBn7Vyxph9SfXqsPhyVi+SefpPVJFFN3CNABsEnxlzEZVLPQSTw4BbN/VegmiVfxz6OjY8M1aoHvly9rFri0ZXimnQ=" target="_blank" class="">
spectrumscale.org</a><br class="">
</font></tt><tt class=""><font size="2" class=""><a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C8a00ac1e037d45913c8708d5f7de60ac%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636687456834191353&sdata=2csVvV7tvgg8fMM01RLj5fY8uhvIK44k4hRsD9vjuV0%3D&reserved=0" originalsrc="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" shash="BEbhAre5gljrCJDIxKVAWVesEbsHQ05+XL76f0pze8ujhdvzCmAZ82dFMbQ0+zXxEq1Sq1ktdFwNlItyqLFnxu+rlifvAuyyVP1Csz62y76a9YFxswN+04XCWavTmEWZX8dA3J27jtF0WY8e7uQn0G8FawMsrAXpD/o1yZHE5jg=" target="_blank" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></tt><tt class=""><font size="2" class=""><br class="">
</font></tt><br class="">
<br class="">
<br class="">
<div class=""><br class="webkit-block-placeholder">
</div>
</div>
_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C8a00ac1e037d45913c8708d5f7de60ac%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636687456834201361&sdata=hsZ8eOtS9sQhGAh76vk3UY3KTpol0VCfAVaD6Kw9m00%3D&reserved=0" originalsrc="http://spectrumscale.org" shash="hunx9yA6Vxk6U5q+y9L0g36h6dtnl0g9x34DsPsRw5VB5vp9Bfa7eDR/Blf5vkr2luGQN2wJrNMt+/Il1AjzQEf7JCNo/JGIiVbNiVg8szBfPmnNVy1eSNyWno/m1bdky8VVEXbNtns94SY5o0+aKFCpzyO9ZFNJE2Y2pfFBRxQ=" rel="noreferrer" target="_blank" class="">
spectrumscale.org</a><br class="">
<a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C8a00ac1e037d45913c8708d5f7de60ac%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636687456834211369&sdata=enjtshAXuqo0g6fqmUJOnCKL88MujJuDUWTXdauvx2A%3D&reserved=0" originalsrc="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" shash="pX+zZN5l1FHVWU9Urr4MGNrXSMs8ZzFB2BKpylknMoa9rmp2aoOmxSSbykwOM+OPcludGdpVQaakeGwwZY+pvDGWTvoHHrqim5wY/ykG73NR0nAzjXi0X8B+T8LVsUCEXHztVy32YFU7BSSgLfSiZ2xG0CXq6+4faBZBRtrlEHM=" rel="noreferrer" target="_blank" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="">
</blockquote>
</div>
</div>
</div>
_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="http://spectrumscale.org/" class="">spectrumscale.org</a><br class="">
<a href="https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C8a00ac1e037d45913c8708d5f7de60ac%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636687456834221377&sdata=MuPoxpCweqPxLR%2FAaWIgP%2BIkh0bUEVeG3cCzwoZoyE0%3D&reserved=0" class="">https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C8a00ac1e037d45913c8708d5f7de60ac%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636687456834221377&sdata=MuPoxpCweqPxLR%2FAaWIgP%2BIkh0bUEVeG3cCzwoZoyE0%3D&reserved=0</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>