<div dir="ltr">We do have different amounts of space in the system pool which had the changes applied:<div><br></div><div><div>[root@scg4-hn01 ~]# mmdf gsfs0 -P system</div><div>disk disk size failure holds holds free KB free KB</div><div>name in KB group metadata data in full blocks in fragments</div><div>--------------- ------------- -------- -------- ----- -------------------- -------------------</div><div>Disks in storage pool: system (Maximum disk size allowed is 3.6 TB)</div><div>VD000 377487360 100 Yes No 143109120 ( 38%) 35708688 ( 9%) </div><div>DMD_NSD_804 377487360 100 Yes No 79526144 ( 21%) 2924584 ( 1%) </div><div>VD002 377487360 100 Yes No 143067136 ( 38%) 35713888 ( 9%) </div><div>DMD_NSD_802 377487360 100 Yes No 79570432 ( 21%) 2926672 ( 1%) </div><div>VD004 377487360 100 Yes No 143107584 ( 38%) 35727776 ( 9%) </div><div>DMD_NSD_805 377487360 200 Yes No 79555584 ( 21%) 2940040 ( 1%) </div><div>VD001 377487360 200 Yes No 142964992 ( 38%) 35805384 ( 9%) </div><div>DMD_NSD_803 377487360 200 Yes No 79580160 ( 21%) 2919560 ( 1%) </div><div>VD003 377487360 200 Yes No 143132672 ( 38%) 35764200 ( 9%) </div><div>DMD_NSD_801 377487360 200 Yes No 79550208 ( 21%) 2915232 ( 1%) </div><div> ------------- -------------------- -------------------</div><div>(pool total) 3774873600 1113164032 ( 29%) 193346024 ( 5%)</div></div><div><br></div><div><br></div><div>and mmldisk shows that there is a problem with replication:</div><div><br></div><div>...</div><div><div>Number of quorum disks: 5 </div><div>Read quorum value: 3</div><div>Write quorum value: 3</div><div>Attention: Due to an earlier configuration change the file system</div><div>is no longer properly replicated.</div></div><div><br></div><div><br></div><div>I thought a 'mmrestripe -r' would fix this, not that I have to fix it first before restriping?</div><div><br></div><div>jbh</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Nov 2, 2017 at 9:45 AM, Frederick Stock <span dir="ltr"><<a href="mailto:stockf@us.ibm.com" target="_blank">stockf@us.ibm.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><font size="2" face="sans-serif">Assuming you are replicating data and metadata
have you confirmed that all failure groups have the same free space? That
is could it be that one of your failure groups has less space than the
others? You can verify this with the output of mmdf and look at the
NSD sizes and space available.</font><br><br><font size="3" face="sans-serif">Fred<br>______________________________<wbr>____________________<br>Fred Stock | IBM Pittsburgh Lab | <a href="tel:(720)%20430-8821" value="+17204308821" target="_blank">720-430-8821</a><br><a href="mailto:stockf@us.ibm.com" target="_blank">stockf@us.ibm.com</a></font><br><br><br><br><font size="1" color="#5f5f5f" face="sans-serif">From:
</font><font size="1" face="sans-serif">John Hanks <<a href="mailto:griznog@gmail.com" target="_blank">griznog@gmail.com</a>></font><span class=""><br><font size="1" color="#5f5f5f" face="sans-serif">To:
</font><font size="1" face="sans-serif">gpfsug main discussion
list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.<wbr>org</a>></font><br></span><font size="1" color="#5f5f5f" face="sans-serif">Date:
</font><font size="1" face="sans-serif">11/02/2017 12:20 PM</font><span class=""><br><font size="1" color="#5f5f5f" face="sans-serif">Subject:
</font><font size="1" face="sans-serif">Re: [gpfsug-discuss]
mmrestripefs "No space left on device"</font><br></span><font size="1" color="#5f5f5f" face="sans-serif">Sent by:
</font><font size="1" face="sans-serif"><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@<wbr>spectrumscale.org</a></font><br><hr noshade><span class=""><br><br><br><font size="3">Addendum to last message:</font><br><br><font size="3">We haven't upgraded recently as far as I know (I just
inherited this a couple of months ago.) but am planning an outage soon
to upgrade from 4.2.0-4 to 4.2.3-5. </font><br><br><font size="3">My growing collection of output files generally contain
something like</font><br><br></span><font size="3">This inode list was generated in the Parallel Inode Traverse
on Thu Nov 2 08:34:22 2017</font><span class=""><br><font size="3">INODE_NUMBER DUMMY_INFO SNAPSHOT_ID ISGLOBAL_SNAPSHOT
INDEPENDENT_FSETID MEMO(INODE_FLAGS FILE_TYPE [ERROR])</font><br></span><span class=""><font size="3"> 53506 0:0
0 1
0
illreplicated REGULAR_FILE RESERVED
Error: 28 No space left on device</font><br><br><font size="3">With that inode varying slightly.</font><br><br></span><font size="3">jbh</font><br><span class=""><br><font size="3">On Thu, Nov 2, 2017 at 8:55 AM, Scott Fadden <</font><a href="mailto:sfadden@us.ibm.com" target="_blank"><font size="3" color="blue"><u>sfadden@us.ibm.com</u></font></a><font size="3">>
wrote:</font><br></span><div><div class="h5"><font size="2" face="Arial">Sorry just reread as I hit send and
saw this was mmrestripe, in my case it was mmdeledisk.</font><br><font size="2" face="Arial"> </font><br><font size="2" face="Arial">Did you try running the command on just one
pool. Or using -B instead?</font><br><font size="2" face="Arial"> </font><br><font size="2" face="Arial">What is the file it is complaining about
in "/var/mmfs/tmp/gsfs0.pit.<wbr>interestingInodes.12888779711" ?</font><br><font size="2" face="Arial"> </font><br><font size="2" face="Arial">Looks like it could be related to the maxfeaturelevel
of the cluster. Have you recently upgraded? Is everything up to the same
level? </font><br><font size="2" face="Arial"> </font><br><font size="2" face="Arial">Scott Fadden<br>Spectrum Scale - Technical Marketing<br>Phone: </font><a href="tel:(503)%20880-5833" target="_blank"><font size="2" color="blue" face="Arial"><u>(503)
880-5833</u></font></a><font size="2" color="blue" face="Arial"><u><br></u></font><a href="mailto:sfadden@us.ibm.com" target="_blank"><font size="2" color="blue" face="Arial"><u>sfadden@us.ibm.com</u></font></a><font size="2" color="blue" face="Arial"><u><br></u></font><a href="http://www.ibm.com/systems/storage/spectrum/scale" target="_blank"><font size="2" color="blue" face="Arial"><u>http://www.ibm.com/systems/<wbr>storage/spectrum/scale</u></font></a><br><font size="2" face="Arial"> </font><br><font size="2" face="Arial"> </font><br><font size="2" face="Arial">----- Original message -----<br>From: Scott Fadden/Portland/IBM<br>To: </font><a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank"><font size="2" color="blue" face="Arial"><u>gpfsug-discuss@spectrumscale.<wbr>org</u></font></a><font size="2" face="Arial"><br>Cc: </font><a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank"><font size="2" color="blue" face="Arial"><u>gpfsug-discuss@spectrumscale.<wbr>org</u></font></a><font size="2" face="Arial"><br>Subject: Re: [gpfsug-discuss] mmrestripefs "No space left on device"<br>Date: Thu, Nov 2, 2017 8:44 AM<br> </font><br><font size="2" face="Arial">I opened a defect on this the other day,
in my case it was an incorrect error message. What it meant to say was,"The
pool is not empty." Are you trying to remove the last disk in a pool?
If so did you empty the pool with a MIGRATE policy first? </font><br><font size="2" face="Arial"> </font><br><font size="2" face="Arial"> </font><br><font size="2" face="Arial">Scott Fadden<br>Spectrum Scale - Technical Marketing<br>Phone: </font><a href="tel:(503)%20880-5833" target="_blank"><font size="2" color="blue" face="Arial"><u>(503)
880-5833</u></font></a><font size="2" color="blue" face="Arial"><u><br></u></font><a href="mailto:sfadden@us.ibm.com" target="_blank"><font size="2" color="blue" face="Arial"><u>sfadden@us.ibm.com</u></font></a><font size="2" color="blue" face="Arial"><u><br></u></font><a href="http://www.ibm.com/systems/storage/spectrum/scale" target="_blank"><font size="2" color="blue" face="Arial"><u>http://www.ibm.com/systems/<wbr>storage/spectrum/scale</u></font></a><br><font size="2" face="Arial"> </font><br><font size="2" face="Arial"> </font><br><font size="2" face="Arial">----- Original message -----<br>From: John Hanks <</font><a href="mailto:griznog@gmail.com" target="_blank"><font size="2" color="blue" face="Arial"><u>griznog@gmail.com</u></font></a><font size="2" face="Arial">><br>Sent by: </font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank"><font size="2" color="blue" face="Arial"><u>gpfsug-discuss-bounces@<wbr>spectrumscale.org</u></font></a><font size="2" face="Arial"><br>To: gpfsug main discussion list <</font><a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank"><font size="2" color="blue" face="Arial"><u>gpfsug-discuss@spectrumscale.<wbr>org</u></font></a><font size="2" face="Arial">><br>Cc:<br>Subject: Re: [gpfsug-discuss] mmrestripefs "No space left on device"<br>Date: Thu, Nov 2, 2017 8:34 AM<br> </font><br><font size="2" face="Arial">We have no snapshots ( they were the first
to go when we initially hit the full metadata NSDs). </font><br><font size="2" face="Arial"> </font><br><font size="2" face="Arial">I've increased quotas so that no filesets
have hit a space quota. </font><br><font size="2" face="Arial"> </font><br><font size="2" face="Arial">Verified that there are no inode quotas anywhere.</font><br><font size="2" face="Arial"> </font><br><font size="2" face="Arial">mmdf shows the least amount of free space
on any nsd to be 9% free.</font><br><font size="2" face="Arial"> </font><br><font size="2" face="Arial">Still getting this error:</font><br><font size="2" face="Arial"> </font><br><font size="2" face="Arial">[root@scg-gs0 ~]# mmrestripefs gsfs0 -r -N
scg-gs0,scg-gs1,scg-gs2,scg-<wbr>gs3</font><br><font size="2" face="Arial">Scanning file system metadata, phase 1 ... </font><br><font size="2" face="Arial">Scan completed successfully.</font><br><font size="2" face="Arial">Scanning file system metadata, phase 2 ... </font><br><font size="2" face="Arial">Scanning file system metadata for sas0 storage
pool</font><br><font size="2" face="Arial">Scanning file system metadata for sata0 storage
pool</font><br><font size="2" face="Arial">Scan completed successfully.</font><br><font size="2" face="Arial">Scanning file system metadata, phase 3 ... </font><br><font size="2" face="Arial">Scan completed successfully.</font><br><font size="2" face="Arial">Scanning file system metadata, phase 4 ... </font><br><font size="2" face="Arial">Scan completed successfully.</font><br><font size="2" face="Arial">Scanning user file metadata ...</font><br><font size="2" face="Arial">Error processing user file metadata.</font><br><font size="2" face="Arial">No space left on device</font><br><font size="2" face="Arial">Check file '/var/mmfs/tmp/gsfs0.pit.<wbr>interestingInodes.12888779711'
on scg-gs0 for inodes with broken disk addresses or failures.</font><br><font size="2" face="Arial">mmrestripefs: Command failed. Examine previous
error messages to determine cause.</font><br><font size="2" face="Arial"> </font><br><font size="2" face="Arial">I should note too that this fails almost
immediately, far to quickly to fill up any location it could be trying
to write to.</font><br><font size="2" face="Arial"> </font><br><font size="2" face="Arial">jbh</font><br><font size="2" face="Arial"> </font><br><font size="2" face="Arial">On Thu, Nov 2, 2017 at 7:57 AM, David Johnson
<</font><a href="mailto:david_johnson@brown.edu" target="_blank"><font size="2" color="blue" face="Arial"><u>david_johnson@brown.edu</u></font></a><font size="2" face="Arial">>
wrote: </font><br><font size="2" face="Arial">One thing that may be relevant is if you
have snapshots, depending on your release level, </font><br><font size="2" face="Arial">inodes in the snapshot may considered immutable,
and will not be migrated. Once the snapshots</font><br><font size="2" face="Arial">have been deleted, the inodes are freed up
and you won’t see the (somewhat misleading) message</font><br><font size="2" face="Arial">about no space.</font><br><font size="2" face="Arial"> </font><br><font size="2" face="Arial"> — ddj</font><br><font size="2" face="Arial">Dave Johnson</font><br><font size="2" face="Arial">Brown University</font><br><font size="2" face="Arial"> </font><br><font size="2" face="Arial">On Nov 2, 2017, at 10:43 AM, John Hanks <</font><a href="mailto:griznog@gmail.com" target="_blank"><font size="2" color="blue" face="Arial"><u>griznog@gmail.com</u></font></a><font size="2" face="Arial">>
wrote:</font><br><font size="2" face="Arial">Thanks all for the suggestions. </font><br><font size="2" face="Arial"> </font><br><font size="2" face="Arial">Having our metadata NSDs fill up was what
prompted this exercise, but space was previously feed up on those by switching
them from metadata+data to metadataOnly and using a policy to migrate files
out of that pool. So these now have about 30% free space (more if you include
fragmented space). The restripe attempt is just to make a final move of
any remaining data off those devices. All the NSDs now have free space
on them.</font><br><font size="2" face="Arial"> </font><br><font size="2" face="Arial">df -i shows inode usage at about 84%, so
plenty of free inodes for the filesystem as a whole.</font><br><font size="2" face="Arial"> </font><br><font size="2" face="Arial">We did have old .quota files laying
around but removing them didn't have any impact. </font><br><font size="2" face="Arial"> </font><br><font size="2" face="Arial">mmlsfileset fs -L -i is taking a while to
complete, I'll let it simmer while getting to work.</font><br><font size="2" face="Arial"> </font><br><font size="2" face="Arial">mmrepquota does show about a half-dozen filesets
that have hit their quota for space (we don't set quotas on inodes). Once
I'm settled in this morning I'll try giving them a little extra space and
see what happens.</font><br><font size="2" face="Arial"> </font><br><font size="2" face="Arial">jbh</font><br><font size="2" face="Arial"> </font><br><font size="2" face="Arial"> </font><br><font size="2" face="Arial">On Thu, Nov 2, 2017 at 4:19 AM, Oesterlin,
Robert <</font><a href="mailto:Robert.Oesterlin@nuance.com" target="_blank"><font size="2" color="blue" face="Arial"><u>Robert.Oesterlin@nuance.com</u></font></a><font size="2" face="Arial">>
wrote: </font><br><font size="2" face="Arial">One thing that I’ve run into before is that
on older file systems you had the “*.quota” files in the file system
root. If you upgraded the file system to a newer version (so these files
aren’t used) - There was a bug at one time where these didn’t get properly
migrated during a restripe. Solution was to just remove them</font></div></div><p><font size="2" face="Arial"> </font></p><p><font size="2" face="Arial"> </font></p><p><font size="3" face="Arial">Bob Oesterlin</font></p><p></p><div><div class="h5"><font size="3" face="Arial">Sr Principal Storage Engineer, Nuance</font></div></div><p><font size="2" face="Arial"> </font></p><p></p><div><div class="h5"><font size="3" face="Arial"><b>From: </b><</font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank"><font size="3" color="blue" face="Arial"><u>gpfsug-discuss-bounces@<wbr>spectrumscale.org</u></font></a><font size="3" face="Arial">>
on behalf of John Hanks <</font><a href="mailto:griznog@gmail.com" target="_blank"><font size="3" color="blue" face="Arial"><u>griznog@gmail.com</u></font></a><font size="3" face="Arial">><b><br>Reply-To: </b>gpfsug main discussion list <</font><a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank"><font size="3" color="blue" face="Arial"><u>gpfsug-discuss@spectrumscale.<wbr>org</u></font></a><font size="3" face="Arial">><b><br>Date: </b>Wednesday, November 1, 2017 at 5:55 PM<b><br>To: </b>gpfsug <</font><a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank"><font size="3" color="blue" face="Arial"><u>gpfsug-discuss@spectrumscale.<wbr>org</u></font></a><font size="3" face="Arial">><b><br>Subject: </b>[EXTERNAL] [gpfsug-discuss] mmrestripefs "No space left
on device"</font></div></div><p><font size="2" face="Arial"> </font></p><p><a name="m_-4512031540600214981_m_-7946021508789305345_m_7792501056088615300_m_6737186114830475911__MailOriginalBody"></a><font size="2" face="Arial">Hi
all,</font></p><p><font size="2" face="Arial"> </font></p><p></p><div><div class="h5"><font size="2" face="Arial">I'm trying to do a restripe after setting
some nsds to metadataOnly and I keep running into this error:</font></div></div><p><font size="2" face="Arial"> </font></p><p></p><div><div class="h5"><font size="2" face="Arial">Scanning user file metadata ...</font></div></div><p></p><div><div class="h5"><font size="2" face="Arial"> 0.01 % complete on Wed Nov
1 15:36:01 2017 ( 40960 inodes with total
531689 MB data processed)</font></div></div><p></p><div><div class="h5"><font size="2" face="Arial">Error processing user file metadata. </font></div></div><p></p><div><div class="h5"><font size="2" face="Arial">Check file '/var/mmfs/tmp/gsfs0.pit.<wbr>interestingInodes.12888779708'
on scg-gs0 for inodes with broken disk addresses or failures.</font></div></div><p></p><div><div class="h5"><font size="2" face="Arial">mmrestripefs: Command failed. Examine previous
error messages to determine cause.</font></div></div><p><font size="2" face="Arial"> </font></p><p></p><div><div class="h5"><font size="2" face="Arial">The file it points to says:</font></div></div><p><font size="2" face="Arial"> </font></p><p></p><div><div class="h5"><font size="2" face="Arial">This inode list was generated in the Parallel
Inode Traverse on Wed Nov 1 15:36:06 2017</font></div></div><p></p><div><div class="h5"><font size="2" face="Arial">INODE_NUMBER DUMMY_INFO SNAPSHOT_ID ISGLOBAL_SNAPSHOT
INDEPENDENT_FSETID MEMO(INODE_FLAGS FILE_TYPE [ERROR])</font></div></div><p></p><div><div class="h5"><font size="2" face="Arial"> 53504 0:0
0 1
0
illreplicated REGULAR_FILE RESERVED
Error: 28 No space left on device</font></div></div><p><font size="2" face="Arial"> </font></p><p><font size="2" face="Arial"> </font></p><p></p><div><div class="h5"><font size="2" face="Arial">/var on the node I am running this on has
> 128 GB free, all the NSDs have plenty of free space, the filesystem
being restriped has plenty of free space and if I watch the node while
running this no filesystem on it even starts to get full. Could someone
tell me where mmrestripefs is attempting to write and/or how to point it
at a different location?</font></div></div><p><font size="2" face="Arial"> </font></p><p><font size="2" face="Arial">Thanks,</font></p><p><font size="2" face="Arial"> </font></p><p><font size="2" face="Arial">jbh</font></p><p></p><div><div class="h5"><font size="2" face="Arial">______________________________<wbr>_________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=WDtkF9zLTGGYqFnVnJ3rywZM6KHROA4FpMYi6cUkkKY&m=hKtOnoUDijNQoFnSlxQfek9m6h2qKbqjcCswbjHg2-E&s=n5P1NWESV2GUb3EXICXGj62_QDAPfSAWVPz_i59CNKk&e=" target="_blank"><font size="2" color="blue" face="Arial"><u>spectrumscale.org</u></font></a><font size="2" color="blue" face="Arial"><u><br></u></font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=uLFESUsuxpmf07haYD3Sl-DpeYkm3t_r0WVV2AZ9Jk0&s=zARWNuUgVecPk0qJwJdRIi0l_U9K7Z-xnnr5vNm1IZo&e=" target="_blank"><font size="2" color="blue" face="Arial"><u>http://gpfsug.org/mailman</u></font></a><font size="2" face="Arial">/<wbr>listinfo/gpfsug-discuss</font><br><font size="2" face="Arial"><br>______________________________<wbr>_________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=WDtkF9zLTGGYqFnVnJ3rywZM6KHROA4FpMYi6cUkkKY&m=hKtOnoUDijNQoFnSlxQfek9m6h2qKbqjcCswbjHg2-E&s=n5P1NWESV2GUb3EXICXGj62_QDAPfSAWVPz_i59CNKk&e=" target="_blank"><font size="2" color="blue" face="Arial"><u>spectrumscale.org</u></font></a><font size="2" color="blue" face="Arial"><u><br></u></font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=uLFESUsuxpmf07haYD3Sl-DpeYkm3t_r0WVV2AZ9Jk0&s=zARWNuUgVecPk0qJwJdRIi0l_U9K7Z-xnnr5vNm1IZo&e=" target="_blank"><font size="2" color="blue" face="Arial"><u>http://gpfsug.org/mailman</u></font></a><font size="2" face="Arial">/<wbr>listinfo/gpfsug-discuss<br> </font><br><tt><font size="2">______________________________<wbr>_________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font></tt><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=uLFESUsuxpmf07haYD3Sl-DpeYkm3t_r0WVV2AZ9Jk0&s=DGJeAf81dkJPqeCYJhjPiOUDTCAVRO-KEsvBx-HSzUM&e=" target="_blank"><tt><font size="2" color="blue"><u>spectrumscale.org</u></font></tt></a><br><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=WDtkF9zLTGGYqFnVnJ3rywZM6KHROA4FpMYi6cUkkKY&m=hKtOnoUDijNQoFnSlxQfek9m6h2qKbqjcCswbjHg2-E&s=j7eYU1VnwYXrTnflbJki13EfnMjqAro0RdCiLkVrgzE&e=" target="_blank"><tt><font size="2" color="blue"><u>https://urldefense.proofpoint.<wbr>com/v2/url?u=http-3A__gpfsug.<wbr>org_mailman_listinfo_gpfsug-<wbr>2Ddiscuss&d=DwICAg&c=jf_<wbr>iaSHvJObTbx-siA1ZOg&r=<wbr>WDtkF9zLTGGYqFnVnJ3rywZM6KHROA<wbr>4FpMYi6cUkkKY&m=<wbr>hKtOnoUDijNQoFnSlxQfek9m6h2qKb<wbr>qjcCswbjHg2-E&s=<wbr>j7eYU1VnwYXrTnflbJki13EfnMjqAr<wbr>o0RdCiLkVrgzE&e=</u></font></tt></a><br><font size="2" face="Arial"> </font><br><font size="2" face="Arial"> </font><br><font size="3"><br><br>______________________________<wbr>_________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=uLFESUsuxpmf07haYD3Sl-DpeYkm3t_r0WVV2AZ9Jk0&s=DGJeAf81dkJPqeCYJhjPiOUDTCAVRO-KEsvBx-HSzUM&e=" target="_blank"><font size="3" color="blue"><u>spectrumscale.org</u></font></a><font size="3" color="blue"><u><br></u></font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=uLFESUsuxpmf07haYD3Sl-DpeYkm3t_r0WVV2AZ9Jk0&s=RGgSZEisfDpxsKl3PFUWh6DtzD_FF6spqHVpo_0joLY&e=" target="_blank"><font size="3" color="blue"><u>http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</u></font></a><font size="3"><br></font><br><tt><font size="2">______________________________<wbr>_________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br></font></tt></div></div><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=uLFESUsuxpmf07haYD3Sl-DpeYkm3t_r0WVV2AZ9Jk0&s=RGgSZEisfDpxsKl3PFUWh6DtzD_FF6spqHVpo_0joLY&e=" target="_blank"><tt><font size="2">https://urldefense.proofpoint.<wbr>com/v2/url?u=http-3A__gpfsug.<wbr>org_mailman_listinfo_gpfsug-<wbr>2Ddiscuss&d=DwICAg&c=jf_<wbr>iaSHvJObTbx-siA1ZOg&r=p_<wbr>1XEUyoJ7-VJxF_w8h9gJh8_<wbr>Wj0Pey73LCLLoxodpw&m=<wbr>uLFESUsuxpmf07haYD3Sl-<wbr>DpeYkm3t_r0WVV2AZ9Jk0&s=<wbr>RGgSZEisfDpxsKl3PFUWh6DtzD_<wbr>FF6spqHVpo_0joLY&e=</font></tt></a><tt><font size="2"><br></font></tt><br><br><br>
<p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><br>______________________________<wbr>_________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</a><br>
<br></blockquote></div><br></div>