<font size=2 face="sans-serif">Did you run the tsfindinode command to
see where that file is located? Also, what does the mmdf show for
your other pools notably the sas0 storage pool?</font><br><br><font size=3 face="sans-serif">Fred<br>__________________________________________________<br>Fred Stock | IBM Pittsburgh Lab | 720-430-8821<br>stockf@us.ibm.com</font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">John Hanks <griznog@gmail.com></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug main discussion
list <gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">11/02/2017 01:17 PM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
mmrestripefs "No space left on device"</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><font size=3>We do have different amounts of space in the system pool
which had the changes applied:</font><br><br><font size=3>[root@scg4-hn01 ~]# mmdf gsfs0 -P system</font><br><font size=3>disk
disk size failure holds holds
free KB free
KB</font><br><font size=3>name
in KB group metadata data
in full blocks in fragments</font><br><font size=3>--------------- ------------- -------- -------- -----
-------------------- -------------------</font><br><font size=3>Disks in storage pool: system (Maximum disk size allowed
is 3.6 TB)</font><br><font size=3>VD000
377487360 100 Yes No
143109120 ( 38%) 35708688 ( 9%) </font><br><font size=3>DMD_NSD_804 377487360
100 Yes No
79526144 ( 21%) 2924584 ( 1%) </font><br><font size=3>VD002
377487360 100 Yes No
143067136 ( 38%) 35713888 ( 9%) </font><br><font size=3>DMD_NSD_802 377487360
100 Yes No
79570432 ( 21%) 2926672 ( 1%) </font><br><font size=3>VD004
377487360 100 Yes No
143107584 ( 38%) 35727776 ( 9%) </font><br><font size=3>DMD_NSD_805 377487360
200 Yes No
79555584 ( 21%) 2940040 ( 1%) </font><br><font size=3>VD001
377487360 200 Yes No
142964992 ( 38%) 35805384 ( 9%) </font><br><font size=3>DMD_NSD_803 377487360
200 Yes No
79580160 ( 21%) 2919560 ( 1%) </font><br><font size=3>VD003
377487360 200 Yes No
143132672 ( 38%) 35764200 ( 9%) </font><br><font size=3>DMD_NSD_801 377487360
200 Yes No
79550208 ( 21%) 2915232 ( 1%) </font><br><font size=3>
-------------
-------------------- -------------------</font><br><font size=3>(pool total) 3774873600
1113164032 ( 29%) 193346024 ( 5%)</font><br><br><br><font size=3>and mmldisk shows that there is a problem with replication:</font><br><br><font size=3>...</font><br><font size=3>Number of quorum disks: 5 </font><br><font size=3>Read quorum value: 3</font><br><font size=3>Write quorum value: 3</font><br><font size=3>Attention: Due to an earlier configuration change the
file system</font><br><font size=3>is no longer properly replicated.</font><br><br><br><font size=3>I thought a 'mmrestripe -r' would fix this, not that I
have to fix it first before restriping?</font><br><br><font size=3>jbh</font><br><br><br><font size=3>On Thu, Nov 2, 2017 at 9:45 AM, Frederick Stock <</font><a href=mailto:stockf@us.ibm.com target=_blank><font size=3 color=blue><u>stockf@us.ibm.com</u></font></a><font size=3>>
wrote:</font><br><font size=2 face="sans-serif">Assuming you are replicating data and
metadata have you confirmed that all failure groups have the same free
space? That is could it be that one of your failure groups has less
space than the others? You can verify this with the output of mmdf
and look at the NSD sizes and space available.</font><font size=3><br></font><font size=3 face="sans-serif"><br>Fred<br>__________________________________________________<br>Fred Stock | IBM Pittsburgh Lab | </font><a href="tel:(720)%20430-8821" target=_blank><font size=3 color=blue face="sans-serif"><u>720-430-8821</u></font></a><font size=3 color=blue face="sans-serif"><u><br></u></font><a href=mailto:stockf@us.ibm.com target=_blank><font size=3 color=blue face="sans-serif"><u>stockf@us.ibm.com</u></font></a><font size=3><br><br><br></font><font size=1 color=#5f5f5f face="sans-serif"><br>From: </font><font size=1 face="sans-serif">John
Hanks <</font><a href=mailto:griznog@gmail.com target=_blank><font size=1 color=blue face="sans-serif"><u>griznog@gmail.com</u></font></a><font size=1 face="sans-serif">></font><font size=1 color=#5f5f5f face="sans-serif"><br>To: </font><font size=1 face="sans-serif">gpfsug
main discussion list <</font><a href="mailto:gpfsug-discuss@spectrumscale.org" target=_blank><font size=1 color=blue face="sans-serif"><u>gpfsug-discuss@spectrumscale.org</u></font></a><font size=1 face="sans-serif">></font><font size=1 color=#5f5f5f face="sans-serif"><br>Date: </font><font size=1 face="sans-serif">11/02/2017
12:20 PM</font><font size=1 color=#5f5f5f face="sans-serif"><br>Subject: </font><font size=1 face="sans-serif">Re:
[gpfsug-discuss] mmrestripefs "No space left on device"</font><font size=1 color=#5f5f5f face="sans-serif"><br>Sent by: </font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target=_blank><font size=1 color=blue face="sans-serif"><u>gpfsug-discuss-bounces@spectrumscale.org</u></font></a><font size=3><br></font><hr noshade><font size=3><br><br><br>Addendum to last message:<br><br>We haven't upgraded recently as far as I know (I just inherited this a
couple of months ago.) but am planning an outage soon to upgrade from 4.2.0-4
to 4.2.3-5. <br><br>My growing collection of output files generally contain something like<br><br>This inode list was generated in the Parallel Inode Traverse on Thu Nov
2 08:34:22 2017<br>INODE_NUMBER DUMMY_INFO SNAPSHOT_ID ISGLOBAL_SNAPSHOT INDEPENDENT_FSETID
MEMO(INODE_FLAGS FILE_TYPE [ERROR])<br> 53506 0:0 0
1
0
illreplicated REGULAR_FILE RESERVED Error: 28 No space left on device<br><br>With that inode varying slightly.<br><br>jbh<br><br>On Thu, Nov 2, 2017 at 8:55 AM, Scott Fadden <</font><a href=mailto:sfadden@us.ibm.com target=_blank><font size=3 color=blue><u>sfadden@us.ibm.com</u></font></a><font size=3>>
wrote:</font><br><font size=2 face="Arial">Sorry just reread as I hit send and
saw this was mmrestripe, in my case it was mmdeledisk.<br> <br>Did you try running the command on just one pool. Or using -B instead?<br> <br>What is the file it is complaining about in "/var/mmfs/tmp/gsfs0.pit.interestingInodes.12888779711"
?<br> <br>Looks like it could be related to the maxfeaturelevel of the cluster. Have
you recently upgraded? Is everything up to the same level? <br> <br>Scott Fadden<br>Spectrum Scale - Technical Marketing<br>Phone: </font><a href="tel:(503)%20880-5833" target=_blank><font size=2 color=blue face="Arial"><u>(503)
880-5833</u></font></a><font size=3 color=blue><u><br></u></font><a href=mailto:sfadden@us.ibm.com target=_blank><font size=2 color=blue face="Arial"><u>sfadden@us.ibm.com</u></font></a><font size=3 color=blue><u><br></u></font><a href=http://www.ibm.com/systems/storage/spectrum/scale target=_blank><font size=2 color=blue face="Arial"><u>http://www.ibm.com/systems/storage/spectrum/scale</u></font></a><font size=2 face="Arial"><br> <br> <br>----- Original message -----<br>From: Scott Fadden/Portland/IBM<br>To: </font><a href="mailto:gpfsug-discuss@spectrumscale.org" target=_blank><font size=2 color=blue face="Arial"><u>gpfsug-discuss@spectrumscale.org</u></font></a><font size=2 face="Arial"><br>Cc: </font><a href="mailto:gpfsug-discuss@spectrumscale.org" target=_blank><font size=2 color=blue face="Arial"><u>gpfsug-discuss@spectrumscale.org</u></font></a><font size=2 face="Arial"><br>Subject: Re: [gpfsug-discuss] mmrestripefs "No space left on device"<br>Date: Thu, Nov 2, 2017 8:44 AM<br> <br>I opened a defect on this the other day, in my case it was an incorrect
error message. What it meant to say was,"The pool is not empty."
Are you trying to remove the last disk in a pool? If so did you empty the
pool with a MIGRATE policy first? <br> <br> <br>Scott Fadden<br>Spectrum Scale - Technical Marketing<br>Phone: </font><a href="tel:(503)%20880-5833" target=_blank><font size=2 color=blue face="Arial"><u>(503)
880-5833</u></font></a><font size=3 color=blue><u><br></u></font><a href=mailto:sfadden@us.ibm.com target=_blank><font size=2 color=blue face="Arial"><u>sfadden@us.ibm.com</u></font></a><font size=3 color=blue><u><br></u></font><a href=http://www.ibm.com/systems/storage/spectrum/scale target=_blank><font size=2 color=blue face="Arial"><u>http://www.ibm.com/systems/storage/spectrum/scale</u></font></a><font size=2 face="Arial"><br> <br> <br>----- Original message -----<br>From: John Hanks <</font><a href=mailto:griznog@gmail.com target=_blank><font size=2 color=blue face="Arial"><u>griznog@gmail.com</u></font></a><font size=2 face="Arial">><br>Sent by: </font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target=_blank><font size=2 color=blue face="Arial"><u>gpfsug-discuss-bounces@spectrumscale.org</u></font></a><font size=2 face="Arial"><br>To: gpfsug main discussion list <</font><a href="mailto:gpfsug-discuss@spectrumscale.org" target=_blank><font size=2 color=blue face="Arial"><u>gpfsug-discuss@spectrumscale.org</u></font></a><font size=2 face="Arial">><br>Cc:<br>Subject: Re: [gpfsug-discuss] mmrestripefs "No space left on device"<br>Date: Thu, Nov 2, 2017 8:34 AM<br> <br>We have no snapshots ( they were the first to go when we initially hit
the full metadata NSDs). <br> <br>I've increased quotas so that no filesets have hit a space quota. <br> <br>Verified that there are no inode quotas anywhere.<br> <br>mmdf shows the least amount of free space on any nsd to be 9% free.<br> <br>Still getting this error:<br> <br>[root@scg-gs0 ~]# mmrestripefs gsfs0 -r -N scg-gs0,scg-gs1,scg-gs2,scg-gs3<br>Scanning file system metadata, phase 1 ... <br>Scan completed successfully.<br>Scanning file system metadata, phase 2 ... <br>Scanning file system metadata for sas0 storage pool<br>Scanning file system metadata for sata0 storage pool<br>Scan completed successfully.<br>Scanning file system metadata, phase 3 ... <br>Scan completed successfully.<br>Scanning file system metadata, phase 4 ... <br>Scan completed successfully.<br>Scanning user file metadata ...<br>Error processing user file metadata.<br>No space left on device<br>Check file '/var/mmfs/tmp/gsfs0.pit.interestingInodes.12888779711' on scg-gs0
for inodes with broken disk addresses or failures.<br>mmrestripefs: Command failed. Examine previous error messages to determine
cause.<br> <br>I should note too that this fails almost immediately, far to quickly to
fill up any location it could be trying to write to.<br> <br>jbh<br> <br>On Thu, Nov 2, 2017 at 7:57 AM, David Johnson <</font><a href=mailto:david_johnson@brown.edu target=_blank><font size=2 color=blue face="Arial"><u>david_johnson@brown.edu</u></font></a><font size=2 face="Arial">>
wrote: <br>One thing that may be relevant is if you have snapshots, depending on your
release level, <br>inodes in the snapshot may considered immutable, and will not be migrated.
Once the snapshots<br>have been deleted, the inodes are freed up and you won’t see the (somewhat
misleading) message<br>about no space.<br> <br> — ddj<br>Dave Johnson<br>Brown University<br> <br>On Nov 2, 2017, at 10:43 AM, John Hanks <</font><a href=mailto:griznog@gmail.com target=_blank><font size=2 color=blue face="Arial"><u>griznog@gmail.com</u></font></a><font size=2 face="Arial">>
wrote:<br>Thanks all for the suggestions. <br> <br>Having our metadata NSDs fill up was what prompted this exercise, but space
was previously feed up on those by switching them from metadata+data to
metadataOnly and using a policy to migrate files out of that pool. So these
now have about 30% free space (more if you include fragmented space). The
restripe attempt is just to make a final move of any remaining data off
those devices. All the NSDs now have free space on them.<br> <br>df -i shows inode usage at about 84%, so plenty of free inodes for the
filesystem as a whole.<br> <br>We did have old .quota files laying around but removing them didn't
have any impact. <br> <br>mmlsfileset fs -L -i is taking a while to complete, I'll let it simmer
while getting to work.<br> <br>mmrepquota does show about a half-dozen filesets that have hit their quota
for space (we don't set quotas on inodes). Once I'm settled in this morning
I'll try giving them a little extra space and see what happens.<br> <br>jbh<br> <br> <br>On Thu, Nov 2, 2017 at 4:19 AM, Oesterlin, Robert <</font><a href=mailto:Robert.Oesterlin@nuance.com target=_blank><font size=2 color=blue face="Arial"><u>Robert.Oesterlin@nuance.com</u></font></a><font size=2 face="Arial">>
wrote: <br>One thing that I’ve run into before is that on older file systems you
had the “*.quota” files in the file system root. If you upgraded the
file system to a newer version (so these files aren’t used) - There was
a bug at one time where these didn’t get properly migrated during a restripe.
Solution was to just remove them</font><p><font size=2 face="Arial"> </font><p><font size=2 face="Arial"> </font><p><font size=3 face="Arial">Bob Oesterlin</font><p><font size=3 face="Arial">Sr Principal Storage Engineer, Nuance</font><p><font size=2 face="Arial"> </font><p><font size=3 face="Arial"><b>From: </b><</font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target=_blank><font size=3 color=blue face="Arial"><u>gpfsug-discuss-bounces@spectrumscale.org</u></font></a><font size=3 face="Arial">>
on behalf of John Hanks <</font><a href=mailto:griznog@gmail.com target=_blank><font size=3 color=blue face="Arial"><u>griznog@gmail.com</u></font></a><font size=3 face="Arial">><b><br>Reply-To: </b>gpfsug main discussion list <</font><a href="mailto:gpfsug-discuss@spectrumscale.org" target=_blank><font size=3 color=blue face="Arial"><u>gpfsug-discuss@spectrumscale.org</u></font></a><font size=3 face="Arial">><b><br>Date: </b>Wednesday, November 1, 2017 at 5:55 PM<b><br>To: </b>gpfsug <</font><a href="mailto:gpfsug-discuss@spectrumscale.org" target=_blank><font size=3 color=blue face="Arial"><u>gpfsug-discuss@spectrumscale.org</u></font></a><font size=3 face="Arial">><b><br>Subject: </b>[EXTERNAL] [gpfsug-discuss] mmrestripefs "No space left
on device"</font><p><font size=2 face="Arial"> </font><p><a name="m_-4512031540600214981_m_-7946021508789305345_m_7792501056088615300_m_6737186114830475911__MailOriginalBody"></a><font size=2 face="Arial">Hi
all,</font><p><font size=2 face="Arial"> </font><p><font size=2 face="Arial">I'm trying to do a restripe after setting
some nsds to metadataOnly and I keep running into this error:</font><p><font size=2 face="Arial"> </font><p><font size=2 face="Arial">Scanning user file metadata ...</font><p><font size=2 face="Arial"> 0.01 % complete on Wed Nov
1 15:36:01 2017 ( 40960 inodes with total
531689 MB data processed)</font><p><font size=2 face="Arial">Error processing user file metadata. </font><p><font size=2 face="Arial">Check file '/var/mmfs/tmp/gsfs0.pit.interestingInodes.12888779708'
on scg-gs0 for inodes with broken disk addresses or failures.</font><p><font size=2 face="Arial">mmrestripefs: Command failed. Examine previous
error messages to determine cause.</font><p><font size=2 face="Arial"> </font><p><font size=2 face="Arial">The file it points to says:</font><p><font size=2 face="Arial"> </font><p><font size=2 face="Arial">This inode list was generated in the Parallel
Inode Traverse on Wed Nov 1 15:36:06 2017</font><p><font size=2 face="Arial">INODE_NUMBER DUMMY_INFO SNAPSHOT_ID ISGLOBAL_SNAPSHOT
INDEPENDENT_FSETID MEMO(INODE_FLAGS FILE_TYPE [ERROR])</font><p><font size=2 face="Arial"> 53504 0:0
0 1
0
illreplicated REGULAR_FILE RESERVED
Error: 28 No space left on device</font><p><font size=2 face="Arial"> </font><p><font size=2 face="Arial"> </font><p><font size=2 face="Arial">/var on the node I am running this on has
> 128 GB free, all the NSDs have plenty of free space, the filesystem
being restriped has plenty of free space and if I watch the node while
running this no filesystem on it even starts to get full. Could someone
tell me where mmrestripefs is attempting to write and/or how to point it
at a different location?</font><p><font size=2 face="Arial"> </font><p><font size=2 face="Arial">Thanks,</font><p><font size=2 face="Arial"> </font><p><font size=2 face="Arial">jbh</font><p><font size=2 face="Arial">_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=WDtkF9zLTGGYqFnVnJ3rywZM6KHROA4FpMYi6cUkkKY&m=hKtOnoUDijNQoFnSlxQfek9m6h2qKbqjcCswbjHg2-E&s=n5P1NWESV2GUb3EXICXGj62_QDAPfSAWVPz_i59CNKk&e=" target=_blank><font size=2 color=blue face="Arial"><u>spectrumscale.org</u></font></a><font size=3 color=blue><u><br></u></font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=uLFESUsuxpmf07haYD3Sl-DpeYkm3t_r0WVV2AZ9Jk0&s=zARWNuUgVecPk0qJwJdRIi0l_U9K7Z-xnnr5vNm1IZo&e=" target=_blank><font size=2 color=blue face="Arial"><u>http://gpfsug.org/mailman</u></font></a><font size=2 face="Arial">/listinfo/gpfsug-discuss<br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=WDtkF9zLTGGYqFnVnJ3rywZM6KHROA4FpMYi6cUkkKY&m=hKtOnoUDijNQoFnSlxQfek9m6h2qKbqjcCswbjHg2-E&s=n5P1NWESV2GUb3EXICXGj62_QDAPfSAWVPz_i59CNKk&e=" target=_blank><font size=2 color=blue face="Arial"><u>spectrumscale.org</u></font></a><font size=3 color=blue><u><br></u></font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=uLFESUsuxpmf07haYD3Sl-DpeYkm3t_r0WVV2AZ9Jk0&s=zARWNuUgVecPk0qJwJdRIi0l_U9K7Z-xnnr5vNm1IZo&e=" target=_blank><font size=2 color=blue face="Arial"><u>http://gpfsug.org/mailman</u></font></a><font size=2 face="Arial">/listinfo/gpfsug-discuss<br> </font><tt><font size=2><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font></tt><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=uLFESUsuxpmf07haYD3Sl-DpeYkm3t_r0WVV2AZ9Jk0&s=DGJeAf81dkJPqeCYJhjPiOUDTCAVRO-KEsvBx-HSzUM&e=" target=_blank><tt><font size=2 color=blue><u>spectrumscale.org</u></font></tt></a><font size=3 color=blue><u><br></u></font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=WDtkF9zLTGGYqFnVnJ3rywZM6KHROA4FpMYi6cUkkKY&m=hKtOnoUDijNQoFnSlxQfek9m6h2qKbqjcCswbjHg2-E&s=j7eYU1VnwYXrTnflbJki13EfnMjqAro0RdCiLkVrgzE&e=" target=_blank><tt><font size=2 color=blue><u>https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=WDtkF9zLTGGYqFnVnJ3rywZM6KHROA4FpMYi6cUkkKY&m=hKtOnoUDijNQoFnSlxQfek9m6h2qKbqjcCswbjHg2-E&s=j7eYU1VnwYXrTnflbJki13EfnMjqAro0RdCiLkVrgzE&e=</u></font></tt></a><font size=2 face="Arial"><br> <br> </font><font size=3><br><br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=uLFESUsuxpmf07haYD3Sl-DpeYkm3t_r0WVV2AZ9Jk0&s=DGJeAf81dkJPqeCYJhjPiOUDTCAVRO-KEsvBx-HSzUM&e=" target=_blank><font size=3 color=blue><u>spectrumscale.org</u></font></a><font size=3 color=blue><u><br></u></font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=uLFESUsuxpmf07haYD3Sl-DpeYkm3t_r0WVV2AZ9Jk0&s=RGgSZEisfDpxsKl3PFUWh6DtzD_FF6spqHVpo_0joLY&e=" target=_blank><font size=3 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></a><font size=3><br></font><tt><font size=2><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font></tt><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=XPw1EyoosGN5bt3yLIT1JbUJ73B6iWH2gBaDJ2xHW8M&s=WvredVor59NfZe-GxK5qa27t7_OT-zg1uOs__CSYmJM&e=" target=_blank><tt><font size=2 color=blue><u>spectrumscale.org</u></font></tt></a><br><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=uLFESUsuxpmf07haYD3Sl-DpeYkm3t_r0WVV2AZ9Jk0&s=RGgSZEisfDpxsKl3PFUWh6DtzD_FF6spqHVpo_0joLY&e=" target=_blank><tt><font size=2 color=blue><u>https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=uLFESUsuxpmf07haYD3Sl-DpeYkm3t_r0WVV2AZ9Jk0&s=RGgSZEisfDpxsKl3PFUWh6DtzD_FF6spqHVpo_0joLY&e=</u></font></tt></a><font size=3><br><br><br></font><br><font size=3><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=XPw1EyoosGN5bt3yLIT1JbUJ73B6iWH2gBaDJ2xHW8M&s=WvredVor59NfZe-GxK5qa27t7_OT-zg1uOs__CSYmJM&e=" target=_blank><font size=3 color=blue><u>spectrumscale.org</u></font></a><font size=3 color=blue><u><br></u></font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=XPw1EyoosGN5bt3yLIT1JbUJ73B6iWH2gBaDJ2xHW8M&s=yDRpuvz3LOTwvP2pkIJEU7NWUxwMOcYHyXBRoWCPF-s&e=" target=_blank><font size=3 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></a><font size=3><br></font><br><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=XPw1EyoosGN5bt3yLIT1JbUJ73B6iWH2gBaDJ2xHW8M&s=yDRpuvz3LOTwvP2pkIJEU7NWUxwMOcYHyXBRoWCPF-s&e="><tt><font size=2>https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=XPw1EyoosGN5bt3yLIT1JbUJ73B6iWH2gBaDJ2xHW8M&s=yDRpuvz3LOTwvP2pkIJEU7NWUxwMOcYHyXBRoWCPF-s&e=</font></tt></a><tt><font size=2><br></font></tt><br><br><BR>