[gpfsug-discuss] mmrestripefs "No space left on device"

[John Hanks]

Thanks all for the suggestions.

Having our metadata NSDs fill up was what prompted this exercise, but space
was previously feed up on those by switching them from metadata+data to
metadataOnly and using a policy to migrate files out of that pool. So these
now have about 30% free space (more if you include fragmented space). The
restripe attempt is just to make a final move of any remaining data off
those devices. All the NSDs now have free space on them.

df -i shows inode usage at about 84%, so plenty of free inodes for the
filesystem as a whole.

We did have old  .quota files laying around but removing them didn't have
any impact.

mmlsfileset fs -L -i is taking a while to complete, I'll let it simmer
while getting to work.

mmrepquota does show about a half-dozen filesets that have hit their quota
for space (we don't set quotas on inodes). Once I'm settled in this morning
I'll try giving them a little extra space and see what happens.

jbh


On Thu, Nov 2, 2017 at 4:19 AM, Oesterlin, Robert <
Robert.Oesterlin at nuance.com> wrote:

> One thing that I’ve run into before is that on older file systems you had
> the “*.quota” files in the file system root. If you upgraded the file
> system to a newer version (so these files aren’t used) - There was a bug at
> one time where these didn’t get properly migrated during a restripe.
> Solution was to just remove them
>
>
>
>
>
> Bob Oesterlin
>
> Sr Principal Storage Engineer, Nuance
>
>
>
> *From: *<gpfsug-discuss-bounces at spectrumscale.org> on behalf of John
> Hanks <griznog at gmail.com>
> *Reply-To: *gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> *Date: *Wednesday, November 1, 2017 at 5:55 PM
> *To: *gpfsug <gpfsug-discuss at spectrumscale.org>
> *Subject: *[EXTERNAL] [gpfsug-discuss] mmrestripefs "No space left on
> device"
>
>
>
> Hi all,
>
>
>
> I'm trying to do a restripe after setting some nsds to metadataOnly and I
> keep running into this error:
>
>
>
> Scanning user file metadata ...
>
>    0.01 % complete on Wed Nov  1 15:36:01 2017  (     40960 inodes with
> total     531689 MB data processed)
>
> Error processing user file metadata.
>
> Check file '/var/mmfs/tmp/gsfs0.pit.interestingInodes.12888779708' on
> scg-gs0 for inodes with broken disk addresses or failures.
>
> mmrestripefs: Command failed. Examine previous error messages to determine
> cause.
>
>
>
> The file it points to says:
>
>
>
> This inode list was generated in the Parallel Inode Traverse on Wed Nov  1
> 15:36:06 2017
>
> INODE_NUMBER DUMMY_INFO SNAPSHOT_ID ISGLOBAL_SNAPSHOT INDEPENDENT_FSETID
> MEMO(INODE_FLAGS FILE_TYPE [ERROR])
>
>  53504        0:0        0           1                 0
> illreplicated REGULAR_FILE RESERVED Error: 28 No space left on device
>
>
>
>
>
> /var on the node I am running this on has > 128 GB free, all the NSDs have
> plenty of free space, the filesystem being restriped has plenty of free
> space and if I watch the node while running this no filesystem on it even
> starts to get full. Could someone tell me where mmrestripefs is attempting
> to write and/or how to point it at a different location?
>
>
>
> Thanks,
>
>
>
> jbh
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171102/7410038c/attachment-0002.htm>