<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">One thing that may be relevant is if you have snapshots, depending on your release level,<div class="">inodes in the snapshot may considered immutable, and will not be migrated. Once the snapshots</div><div class="">have been deleted, the inodes are freed up and you won’t see the (somewhat misleading) message</div><div class="">about no space.</div><div class=""><br class=""></div><div class=""> — ddj</div><div class="">Dave Johnson</div><div class="">Brown University</div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Nov 2, 2017, at 10:43 AM, John Hanks <<a href="mailto:griznog@gmail.com" class="">griznog@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Thanks all for the suggestions. <div class=""><br class=""></div><div class="">Having our metadata NSDs fill up was what prompted this exercise, but space was previously feed up on those by switching them from metadata+data to metadataOnly and using a policy to migrate files out of that pool. So these now have about 30% free space (more if you include fragmented space). The restripe attempt is just to make a final move of any remaining data off those devices. All the NSDs now have free space on them.</div><div class=""><br class=""></div><div class="">df -i shows inode usage at about 84%, so plenty of free inodes for the filesystem as a whole.</div><div class=""><br class=""></div><div class="">We did have old .quota files laying around but removing them didn't have any impact. </div><div class=""><br class=""></div><div class="">mmlsfileset fs -L -i is taking a while to complete, I'll let it simmer while getting to work.</div><div class=""><br class=""></div><div class="">mmrepquota does show about a half-dozen filesets that have hit their quota for space (we don't set quotas on inodes). Once I'm settled in this morning I'll try giving them a little extra space and see what happens.</div><div class=""><br class=""></div><div class="">jbh</div><div class=""><br class=""></div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Thu, Nov 2, 2017 at 4:19 AM, Oesterlin, Robert <span dir="ltr" class=""><<a href="mailto:Robert.Oesterlin@nuance.com" target="_blank" class="">Robert.Oesterlin@nuance.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="#0563C1" vlink="#954F72" class="">
<div class="m_6737186114830475911WordSection1"><p class="MsoNormal">One thing that I’ve run into before is that on older file systems you had the “*.quota” files in the file system root. If you upgraded the file system to a newer version (so these files aren’t used) - There was a bug at one time where these
didn’t get properly migrated during a restripe. Solution was to just remove them<u class=""></u><u class=""></u></p><p class="MsoNormal"><u class=""></u> <u class=""></u></p>
<div class=""><p class="MsoNormal"><u class=""></u> <u class=""></u></p><p class="MsoNormal"><span style="font-size:12.0pt" class="">Bob Oesterlin<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span style="font-size:12.0pt" class="">Sr Principal Storage Engineer, Nuance<u class=""></u><u class=""></u></span></p>
</div><p class="MsoNormal"><u class=""></u> <u class=""></u></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in" class=""><p class="MsoNormal"><b class=""><span style="font-size: 12pt;" class="">From: </span></b><span style="font-size: 12pt;" class=""><<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank" class="">gpfsug-discuss-bounces@<wbr class="">spectrumscale.org</a>> on behalf of John Hanks <<a href="mailto:griznog@gmail.com" target="_blank" class="">griznog@gmail.com</a>><br class="">
<b class="">Reply-To: </b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank" class="">gpfsug-discuss@spectrumscale.<wbr class="">org</a>><br class="">
<b class="">Date: </b>Wednesday, November 1, 2017 at 5:55 PM<br class="">
<b class="">To: </b>gpfsug <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank" class="">gpfsug-discuss@spectrumscale.<wbr class="">org</a>><br class="">
<b class="">Subject: </b>[EXTERNAL] [gpfsug-discuss] mmrestripefs "No space left on device"<u class=""></u><u class=""></u></span></p>
</div><span class="">
<div class=""><p class="MsoNormal"><u class=""></u> <u class=""></u></p>
</div>
<div class="">
<div class=""><p class="MsoNormal"><a name="m_6737186114830475911__MailOriginalBody" class="">Hi all,<u class=""></u><u class=""></u></a></p>
</div>
<div class=""><p class="MsoNormal"><span class=""><u class=""></u> <u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span class="">I'm trying to do a restripe after setting some nsds to metadataOnly and I keep running into this error:<u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span class=""><u class=""></u> <u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span class="">Scanning user file metadata ...<u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span class=""> 0.01 % complete on Wed Nov 1 15:36:01 2017 ( 40960 inodes with total 531689 MB data processed)<u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span class="">Error processing user file metadata. <u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span class="">Check file '/var/mmfs/tmp/gsfs0.pit.<wbr class="">interestingInodes.12888779708' on scg-gs0 for inodes with broken disk addresses or failures.<u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span class="">mmrestripefs: Command failed. Examine previous error messages to determine cause.<u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span class=""><u class=""></u> <u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span class="">The file it points to says:<u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span class=""><u class=""></u> <u class=""></u></span></p>
</div>
<div class="">
<div class=""><p class="MsoNormal"><span class="">This inode list was generated in the Parallel Inode Traverse on Wed Nov 1 15:36:06 2017<u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span class="">INODE_NUMBER DUMMY_INFO SNAPSHOT_ID ISGLOBAL_SNAPSHOT INDEPENDENT_FSETID MEMO(INODE_FLAGS FILE_TYPE [ERROR])<u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span class=""> 53504 0:0 0 1 0 illreplicated REGULAR_FILE RESERVED Error: 28 No space left on device<u class=""></u><u class=""></u></span></p>
</div>
</div>
<div class=""><p class="MsoNormal"><span class=""><u class=""></u> <u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span class=""><u class=""></u> <u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span class="">/var on the node I am running this on has > 128 GB free, all the NSDs have plenty of free space, the filesystem being restriped has plenty of free space and if I watch the node while running
this no filesystem on it even starts to get full. Could someone tell me where mmrestripefs is attempting to write and/or how to point it at a different location?<u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span class=""><u class=""></u> <u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span class="">Thanks,<u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span class=""><u class=""></u> <u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span class="">jbh<u class=""></u><u class=""></u></span></p>
</div>
</div>
</span></div>
</div>
</blockquote></div><br class=""></div>
_______________________________________________<br class="">gpfsug-discuss mailing list<br class="">gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a><br class=""><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class=""></div></blockquote></div><br class=""></div></body></html>