<div dir="ltr">We have no snapshots ( they were the first to go when we initially hit the full metadata NSDs). <div><br></div><div>I've increased quotas so that no filesets have hit a space quota. </div><div><br></div><div>Verified that there are no inode quotas anywhere.</div><div><br></div><div>mmdf shows the least amount of free space on any nsd to be 9% free.</div><div><br></div><div>Still getting this error:</div><div><br></div><div><div>[root@scg-gs0 ~]# mmrestripefs gsfs0 -r -N scg-gs0,scg-gs1,scg-gs2,scg-gs3</div><div>Scanning file system metadata, phase 1 ... </div><div>Scan completed successfully.</div><div>Scanning file system metadata, phase 2 ... </div><div>Scanning file system metadata for sas0 storage pool</div><div>Scanning file system metadata for sata0 storage pool</div><div>Scan completed successfully.</div><div>Scanning file system metadata, phase 3 ... </div><div>Scan completed successfully.</div><div>Scanning file system metadata, phase 4 ... </div><div>Scan completed successfully.</div><div>Scanning user file metadata ...</div><div>Error processing user file metadata.</div><div>No space left on device</div><div>Check file '/var/mmfs/tmp/gsfs0.pit.interestingInodes.12888779711' on scg-gs0 for inodes with broken disk addresses or failures.</div><div>mmrestripefs: Command failed. Examine previous error messages to determine cause.</div></div><div><br></div><div>I should note too that this fails almost immediately, far to quickly to fill up any location it could be trying to write to.</div><div><br></div><div>jbh</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Nov 2, 2017 at 7:57 AM, David Johnson <span dir="ltr"><<a href="mailto:david_johnson@brown.edu" target="_blank">david_johnson@brown.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">One thing that may be relevant is if you have snapshots, depending on your release level,<div>inodes in the snapshot may considered immutable, and will not be migrated. Once the snapshots</div><div>have been deleted, the inodes are freed up and you won’t see the (somewhat misleading) message</div><div>about no space.</div><div><br></div><div> — ddj</div><div>Dave Johnson</div><div>Brown University</div><div><br><div><blockquote type="cite"><div><div class="h5"><div>On Nov 2, 2017, at 10:43 AM, John Hanks <<a href="mailto:griznog@gmail.com" target="_blank">griznog@gmail.com</a>> wrote:</div><br class="m_7792501056088615300Apple-interchange-newline"></div></div><div><div><div class="h5"><div dir="ltr">Thanks all for the suggestions. <div><br></div><div>Having our metadata NSDs fill up was what prompted this exercise, but space was previously feed up on those by switching them from metadata+data to metadataOnly and using a policy to migrate files out of that pool. So these now have about 30% free space (more if you include fragmented space). The restripe attempt is just to make a final move of any remaining data off those devices. All the NSDs now have free space on them.</div><div><br></div><div>df -i shows inode usage at about 84%, so plenty of free inodes for the filesystem as a whole.</div><div><br></div><div>We did have old .quota files laying around but removing them didn't have any impact. </div><div><br></div><div>mmlsfileset fs -L -i is taking a while to complete, I'll let it simmer while getting to work.</div><div><br></div><div>mmrepquota does show about a half-dozen filesets that have hit their quota for space (we don't set quotas on inodes). Once I'm settled in this morning I'll try giving them a little extra space and see what happens.</div><div><br></div><div>jbh</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Nov 2, 2017 at 4:19 AM, Oesterlin, Robert <span dir="ltr"><<a href="mailto:Robert.Oesterlin@nuance.com" target="_blank">Robert.Oesterlin@nuance.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="m_7792501056088615300m_6737186114830475911WordSection1"><p class="MsoNormal">One thing that I’ve run into before is that on older file systems you had the “*.quota” files in the file system root. If you upgraded the file system to a newer version (so these files aren’t used) - There was a bug at one time where these
didn’t get properly migrated during a restripe. Solution was to just remove them<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p>
<div><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal"><span style="font-size:12.0pt">Bob Oesterlin<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:12.0pt">Sr Principal Storage Engineer, Nuance<u></u><u></u></span></p>
</div><p class="MsoNormal"><u></u> <u></u></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in"><p class="MsoNormal"><b><span style="font-size:12pt">From: </span></b><span style="font-size:12pt"><<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectr<wbr>umscale.org</a>> on behalf of John Hanks <<a href="mailto:griznog@gmail.com" target="_blank">griznog@gmail.com</a>><br>
<b>Reply-To: </b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.<wbr>org</a>><br>
<b>Date: </b>Wednesday, November 1, 2017 at 5:55 PM<br>
<b>To: </b>gpfsug <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.<wbr>org</a>><br>
<b>Subject: </b>[EXTERNAL] [gpfsug-discuss] mmrestripefs "No space left on device"<u></u><u></u></span></p>
</div><span>
<div><p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div><p class="MsoNormal"><a name="m_7792501056088615300_m_6737186114830475911__MailOriginalBody">Hi all,<u></u><u></u></a></p>
</div>
<div><p class="MsoNormal"><span><u></u> <u></u></span></p>
</div>
<div><p class="MsoNormal"><span>I'm trying to do a restripe after setting some nsds to metadataOnly and I keep running into this error:<u></u><u></u></span></p>
</div>
<div><p class="MsoNormal"><span><u></u> <u></u></span></p>
</div>
<div><p class="MsoNormal"><span>Scanning user file metadata ...<u></u><u></u></span></p>
</div>
<div><p class="MsoNormal"><span> 0.01 % complete on Wed Nov 1 15:36:01 2017 ( 40960 inodes with total 531689 MB data processed)<u></u><u></u></span></p>
</div>
<div><p class="MsoNormal"><span>Error processing user file metadata. <u></u><u></u></span></p>
</div>
<div><p class="MsoNormal"><span>Check file '/var/mmfs/tmp/gsfs0.pit.inter<wbr>estingInodes.12888779708' on scg-gs0 for inodes with broken disk addresses or failures.<u></u><u></u></span></p>
</div>
<div><p class="MsoNormal"><span>mmrestripefs: Command failed. Examine previous error messages to determine cause.<u></u><u></u></span></p>
</div>
<div><p class="MsoNormal"><span><u></u> <u></u></span></p>
</div>
<div><p class="MsoNormal"><span>The file it points to says:<u></u><u></u></span></p>
</div>
<div><p class="MsoNormal"><span><u></u> <u></u></span></p>
</div>
<div>
<div><p class="MsoNormal"><span>This inode list was generated in the Parallel Inode Traverse on Wed Nov 1 15:36:06 2017<u></u><u></u></span></p>
</div>
<div><p class="MsoNormal"><span>INODE_NUMBER DUMMY_INFO SNAPSHOT_ID ISGLOBAL_SNAPSHOT INDEPENDENT_FSETID MEMO(INODE_FLAGS FILE_TYPE [ERROR])<u></u><u></u></span></p>
</div>
<div><p class="MsoNormal"><span> 53504 0:0 0 1 0 illreplicated REGULAR_FILE RESERVED Error: 28 No space left on device<u></u><u></u></span></p>
</div>
</div>
<div><p class="MsoNormal"><span><u></u> <u></u></span></p>
</div>
<div><p class="MsoNormal"><span><u></u> <u></u></span></p>
</div>
<div><p class="MsoNormal"><span>/var on the node I am running this on has > 128 GB free, all the NSDs have plenty of free space, the filesystem being restriped has plenty of free space and if I watch the node while running
this no filesystem on it even starts to get full. Could someone tell me where mmrestripefs is attempting to write and/or how to point it at a different location?<u></u><u></u></span></p>
</div>
<div><p class="MsoNormal"><span><u></u> <u></u></span></p>
</div>
<div><p class="MsoNormal"><span>Thanks,<u></u><u></u></span></p>
</div>
<div><p class="MsoNormal"><span><u></u> <u></u></span></p>
</div>
<div><p class="MsoNormal"><span>jbh<u></u><u></u></span></p>
</div>
</div>
</span></div>
</div>
</blockquote></div><br></div></div></div><span class="">
______________________________<wbr>_________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br></span><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</a><br></div></blockquote></div><br></div></div><br>______________________________<wbr>_________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</a><br>
<br></blockquote></div><br></div>