<font size=2 face="sans-serif">mmdf statistics are not real-time accurate,
there is a trade off in accuracy vs the cost of polling each node that
might have the file system mounted.</font><br><br><font size=2 face="sans-serif">That said, here are some possibilities,
in increasing order of impact on users and your possible desperation ;-)</font><br><br><font size=2 face="sans-serif">A1. Wait a while (at most a few minutes)
and see if the mmdf stats are updated.</font><br><br><font size=2 face="sans-serif">A2. </font><br><font size=2 face="sans-serif">mmchmgr fs another-node </font><br><br><font size=2 face="sans-serif">may force new stats to be sent to the
new fs manager. (Not sure but I expect it will.)</font><br><font size=2 face="sans-serif"> </font><br><font size=2 face="sans-serif">B. Briefly quiesce the file system with:
</font><br><br><font size=2 face="sans-serif">mmfsctl fs suspend; mmfsctl fs resume;
</font><br><br><br><font size=2 face="sans-serif">C. If you have no users active ... I'm
pretty sure </font><br><br><font size=2 face="sans-serif">mmumount fs -a ; mmmount fs -a; </font><br><br><font size=2 face="sans-serif">will clear the problem ... but
there's always</font><br><br><font size=2 face="sans-serif">D. </font><br><font size=2 face="sans-serif">mmshutdown -a ; mmstartup -a </font><br><br><font size=2 face="sans-serif">E.</font><br><font size=2 face="sans-serif">If none of those resolve the situation
something is hosed -- </font><br><br><font size=2 face="sans-serif">F.</font><br><font size=2 face="sans-serif">hope that </font><br><br><font size=2 face="sans-serif">mmfsck </font><br><br><font size=2 face="sans-serif">can fix it.<br></font><br><br><font size=2 face="sans-serif">--marc</font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Luke Raimbach <Luke.Raimbach@crick.ac.uk></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug main discussion
list <gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">07/03/2016 10:55 AM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
Trapped Inodes</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><font size=2 color=#004080 face="Calibri">Hi Marc,</font><br><font size=2 color=#004080 face="Calibri"> </font><br><font size=2 color=#004080 face="Calibri">Thanks for that suggestions.
This seems to have removed the NULL fileset from the list, however mmdf
now shows even more strange statistics:</font><br><font size=2 color=#004080 face="Calibri"> </font><br><font size=2 color=#004080 face="Calibri">Inode Information</font><br><font size=2 color=#004080 face="Calibri">-----------------</font><br><font size=2 color=#004080 face="Calibri">Total number of used inodes
in all Inode spaces: -103900000</font><br><font size=2 color=#004080 face="Calibri">Total number of free inodes
in all Inode spaces: -24797856</font><br><font size=2 color=#004080 face="Calibri">Total number of allocated
inodes in all Inode spaces: -128697856</font><br><font size=2 color=#004080 face="Calibri">Total of Maximum number of
inodes in all Inode spaces: -103900000</font><br><font size=2 color=#004080 face="Calibri"> </font><br><font size=2 color=#004080 face="Calibri"> </font><br><font size=2 color=#004080 face="Calibri">Any ideas why these negative
numbers are being reported?</font><br><font size=2 color=#004080 face="Calibri"> </font><br><font size=2 color=#004080 face="Calibri">Cheers,</font><br><font size=2 color=#004080 face="Calibri">Luke.</font><br><a name=_MailEndCompose></a><font size=2 color=#004080 face="Calibri"> </font><br><font size=2 face="Calibri"><b>From:</b> gpfsug-discuss-bounces@spectrumscale.org
[</font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><font size=2 face="Calibri">mailto:gpfsug-discuss-bounces@spectrumscale.org</font></a><font size=2 face="Calibri">]
<b>On Behalf Of </b>Marc A Kaplan<b><br>Sent:</b> 02 July 2016 20:17<b><br>To:</b> gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><b><br>Subject:</b> Re: [gpfsug-discuss] Trapped Inodes</font><br><font size=3 face="Times New Roman"> </font><br><font size=2 face="Arial">I have been informed that it is possible
that a glitch can occur (for example an abrupt shutdown) which can leave
you in a situation where it looks like all snapshots are deleted, but there
is still a hidden snapshot that must be cleaned up... </font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br>The workaround is to create a snapshot `mmcrsnapshot fs dummy` and
then delete it `mmdelsnapshot fs dummy` and see if that clears up the situation...</font><font size=3 face="Times New Roman"><br></font><font size=2 face="Arial"><br>--marc</font><font size=3 face="Times New Roman"><br><br><br></font><font size=1 color=#5f5f5f face="Arial"><br>From: </font><font size=1 face="Arial">Luke
Raimbach <</font><a href=mailto:Luke.Raimbach@crick.ac.uk><font size=1 color=blue face="Arial"><u>Luke.Raimbach@crick.ac.uk</u></font></a><font size=1 face="Arial">></font><font size=1 color=#5f5f5f face="Arial"><br>To: </font><font size=1 face="Arial">gpfsug
main discussion list <</font><a href="mailto:gpfsug-discuss@spectrumscale.org"><font size=1 color=blue face="Arial"><u>gpfsug-discuss@spectrumscale.org</u></font></a><font size=1 face="Arial">></font><font size=1 color=#5f5f5f face="Arial"><br>Date: </font><font size=1 face="Arial">07/02/2016
06:05 AM</font><font size=1 color=#5f5f5f face="Arial"><br>Subject: </font><font size=1 face="Arial">Re:
[gpfsug-discuss] Trapped Inodes</font><font size=1 color=#5f5f5f face="Arial"><br>Sent by: </font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><font size=1 color=blue face="Arial"><u>gpfsug-discuss-bounces@spectrumscale.org</u></font></a><div align=center><hr noshade></div><br><font size=3 face="Times New Roman"><br><br><br>Hi Marc,</font><p><font size=3 face="Times New Roman">Thanks for the suggestion.</font><p><font size=3 face="Times New Roman">Snapshots were my first suspect
but there are none anywhere on the filesystem.</font><p><font size=3 face="Times New Roman">Cheers,<br>Luke.</font><p><font size=3 face="Times New Roman">On 1 Jul 2016 5:30 pm, Marc A Kaplan
<</font><a href=mailto:makaplan@us.ibm.com><font size=3 color=blue face="Times New Roman"><u>makaplan@us.ibm.com</u></font></a><font size=3 face="Times New Roman">>
wrote:</font><font size=2 face="Arial"><br>Question and Suggestion: Do you have any snapshots that might include
files that were in the fileset you are attempting to delete? Deleting
those snapshots will allow the fileset deletion to complete. The
snapshots are kinda intertwined with what was the "live" copy
of the inodes. In the GPFS "ditto" implementation of snapshotting,
for a file that has not changed since the snapshot operation, the
snapshot copy is not really a copy but just a pointer to the "live"
file. So even after you have logically deleted the "live"
files, the snapshot still points to those inodes you thought you deleted.
Rather than invalidate the snapshot, (you wouldn't want that, would
you?!) GPFS holds onto the inodes, until they are no longer referenced
by any snapshot.<br><br>--marc </font><font size=3 face="Times New Roman"><br><br></font><font size=1 color=#5f5f5f face="Arial"><br><br>From: </font><font size=1 face="Arial">Luke
Raimbach <</font><a href=mailto:Luke.Raimbach@crick.ac.uk><font size=1 color=blue face="Arial"><u>Luke.Raimbach@crick.ac.uk</u></font></a><font size=1 face="Arial">></font><font size=1 color=#5f5f5f face="Arial"><br>To: </font><font size=1 face="Arial">gpfsug
main discussion list <</font><a href="mailto:gpfsug-discuss@spectrumscale.org"><font size=1 color=blue face="Arial"><u>gpfsug-discuss@spectrumscale.org</u></font></a><font size=1 face="Arial">></font><font size=1 color=#5f5f5f face="Arial"><br>Date: </font><font size=1 face="Arial">07/01/2016
06:32 AM</font><font size=1 color=#5f5f5f face="Arial"><br>Subject: </font><font size=1 face="Arial">[gpfsug-discuss]
Trapped Inodes</font><font size=1 color=#5f5f5f face="Arial"><br>Sent by: </font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><font size=1 color=blue face="Arial"><u>gpfsug-discuss-bounces@spectrumscale.org</u></font></a><div align=center><hr noshade></div><br><font size=3 face="Times New Roman"><br></font><font size=2 face="Courier New"><br><br>Hi All,<br><br>I've run out of inodes on a relatively small filesystem. The total metadata
capacity allows for a maximum of 188,743,680 inodes.<br><br>A fileset containing 158,000,000 inodes was force deleted and has gone
into a bad state, where it is reported as (NULL) and has state "deleted":<br><br>Attributes for fileset (NULL):<br>===============================<br>Status
Deleted<br>Path
--<br>Id
15<br>Root inode
latest:<br>Parent Id
<none><br>Created
Wed Jun 15 14:07:51 2016<br>Comment<br>Inode space
8<br>Maximum number of inodes
158000000<br>Allocated inodes
158000000<br>Permission change flag
chmodAndSetacl<br>afm-associated
No<br><br>Offline mmfsck fixed a few problems, but didn't free these poor, trapped
inodes. Now I've run out and mmdf is telling me crazy things like this:<br><br>Inode Information<br>-----------------<br>Total number of used inodes in all Inode spaces:
0<br>Total number of free inodes in all Inode spaces:
27895680<br>Total number of allocated inodes in all Inode spaces: 27895680<br>Total of Maximum number of inodes in all Inode spaces: 34100000<br><br><br>Current GPFS build: "4.2.0.3".<br><br>Who will help me rescue these inodes?<br><br>Cheers,<br>Luke.<br><br>Luke Raimbach</font><font size=2 face="Cambria Math"></font><font size=2 face="Courier New"><br>Senior HPC Data and Storage Systems Engineer,<br>The Francis Crick Institute,<br>Gibbs Building,<br>215 Euston Road,<br>London NW1 2BE.<br><br>E: </font><a href=mailto:luke.raimbach@crick.ac.uk><font size=2 color=blue face="Courier New"><u>luke.raimbach@crick.ac.uk</u></font></a><font size=2 face="Courier New"><br>W: </font><a href=www.crick.ac.uk><font size=2 color=blue face="Courier New"><u>www.crick.ac.uk</u></font></a><font size=2 face="Courier New"><br><br>The Francis Crick Institute Limited is a registered charity in England
and Wales no. 1140062 and a company registered in England and Wales no.
06885462, with its registered office at 215 Euston Road, London NW1 2BE.<br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org</font><font size=3 color=blue face="Times New Roman"><u><br></u></font><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><font size=2 color=blue face="Courier New"><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></a><p><font size=2 color=#808080 face="Trebuchet MS"><i>The Francis Crick
Institute Limited is a registered charity in England and Wales no. 1140062
and a company registered in England and Wales no. 06885462, with its registered
office at 215 Euston Road, London NW1 2BE. </i></font><font size=2 face="Courier New">_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org</font><font size=3 color=blue face="Times New Roman"><u><br></u></font><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><font size=2 color=blue face="Courier New"><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></a><p><font size=3 face="Times New Roman"> </font><p><font size=2 color=#808080 face="Trebuchet MS"><i>The Francis Crick
Institute Limited is a registered charity in England and Wales no. 1140062
and a company registered in England and Wales no. 06885462, with its registered
office at 215 Euston Road, London NW1 2BE. </i></font><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br></font></tt><p><BR>