[gpfsug-discuss] Odd behaviour with regards to reported free space

[TURNER Aaron]

Dear Jonathan,

This is what I had assumed was the case. Since the system ended up with an enforced reboot before we had time for further investigation I wasn't able to confirm this.

> I can be very confusing for end users, especially when what is holding onto the file is some random zombie process on another node that died last month.

Yes, that's very likely to have been the case.

Regards

Aaron Turner

-----Original Message-----
From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of Jonathan Buzzard
Sent: 18 February 2020 10:50
To: gpfsug-discuss at spectrumscale.org
Subject: Re: [gpfsug-discuss] Odd behaviour with regards to reported free space

On Tue, 2020-02-18 at 09:28 +0000, TURNER Aaron wrote:
> Dear All,
>
> This has happened more than once with both 4.2.3 and 5.0. The
> instances may not be related.
>
> In the first instance, usage was high (over 90%) and so users were
> encouraged to delete files. One user deleted a considerable number of
> files equal to around 10% of the total storage. Reported usage did not
> fall. There were not obviously any waiters. Has anyone seen anything
> similar?
>

I have seen similar behaviour a number of times.

I my experience it is because a process somewhere has an open file handle on one or more files/directories. So you can delete the file and it goes from a directory listing; it's no long visible when you do ls.

However the file has not actually gone, and will continue to count towards total file system usage, user/group/fileset quota's etc.

Once the errant process is found and killed magically the space becomes free.

I can be very confusing for end users, especially when what is holding onto the file is some random zombie process on another node that died last month.


JAB.

--
Jonathan A. Buzzard                         Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG _______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.