[gpfsug-discuss] Quick delete of huge tree
janfrode at tanso.net
Tue Apr 20 12:52:50 BST 2021
A couple of ideas.
The KC recommends adding WEIGHT(DIRECTORY_HASH) to group deletions within a
directory. Then maybe also do it as a 2-step process, in the same policy
run. Where you delete all non-directories first, and then deletes the
directories in a depth-first order using WEIGTH(Length(PATH_NAME)):
RULE 'delnondir' DELETE
WHERE PATH_NAME LIKE '/mypath/%' AND NOT MISC_ATTRIBUTES LIKE '%D%'
RULE 'deldir' DELETE
WHERE PATH_NAME LIKE '/mypath/%' AND MISC_ATTRIBUTES LIKE '%D%'
On Tue, Apr 20, 2021 at 1:18 PM Ulrich Sibiller <
u.sibiller at science-computing.de> wrote:
> Hello *,
> I have to delete a subtree of about ~50 million files in thousands of
> subdirs, ~14TB of data.
> Running a recursive rm is very slow so I setup a simple policy file:
> RULE 'delstuff' DELETE
> WHERE PATH_NAME LIKE '/mypath/%'
> This kinda works but is not really fast, either. It even requires a second
> run because files and
> directories within the tree will be processed in arbitrary order so it
> will happen quite frequently
> that a directory is going to be deleted before its content has been
> removed completely. For those
> dirs I see an error message and have to delete afterwards.
> I am wondering if there's a quicker way. Given the fact that this is a
> whole tree I think there's
> should be a quick way to unlink the complete inode hierachy.
> Unfortunately we are not using a fileset for that tree...
> So are there any ideas how to solve that more efficiently?
> Science + Computing AG
> Vorstandsvorsitzender/Chairman of the board of management:
> Dr. Martin Matzke
> Vorstand/Board of Management:
> Matthias Schempp, Sabine Hohenstein
> Vorsitzender des Aufsichtsrats/
> Chairman of the Supervisory Board:
> Philippe Miltin
> Aufsichtsrat/Supervisory Board:
> Martin Wibbe, Ursula Morgenstern
> Sitz/Registered Office: Tuebingen
> Registergericht/Registration Court: Stuttgart
> Registernummer/Commercial Register No.: HRB 382196
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the gpfsug-discuss