[gpfsug-discuss] quickest way to delete all files (and directories) in a file system

[Jaime Pinto]

Hi Tina

Please see attachment for a working version of 'mmrmdir', that I've been 
using for over 10 years. You may have to tweak it a bit for the name of 
the node you want to run it from, and the location of the policy (also 
attached).

I have used all the suggested ways so far on this thread to delete files 
in bulk. I still prefer to use this script when I don't want to disturb 
anything else on the cluster setup, in particular multi-cluster as you 
appear to have. It gives absolute and fine control of what to delete. 
You may also use it in test mode, and gradually delete only subsets of 
directories if you wish. Traversing the inodes database and creating the 
list of files to delete is what takes most of the time, whether deleting 
1M or 50M files.

To that effect, deleting and recreating file systems or filesets still 
takes a very long time, if those areas are populated with files.

Best
Jaime







On 6/30/22 07:31, Tina Friedrich wrote:
> Hello everyone,
> 
> this should be a simple question, but we can't quite figure out how to 
> best proceed.
> 
> We have some file systems that we want to, basically, empty out. As in 
> remove all files and directories currently on them. Both contain a 
> pretty large number of files/directories (something like 50,000,000, 
> with sometimes silly characters in the file names). 'rm -rf' clearly 
> isn't the way to go forward.
> 
> We've come up with either 'mmapplypolicy' (i.e. a policy to remove all 
> files) or removing and re-creating the file systems as options (open to 
> other suggestions!).
> 
> We want the file systems still; ideally without having to redo the 
> authentication and key swaps etc for the 'remote' clusters using them.
> 
> This is a Lenovo DSS, but I don't think it makes much of a difference.
> 
> So - what's the best way to proceed?
> 
> If it is mmapplypolicy - does anyone have a (tested/known working) 
> example of a policy to simply remove all files?
> 
> Thanks,
> Tina
> 

---
Jaime Pinto - Storage Analyst
SciNet HPC Consortium - www.scinet.utoronto.ca
University of Toronto
661 University Ave. (MaRS), Suite 1140
Toronto, ON, M5G1M1
P: 416-978-2755
C: 416-505-1477
-------------- next part --------------
#!/bin/bash

echo ""
echo "Command issued: "$0" "$@
echo ""

if [ "${HOSTNAME:0:12}" != datamover ]; then
	echo "You can only use mmrmdir on the datamovers"
	echo
	exit
fi

if [ "$1" == "" ] || [ "$1" == "-h" ] || [ "$1" == "-help" ] || [ "$1" == "--h" ] || [ "$1" == "--help" ] || [ $# -gt 2 ] || [ "$1" == "-test" ]; then
	echo "Usage: mmrmdir <directory absolute path> [-test]"
	echo "       -test  to verify what will be deleted"
	echo
	exit
fi

if [ "$2" != "" ] &&  [ "$2" != "-test" ]; then
        echo "Usage: mmrmdir <directory absolute path> [-test]"
        echo "       -test  to verify what will be deleted"
        echo
        exit
fi

echo -n "You have 10 seconds to cancel:"
for a in `seq 0 9`; do
    echo -n "  $a"
    sleep 1;
done
echo "  resuming ..."

LOCATION=$1
slash=`echo $LOCATION | grep /`

if [ "$slash" == "" ]; then
	echo "$LOCATION is not an absolute path"
	echo
	exit
fi

if [ "$2" == "-test" ]; then
	mmapplypolicy $LOCATION -P /usr/lpp/mmfs/bin/mmpolicyRules-DELETE-ALL -I test -L 2
else
	mmapplypolicy $LOCATION -P /usr/lpp/mmfs/bin/mmpolicyRules-DELETE-ALL -I defer -L 2
	if [ "$?" != 0 ]
	then
		echo #### there was an error with mmapplypolicy execution ####
	else
		echo removing empty directories in $LOCATION
		rm -rf $LOCATION
	fi
fi
exit 0
-------------- next part --------------
/* Define deletion rules for aged files in /dev/scratch (system pool by default).
   If the file has not been accessed in 90 days AND not owned by root then delete it. */

RULE 'DelSystem' DELETE
	FROM POOL 'system' 
	FOR FILESET('root')