[gpfsug-discuss] AFM - how to update directories with deleted files during prefetch

[Venkateswara R Puvvada]

What is the version of GPFS ?

>Mostly I want to update the directories to make AFM cache aware of file 
deletions on home. On home I can use a policy run to find all directories 
which changed since the last >update and pass them to prefetch on AFM 
cache. 

AFM prefetch has undocumented option to delete files from cache without 
doing lookup from home. It supports all types of list files. Find all 
deleted file from home and prefetch at cache to delete them.

mmafmctl device prefetch -j fileset --list-file 
<list-of-files-to-be-deleted-from-cache> --delete

>Wish – maybe an RFE?
>I can use policy runs to collect all changed items on home since the last 
update. I wish that I can pass this list to afm prefetch to do all updates 
on AFM cache, too. Same >as backup tools use the list to do incremental 
backups.

This feature support was already added but undocumented today. This 
feature will be externalized in future releases.

~Venkat (vpuvvada at in.ibm.com)



From:   "Billich Heinrich Rainer (PSI)" <heiner.billich at psi.ch>
To:     "gpfsug-discuss at spectrumscale.org" 
<gpfsug-discuss at spectrumscale.org>
Date:   06/30/2017 03:37 PM
Subject:        [gpfsug-discuss] AFM - how to update directories with 
deleted files   during prefetch
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Hello

I have a short question about AFM prefetch and some more remarks regarding 
AFM and it’s use for data migration. I understand that many  of you have 
done this for very large amounts of data and number of files. I would 
welcome an input, comments or remarks. Sorry if this is a bit too long for 
a mailing list.

Short:
How can I tell an AFM cache  to update a directory when I do prefetch? I 
know about ‘find .’ or ‘ls –lsR’ but this really is no option for us as it 
takes too long. Mostly I want to update the directories to make AFM cache 
aware of file deletions on home. On home I can use a policy run to find 
all directories which changed since the last update and pass them to 
prefetch on AFM cache. 

I know that I can find some workaround based on the directory list, like 
an ‘ls –lsa’ just for those directories, but this doesn’t sound very 
efficient. And depending on cache effects and timeout settings it may work 
or not (o.k. – it will work most time).

We do regular file deletions and will accumulated millions of deleted 
files on cache over time if we don’t update the directories to make AFM 
cache aware of the deletion.

Background:
We will use AFM to migrate data on filesets to another cluster. We have to 
do this several times in the next few months, hence I want to get a 
reliable and easy to use procedure. The old system is home, the new system 
is a read-only AFM cache. We want to use ‘mmafmctl prefetch’ to move the 
data. Home will be in use while we run the migration. Once almost all data 
is moved we do a (short) break for a last sync and make the read-only AFM 
cache a ‘normal’ fileset. During the break I want  to use policy runs and 
prefetch only and no time consuming ‘ls –lsr’ or ‘find .’ I don’t want to 
use this metadata intensive posix operation during operation, either.

More general:
AFM can be used for data migration. But I don’t see how to use it 
efficiently. How to do incremental transfers, how to ensure that the we 
really have identical copies before we switch and how to keep the switch 
time short , i.e. the time when both old and new aren’t accessible for 
clients,

Wish – maybe an RFE?
I can use policy runs to collect all changed items on home since the last 
update. I wish that I can pass this list to afm prefetch to do all updates 
on AFM cache, too. Same as backup tools use the list to do incremental 
backups.

And a tool to create policy lists of home and cache and to compare the 
lists would be nice, too. As you do this during the break/switch it should 
be fast and reliable and leave no doubts.

Kind regards,

Heiner

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170630/17b621dc/attachment-0002.htm>