[gpfsug-discuss] Backing up GPFS with Rsync

[Ryan Novosielski]

Yup, you want to use the policy engine:

https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adv.doc/bl1adv_policyrules.htm

Something in here ought to help. We do something like this (but I’m reluctant to provide examples as I’m actually suspicious that we don’t have it quite right and are passing far too much stuff to rsync).

--
#BlackLivesMatter
____
|| \\UTGERS,  	 |---------------------------*O*---------------------------
||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
     `'

> On Mar 9, 2021, at 9:19 PM, William Burke <bill.burke.860 at gmail.com> wrote:
> 
>  I would like to know what files were modified/created/deleted (only for the current day) on the GPFS's file system so that I could rsync ONLY those files to a predetermined external location. I am running GPFS 4.2.3.9
> 
> Is there a way to access the GPFS's metadata directly so that I do not have to traverse the filesystem looking for these files? If i use the rsync tool it will scan the file system which is 400+ million files.  Obviously this will be problematic to complete a scan in a day, if it would ever complete single-threaded. There are tools or scripts that run multithreaded rsync but it's still a brute force attempt. and it would be nice to know where the delta of files that have changed.
> 
> I began looking at Spectrum Scale Data Management (DM) API but I am not sure if this is the best approach to looking at the GPFS metadata - inodes, modify times, creation times, etc.
> 
> 
> 
> -- 
> 
> Best Regards,
> 
> William Burke (he/him)
> Lead HPC Engineer
> Advance Research Computing
> 860.255.8832 m | LinkedIn
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss