[gpfsug-discuss] Migrate/syncronize data from Isilon to Scale over NFS?

Jonathan Buzzard jonathan.buzzard at strath.ac.uk
Tue Nov 17 13:53:43 GMT 2020


On 17/11/2020 11:51, Andi Christiansen wrote:
> Hi all,
> 
> thanks for all the information, there was some interesting things
> amount it..
> 
> I kept on going with rsync and ended up making a file with all top
> level user directories and splitting them into chunks of 347 per
> rsync session(total 42000 ish folders). yesterday we had only 14
> sessions with 3000 folders in each and that was too much work for one
> rsync session..

Unless you use something similar to my DB suggestion it is almost 
inevitable that some of those rsync sessions are going to have issues 
and you will have no way to track it or even know it has happened unless 
you do a single final giant catchup/check rsync.

I should add that a copy of the sqlite DB is cover your backside 
protection when a user pops up claiming that you failed to transfer one 
of their vitally important files six months down the line and the old 
system is turned off and scrapped.


JAB.

-- 
Jonathan A. Buzzard                         Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG



More information about the gpfsug-discuss mailing list