[gpfsug-discuss] Migrate/syncronize data from Isilon to Scale over NFS?

[Jonathan Buzzard]

On 17/11/2020 23:17, Chris Schlipalius wrote:
> So at my last job we used to rsync data between isilons across campus, 
> and isilon to Windows File Cluster (and back).
> 
> I recommend using dry run to generate a list of files and then use this 
> to run with rysnc.
> 
> This allows you also to be able to break up the transfer into batches, 
> and check if files have changed before sync (say if your isilon files 
> are not RO.
> 
> Also ensure you have a recent version of rsync that preserves extended 
> attributes and check your ACLS.
> 
> A dry run example:
> 
> https://unix.stackexchange.com/a/261372 
> 
> I always felt more comfortable having a list of files before a sync….
> 

I would counsel in the strongest possible terms against that approach.

Basically you have to be assured that none of your file names have 
"wacky" characters in them, because handling "wacky" characters in file 
names is exceedingly difficult. I cannot stress how hard it is and the 
above example does not handle all "wacky" characters in file names.

So what do I mean by "wacky" characters. Well remember a file name can 
have just about anything in it on Linux with the exception of '/', and 
users especially when using a GUI, and even more so if they are Mac 
users can and do use what I will call "wacky" characters in their file 
names.

The obvious ones are spaces, but it's not just ASCII 0x20, but tabs too. 
Then there is the use of the wildcard characters, especially '?' but 
also '*'.

Not too difficult to handle you might say. Right now deal with a file 
name with a newline character in it :-) Don't ask me how or why you even 
do that but let me assure you that I have seen them on more than one 
occasion. And now your dry run list is broken...

Not only that if you have a few hundred million files to move a list 
just becomes unwieldy anyway.

One thing I didn't mention is that I would run anything with in a screen 
(or tmux if that is your poison) and turn on logging.

For those interested I am in the process of cleaning up the script a bit 
and will post it somewhere in due course.


JAB.

-- 
Jonathan A. Buzzard                         Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG