[gpfsug-discuss] Migrate/syncronize data from Isilon to Scale over NFS?

[Andi Christiansen]

Hi Jonathan,

i would be very interested in seeing your scripts when they are posted. Let me know where to get them!

Thanks a bunch!
Andi Christiansen

> On 11/18/2020 12:48 PM Jonathan Buzzard <jonathan.buzzard at strath.ac.uk> wrote:
> 
>  
> On 17/11/2020 23:17, Chris Schlipalius wrote:
> > So at my last job we used to rsync data between isilons across campus, 
> > and isilon to Windows File Cluster (and back).
> > 
> > I recommend using dry run to generate a list of files and then use this 
> > to run with rysnc.
> > 
> > This allows you also to be able to break up the transfer into batches, 
> > and check if files have changed before sync (say if your isilon files 
> > are not RO.
> > 
> > Also ensure you have a recent version of rsync that preserves extended 
> > attributes and check your ACLS.
> > 
> > A dry run example:
> > 
> > https://unix.stackexchange.com/a/261372 
> > 
> > I always felt more comfortable having a list of files before a sync….
> > 
> 
> I would counsel in the strongest possible terms against that approach.
> 
> Basically you have to be assured that none of your file names have 
> "wacky" characters in them, because handling "wacky" characters in file 
> names is exceedingly difficult. I cannot stress how hard it is and the 
> above example does not handle all "wacky" characters in file names.
> 
> So what do I mean by "wacky" characters. Well remember a file name can 
> have just about anything in it on Linux with the exception of '/', and 
> users especially when using a GUI, and even more so if they are Mac 
> users can and do use what I will call "wacky" characters in their file 
> names.
> 
> The obvious ones are spaces, but it's not just ASCII 0x20, but tabs too. 
> Then there is the use of the wildcard characters, especially '?' but 
> also '*'.
> 
> Not too difficult to handle you might say. Right now deal with a file 
> name with a newline character in it :-) Don't ask me how or why you even 
> do that but let me assure you that I have seen them on more than one 
> occasion. And now your dry run list is broken...
> 
> Not only that if you have a few hundred million files to move a list 
> just becomes unwieldy anyway.
> 
> One thing I didn't mention is that I would run anything with in a screen 
> (or tmux if that is your poison) and turn on logging.
> 
> For those interested I am in the process of cleaning up the script a bit 
> and will post it somewhere in due course.
> 
> 
> JAB.
> 
> -- 
> Jonathan A. Buzzard                         Tel: +44141-5483420
> HPC System Administrator, ARCHIE-WeSt.
> University of Strathclyde, John Anderson Building, Glasgow. G4 0NG
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss