[gpfsug-discuss] Migrate/syncronize data from Isilon to Scale over NFS?

[Chris Schlipalius]

Ø  I would counsel in the strongest possible terms against that approach.

Ø  Basically you have to be assured that none of your file names have "wacky" characters in them, because handling "wacky" characters in file

Ø  names is exceedingly difficult. I cannot stress how hard it is and the above example does not handle all "wacky" characters in file names.

 

Well that’s indeed another kettle of fish if you have irregular/special naming of files, no I didn’t cover that and if you have millions of files, yes a list would be unwieldy, then I would be tarring up dirs. before moving… and then untarring on GPFS  or breaking up the list into sets or sub lists. 

If you have these wacky types of file names well there are fixes as in the rsync manpages… yes not easy but possible..

 

Ie

 

1.       -s, --protect-args

 

2.       As per usual you can escape the spaces, or substitute for spaces. rsync -avuz user at server1.com:"${remote_path// /\\ }" .

 

3.       Single quote the file name and path inside double quotes.

 

 

 

Ø  One thing I didn't mention is that I would run anything with in a screen (or tmux if that is your poison) and turn on logging.

 

Absolutely agree…

 

Ø  For those interested I am in the process of cleaning up the script a bit and will post it somewhere in due course.

Ø  JAB.

 

Would be interesting to see….

 

I’ve also had success on GPFS with DCP and possibly this would be another option 

 

Regards,

Chris Schlipalius

 

Team Lead, Data Storage Infrastructure, Supercomputing Platforms, Pawsey Supercomputing Centre (CSIRO)

1 Bryce Avenue

Kensington  WA  6151

Australia

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20201119/e47e1eb4/attachment-0002.htm>