[gpfsug-discuss] Migrating billions of files?
Chris Schlipalius
chris.schlipalius at pawsey.org.au
Wed Mar 6 09:56:31 GMT 2019
Hi Bob, so Simon has hit the nail on the head.
So it’s a challenge, we used dcp with multiple parallel threads per nsd with mmdsh - 2PB and millions of files, it’s worth a test as it does look after xattribs, but test it.
See https://github.com/hpc/dcp
Test the preserve:
-p, --preserve
Preserve the original files' owner, group, permissions (including the setuid and setgid bits), time of last modification and time of last access. In case duplication of owner or group fails, the setuid and setgid bits are cleared.
-------
We migrated between 12K storage FS a few years back.
My colleague also has tested https://www.nersc.gov/users/storage-and-file-systems/transferring-data/bbcp/ or http://www.slac.stanford.edu/~abh/bbcp/
It’s excellent I hear with xattribs and recursive small files copy.
I steer clear of rsync, different versions do not preserve xattribs and this is a bit of an issue some have found
Regards,
Chris Schlipalius
Team Lead, Data Storage Infrastructure, Data & Visualisation, Pawsey Supercomputing Centre (CSIRO)
13 Burvill Court
Kensington WA 6151
Australia
More information about the gpfsug-discuss
mailing list