[gpfsug-discuss] Migrating billions of files?

Chris Schlipalius chris.schlipalius at pawsey.org.au
Wed Mar 6 09:56:31 GMT 2019


Hi Bob, so Simon has hit the nail on the head.

So it’s a challenge, we used dcp with multiple parallel threads per nsd with mmdsh  - 2PB and millions of files, it’s worth a test as it does look after xattribs, but test it.
See https://github.com/hpc/dcp
Test the preserve:
 -p, --preserve

Preserve the original files' owner, group, permissions (including the setuid and setgid bits), time of last modification and time of last access. In case duplication of owner or group fails, the setuid and setgid bits are cleared.
-------

We migrated between 12K storage FS a few years back.

My colleague also has tested https://www.nersc.gov/users/storage-and-file-systems/transferring-data/bbcp/  or http://www.slac.stanford.edu/~abh/bbcp/ 

It’s excellent I hear with xattribs and recursive small files copy.

I steer clear of rsync, different versions do not preserve xattribs and this is a bit of an issue some have found

Regards,
Chris Schlipalius
 
Team Lead, Data Storage Infrastructure, Data & Visualisation, Pawsey Supercomputing Centre (CSIRO)
13 Burvill Court
Kensington  WA  6151
Australia






More information about the gpfsug-discuss mailing list