[gpfsug-discuss] Migrating billions of files?

[Simon Thompson]

AFM doesn’t work well if you have dependent filesets though .. which we did for quota purposes.

Simon

From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of "YARD at il.ibm.com" <YARD at il.ibm.com>
Reply-To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Date: Wednesday, 6 March 2019 at 09:01
To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Migrating billions of files?

Hi

What permissions you have ? Do u have only Posix , or also SMB attributes ?

If only posix attributes you can do the following:

- rsync (which will work on different filesets/directories in parallel.
- AFM (but in case you need rollback - it will be problematic)


Regards

________________________________





Yaron Daniel

 94 Em Ha'Moshavot Rd

[cid:_1_0FC36C500FC3669C00318CDBC22583B5]

Storage Architect – IL Lab Services (Storage)

 Petach Tiqva, 49527

IBM Global Markets, Systems HW Sales

 Israel







Phone:

+972-3-916-5672





Fax:

+972-3-916-5672





Mobile:

+972-52-8395593





e-mail:

yard at il.ibm.com





IBM Israel<http://www.ibm.com/il/he/>










[IBM Storage Strategy and Solutions v1][IBM Storage Management and Data Protection v1][cid:_1_0FA0428C0FA03A6C00318CDBC22583B5][cid:_1_0FA044940FA03A6C00318CDBC22583B5] [https://acclaim-production-app.s3.amazonaws.com/images/6c2c3858-6df8-45be-ac2b-f93b8da74e20/Data%2BDriven%2BMulti%2BCloud%2BStrategy%2BV1%2Bver%2B4.png]  [FlashSystem A9000/R Foundations] [All Flash Storage Foundations V2]



From:        "Oesterlin, Robert" <Robert.Oesterlin at nuance.com>
To:        gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:        03/05/2019 11:57 PM
Subject:        [gpfsug-discuss] Migrating billions of files?
Sent by:        gpfsug-discuss-bounces at spectrumscale.org
________________________________


I’m looking at migration 3-4 Billion files, maybe 3PB of data between GPFS clusters. Most of the files are small - 60% 8K or less. Ideally I’d like to copy at least 15-20M files per day - ideally 50M.



Any thoughts on how achievable this is? Or what to use? Either with AFM, mpifileutils, rsync.. other? Many of these files would be in 4k inodes. Destination is ESS.





Bob Oesterlin

Sr Principal Storage Engineer, Nuance

 _______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190306/ac8aeb7a/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.gif
Type: image/gif
Size: 1852 bytes
Desc: image001.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190306/ac8aeb7a/attachment-0016.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.gif
Type: image/gif
Size: 4377 bytes
Desc: image002.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190306/ac8aeb7a/attachment-0017.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.gif
Type: image/gif
Size: 5094 bytes
Desc: image003.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190306/ac8aeb7a/attachment-0018.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.gif
Type: image/gif
Size: 4747 bytes
Desc: image004.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190306/ac8aeb7a/attachment-0019.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image005.gif
Type: image/gif
Size: 4558 bytes
Desc: image005.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190306/ac8aeb7a/attachment-0020.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image006.gif
Type: image/gif
Size: 5094 bytes
Desc: image006.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190306/ac8aeb7a/attachment-0021.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image007.gif
Type: image/gif
Size: 4787 bytes
Desc: image007.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190306/ac8aeb7a/attachment-0022.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image008.gif
Type: image/gif
Size: 5055 bytes
Desc: image008.gif
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190306/ac8aeb7a/attachment-0023.gif>