[gpfsug-discuss] Best way to migrate data

[Sanchez, Paul]

Sharding can also work, if you have a storage-connected compute grid in your environment:  If you enumerate all of the directories, then use a non-recursive rsync for each one, you may be able to parallelize the workload by using several clients simultaneously.  It may still max out the links of these clients (assuming your source read throughput and target write throughput bottlenecks aren't encountered first) but it may run that way for 1/100th of the time if you can use 100+ machines.

-Paul
-----Original Message-----
From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of Buterbaugh, Kevin L
Sent: Thursday, October 18, 2018 2:26 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Best way to migrate data

Hi Dwayne,

I’m assuming you can’t just let an rsync run, possibly throttled in some way?  If not, and if you’re just tapping out your network, then would it be possible to go old school?  We have parts of the Medical Center here where their network connections are … um, less than robust.  So they tar stuff up to a portable HD, sneaker net it to us, and we untar is from an NSD server.

HTH, and I really hope that someone has a better idea than that!

Kevin

> On Oct 18, 2018, at 12:19 PM, Dwayne.Hart at med.mun.ca wrote:
> 
> Hi,
> 
> Just wondering what the best recipe for migrating a user’s home directory content from one GFPS file system to another which hosts a larger research GPFS file system? I’m currently using rsync and it has maxed out the client system’s IB interface.
> 
> Best,
> Dwayne 
> —
> Dwayne Hart | Systems Administrator IV
> 
> CHIA, Faculty of Medicine 
> Memorial University of Newfoundland 
> 300 Prince Philip Drive
> St. John’s, Newfoundland | A1B 3V6
> Craig L Dobbin Building | 4M409
> T 709 864 6631
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7Ccca728d2d61f4be06bcd08d6351f3650%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636754805507359478&sdata=2YAiqgqKl4CerlyCn3vJ9v9u%2FrGzbfa7aKxJ0PYV%2Fhc%3D&reserved=0

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss