[gpfsug-discuss] Migrate/syncronize data from Isilon to Scale over NFS?

[Skylar Thompson]

When we did a similar (though larger, at ~2.5PB) migration, we used rsync
as well, but ran one rsync process per Isilon node, and made sure the NFS
clients were hitting separate Isilon nodes for their reads. We also didn't
have more than one rsync process running per client, as the Linux NFS
client (at least in CentOS 6) was terrible when it came to concurrent access.

Whatever method you end up using, I can guarantee you will be much happier
once you are on GPFS. :)

On Mon, Nov 16, 2020 at 08:44:14PM +0100, Andi Christiansen wrote:
> Hi all,
> 
> i have got a case where a customer wants 700TB migrated from isilon to Scale and the only way for him is exporting the same directory on NFS from two different nodes...
> 
> as of now we are using multiple rsync processes on different parts of folders within the main directory. this is really slow and will take forever.. right now 14 rsync processes spread across 3 nodes fetching from 2.. 
> 
> does anyone know of a way to speed it up? right now we see from 1Gbit to 3Gbit if we are lucky(total bandwidth) and there is a total of 30Gbit from scale nodes and 20Gbits from isilon so we should be able to reach just under 20Gbit...
> 
> 
> if anyone have any ideas they are welcome! 
> 
> 
> Thanks in advance 
> Andi Christiansen

> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss


-- 
-- Skylar Thompson (skylar2 at u.washington.edu)
-- Genome Sciences Department (UW Medicine), System Administrator
-- Foege Building S046, (206)-685-7354
-- Pronouns: He/Him/His