[gpfsug-discuss] Migrate/syncronize data from Isilon to Scale over NFS?
Uwe Falke
UWEFALKE at de.ibm.com
Tue Nov 17 08:57:07 GMT 2020
Hi, Andi, sorry I just took your 20Gbit for the sign of 2x10Gbps bons, but
it is over two nodes, so no bonding. But still, I'd expect to open several
TCP connections in parallel per source-target pair (like with several
rsyncs per source node) would bear an advantage (and still I thing NFS
doesn't do that, but I can be wrong).
If more nodes have access to the Isilon data they could also participate
(and don't need NFS exports for that).
Mit freundlichen Grüßen / Kind regards
Dr. Uwe Falke
IT Specialist
Hybrid Cloud Infrastructure / Technology Consulting & Implementation
Services
+49 175 575 2877 Mobile
Rathausstr. 7, 09111 Chemnitz, Germany
uwefalke at de.ibm.com
IBM Services
IBM Data Privacy Statement
IBM Deutschland Business & Technology Services GmbH
Geschäftsführung: Sven Schooss, Stefan Hierl
Sitz der Gesellschaft: Ehningen
Registergericht: Amtsgericht Stuttgart, HRB 17122
From: Uwe Falke/Germany/IBM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: 17/11/2020 09:50
Subject: Re: [EXTERNAL] [gpfsug-discuss] Migrate/syncronize data
from Isilon to Scale over NFS?
Hi Andi,
what about leaving NFS completeley out and using rsync (multiple rsyncs
in parallel, of course) directly between your source and target servers?
I am not sure how many TCP connections (suppose it is NFS4) in parallel
are opened between client and server, using a 2x bonded interface well
requires at least two. That combined with the DB approach suggested by
Jonathan to control the activity of the rsync streams would be my best
guess.
If you have many small files, the overhead might still kill you. Tarring
them up into larger aggregates for transfer would help a lot, but then you
must be sure they won't change or you need to implement your own version
control for that class of files.
Mit freundlichen Grüßen / Kind regards
Dr. Uwe Falke
IT Specialist
Hybrid Cloud Infrastructure / Technology Consulting & Implementation
Services
+49 175 575 2877 Mobile
Rathausstr. 7, 09111 Chemnitz, Germany
uwefalke at de.ibm.com
IBM Services
IBM Data Privacy Statement
IBM Deutschland Business & Technology Services GmbH
Geschäftsführung: Sven Schooss, Stefan Hierl
Sitz der Gesellschaft: Ehningen
Registergericht: Amtsgericht Stuttgart, HRB 17122
From: Andi Christiansen <andi at christiansen.xxx>
To: "gpfsug-discuss at spectrumscale.org"
<gpfsug-discuss at spectrumscale.org>
Date: 16/11/2020 20:44
Subject: [EXTERNAL] [gpfsug-discuss] Migrate/syncronize data from
Isilon to Scale over NFS?
Sent by: gpfsug-discuss-bounces at spectrumscale.org
Hi all,
i have got a case where a customer wants 700TB migrated from isilon to
Scale and the only way for him is exporting the same directory on NFS from
two different nodes...
as of now we are using multiple rsync processes on different parts of
folders within the main directory. this is really slow and will take
forever.. right now 14 rsync processes spread across 3 nodes fetching from
2..
does anyone know of a way to speed it up? right now we see from 1Gbit to
3Gbit if we are lucky(total bandwidth) and there is a total of 30Gbit from
scale nodes and 20Gbits from isilon so we should be able to reach just
under 20Gbit...
if anyone have any ideas they are welcome!
Thanks in advance
Andi Christiansen _______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
More information about the gpfsug-discuss
mailing list