[gpfsug-discuss] naive question about rsync: run it on a client or on NSD server?
Chris Schlipalius
chris.schlipalius at pawsey.org.au
Fri Feb 14 22:47:00 GMT 2020
We have used DCP for this, with mmdsh as DCP is MPI and multi node with auto resume. You can also customise threads numbers etc.
DDN in fact ran it for us first on our NSD servers for a multi petabyte migration project.
It’s in git.
For client side, we recommend and use bbcp, our users use this to sync data. It’s fast and reliable and supports resume also.
If you do use rsync, as suggested, do dryruns and then a sync and then final copy, as is often run on Isilons to keep geographically separate Isilons in sync.
Newest version of rsync also.
Regards, Chris Schlipalius
Team Lead
Data and Storage
The Pawsey Supercomputing Centre
Australia
> On 15 Feb 2020, at 1:28 am, gpfsug-discuss-request at spectrumscale.org wrote:
>
> Send gpfsug-discuss mailing list submissions to
> gpfsug-discuss at spectrumscale.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> or, via email, send a message with subject or body 'help' to
> gpfsug-discuss-request at spectrumscale.org
>
> You can reach the person managing the list at
> gpfsug-discuss-owner at spectrumscale.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of gpfsug-discuss digest..."
>
>
> Today's Topics:
>
> 1. naive question about rsync: run it on a client or on NSD
> server? (Giovanni Bracco)
> 2. Re: naive question about rsync: run it on a client or on NSD
> server? (Simon Thompson)
> 3. Re: naive question about rsync: run it on a client or on NSD
> server? (Sanchez, Paul)
> 4. Re: naive question about rsync: run it on a client or on NSD
> server? (Wahl, Edward)
> 5. Re: mmbackup [--tsm-servers TSMServer[, TSMServer...]]
> (Valdis Kl=?utf-8?Q?=c4=93?=tnieks)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 14 Feb 2020 14:25:08 +0100
> From: Giovanni Bracco <giovanni.bracco at enea.it>
> To: gpfsug-discuss at spectrumscale.org
> Subject: [gpfsug-discuss] naive question about rsync: run it on a
> client or on NSD server?
> Message-ID: <cd38a061-8289-0616-d301-581a9c833a03 at enea.it>
> Content-Type: text/plain; charset=utf-8; format=flowed
>
> We must replicate about 100 TB data between two filesystems supported by
> two different storages (DDN9900 and DDN7990) both connected to the same
> NSD servers (6 of them) and we plan to use rsync.
>
> Non special GPFS attributes, just the standard POSIX one, we plan to use
> the standard rsync.
>
> The question:
> is there any advantage in running the rsync on one of the NSD server or
> is better to run it on a client?
>
> The environment:
> GPFS 4.2.3.19, NSD CentOS7.4, clients mostly CentOS6.4 (connected by IB
> QDR) and CentOS7.3 (connected by OPA), connection between NSD and
> storage with IB QDR)
>
> Giovanni
>
> --
> Giovanni Bracco
> phone +39 351 8804788
> E-mail giovanni.bracco at enea.it
> WWW http://www.afs.enea.it/bracco
>
>
> ------------------------------
>
> Message: 2
> Date: Fri, 14 Feb 2020 14:56:30 +0000
> From: Simon Thompson <S.J.Thompson at bham.ac.uk>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: Re: [gpfsug-discuss] naive question about rsync: run it on a
> client or on NSD server?
> Message-ID: <404B2B75-C094-43CC-9146-C00410F31578 at bham.ac.uk>
> Content-Type: text/plain; charset="utf-8"
>
> I wouldn't run it on an NSD server. Ideally you want to avoid running other processes etc on there.
>
> If you are running on clients, you also might want to look at: https://github.com/hpc/mpifileutils
>
> And use MPI to parallelise the find and copy.
>
> Simon
>
> ?On 14/02/2020, 14:25, "gpfsug-discuss-bounces at spectrumscale.org on behalf of giovanni.bracco at enea.it" <gpfsug-discuss-bounces at spectrumscale.org on behalf of giovanni.bracco at enea.it> wrote:
>
> We must replicate about 100 TB data between two filesystems supported by
> two different storages (DDN9900 and DDN7990) both connected to the same
> NSD servers (6 of them) and we plan to use rsync.
>
> Non special GPFS attributes, just the standard POSIX one, we plan to use
> the standard rsync.
>
> The question:
> is there any advantage in running the rsync on one of the NSD server or
> is better to run it on a client?
>
> The environment:
> GPFS 4.2.3.19, NSD CentOS7.4, clients mostly CentOS6.4 (connected by IB
> QDR) and CentOS7.3 (connected by OPA), connection between NSD and
> storage with IB QDR)
>
> Giovanni
>
> --
> Giovanni Bracco
> phone +39 351 8804788
> E-mail giovanni.bracco at enea.it
> WWW http://www.afs.enea.it/bracco
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
>
> ------------------------------
>
> Message: 3
> Date: Fri, 14 Feb 2020 16:24:40 +0000
> From: "Sanchez, Paul" <Paul.Sanchez at deshaw.com>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: Re: [gpfsug-discuss] naive question about rsync: run it on a
> client or on NSD server?
> Message-ID: <b9c5db70083e459bb92e621162e2f9d6 at deshaw.com>
> Content-Type: text/plain; charset="utf-8"
>
> Some (perhaps obvious) points to consider:
>
> - There are some corner cases (e.g. preserving hard-linked files or sparseness) which require special options.
>
> - Depending on your level of churn, it may be helpful to pre-stage the sync before your cutover so that there is less data movement required, and you're primarily comparing metadata.
>
> - Files on the source filesysytem might change (and become internally inconsistent) during your rsync, so you should generally sync from a snapshot on the source.
>
> - If users can still modify the source filesystem, then you might not get everything. For the final sync, you may need to make the source read-only, or unmount it on clients, kill user processes, or some combination to prevent all new writes from succeeding. (If you're going to use the clients for MPI sync, you obviously need the filesystem to remain mounted there so you may need to take other measures to keep users away.)
>
> - If you decide to do a final "offline" sync, you want it to be fast so users can get back to work sooner, so parallelism is usually a must. If you have lots of filesets, then that's a convenient way to split the work.
>
> - If you have any filesets with many more inodes than the others, keep in mind that those will likely take the longest to complete.
>
> - Test, test, test. You usually won't get this right on the first go or know how long a full sync takes without practice. Remember that you'll need to employ options to delete extraneous files on the target when you're syncing over the top of a previous attempt, since files intentionally deleted on the source aren't usually welcome if they reappear after a migration.
>
> - Verify. Whether you use rsync of dsync, repeating the process with dry-run/no-op flags which report differences can be helpful to increase your confidence in the process. If you don't have time to verify after the final offline sync, hopefully you were able to fit this in during testing.
>
>
> Some thoughts about whether it's appropriate to use NSD servers as sync hosts...
>
> - If they are the managers and they have the best (direct) connectivity to the metadata NSDs, then I would at least consider them before ruling this out, with caveats...
> - do they have enough available RAM and CPU?
> - where do they get their software? Do you trust the version of kernel/libc/rsync there to behave as you expect?
> - if the data NSDs aren't local to these NSD servers, do they have sufficient network connectivity to not cause other problems during the sync?
>
> - Test at low parallelism and work your way up. You can also compare performance of this method with any other, on a small scale, in your environment to see what you can expect from each.
>
> Good luck,
> Paul
>
> -----Original Message-----
> From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of Simon Thompson
> Sent: Friday, February 14, 2020 09:57
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: Re: [gpfsug-discuss] naive question about rsync: run it on a client or on NSD server?
>
> This message was sent by an external party.
>
>
> I wouldn't run it on an NSD server. Ideally you want to avoid running other processes etc on there.
>
> If you are running on clients, you also might want to look at: https://github.com/hpc/mpifileutils
>
> And use MPI to parallelise the find and copy.
>
> Simon
>
> ?On 14/02/2020, 14:25, "gpfsug-discuss-bounces at spectrumscale.org on behalf of giovanni.bracco at enea.it" <gpfsug-discuss-bounces at spectrumscale.org on behalf of giovanni.bracco at enea.it> wrote:
>
> We must replicate about 100 TB data between two filesystems supported by
> two different storages (DDN9900 and DDN7990) both connected to the same
> NSD servers (6 of them) and we plan to use rsync.
>
> Non special GPFS attributes, just the standard POSIX one, we plan to use
> the standard rsync.
>
> The question:
> is there any advantage in running the rsync on one of the NSD server or
> is better to run it on a client?
>
> The environment:
> GPFS 4.2.3.19, NSD CentOS7.4, clients mostly CentOS6.4 (connected by IB
> QDR) and CentOS7.3 (connected by OPA), connection between NSD and
> storage with IB QDR)
>
> Giovanni
>
> --
> Giovanni Bracco
> phone +39 351 8804788
> E-mail giovanni.bracco at enea.it
> WWW http://www.afs.enea.it/bracco
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
> ------------------------------
>
> Message: 4
> Date: Fri, 14 Feb 2020 16:13:30 +0000
> From: "Wahl, Edward" <ewahl at osc.edu>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Subject: Re: [gpfsug-discuss] naive question about rsync: run it on a
> client or on NSD server?
> Message-ID:
> <BN8PR01MB542598BE5F1D3701851E52F0A8150 at BN8PR01MB5425.prod.exchangelabs.com>
>
> Content-Type: text/plain; charset="us-ascii"
>
> Disregarding all the other reasons not to run it on the NSDs, many years of rsync on GPFS has shown us it is ALWAYS faster from clients with reasonable networks and no other overhead.
>
> Ed
>
> -----Original Message-----
> From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of Giovanni Bracco
> Sent: Friday, February 14, 2020 8:25 AM
> To: gpfsug-discuss at spectrumscale.org
> Subject: [gpfsug-discuss] naive question about rsync: run it on a client or on NSD server?
>
> We must replicate about 100 TB data between two filesystems supported by two different storages (DDN9900 and DDN7990) both connected to the same NSD servers (6 of them) and we plan to use rsync.
>
> Non special GPFS attributes, just the standard POSIX one, we plan to use the standard rsync.
>
> The question:
> is there any advantage in running the rsync on one of the NSD server or is better to run it on a client?
>
> The environment:
> GPFS 4.2.3.19, NSD CentOS7.4, clients mostly CentOS6.4 (connected by IB
> QDR) and CentOS7.3 (connected by OPA), connection between NSD and storage with IB QDR)
>
> Giovanni
>
> --
> Giovanni Bracco
> phone +39 351 8804788
> E-mail giovanni.bracco at enea.it
> WWW https://urldefense.com/v3/__http://www.afs.enea.it/bracco__;!!KGKeukY!g5RuD3fGuhmAJMIOdC_LgW0sNdejJCxdMTaLQfVtFcySDF1pkEvsTgu9tB2V$
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug-discuss__;!!KGKeukY!g5RuD3fGuhmAJMIOdC_LgW0sNdejJCxdMTaLQfVtFcySDF1pkEvsTn2QwFQn$
>
>
> ------------------------------
>
> Message: 5
> Date: Fri, 14 Feb 2020 12:28:27 -0500
> From: "Valdis Kl=?utf-8?Q?=c4=93?=tnieks" <valdis.kletnieks at vt.edu>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Cc: Marc A Kaplan <makaplan at us.ibm.com>
> Subject: Re: [gpfsug-discuss] mmbackup [--tsm-servers TSMServer[,
> TSMServer...]]
> Message-ID: <61512.1581701307 at turing-police>
> Content-Type: text/plain; charset="utf-8"
>
> On Tue, 11 Feb 2020 16:44:07 -0500, Jaime Pinto said:
>
>> # /usr/lpp/mmfs/bin/mmbackup /gpfs/fs1/home -N tapenode3-ib ??tsm?servers TAPENODE3,TAPENODE4 -s /dev/shm --tsm-errorlog $tmpDir/home-tsm-errorlog
>
> I got bit by this when cut-n-pasting from IBM documentation - the problem is that
> the web version has characters that *look* like the command-line hyphen character
> but are actually something different.
>
> It's the same problem as cut-n-pasting a command line where the command
> *should* have the standard ascii double-quote, but the webpage has "smart quotes"
> where there's different open and close quote characters. Just even less visually
> obvious...
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: not available
> Type: application/pgp-signature
> Size: 832 bytes
> Desc: not available
> URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20200214/e770e8e2/attachment.sig>
>
> ------------------------------
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
> End of gpfsug-discuss Digest, Vol 97, Issue 12
> **********************************************
More information about the gpfsug-discuss
mailing list