[gpfsug-discuss] gpfsug-discuss Digest, Vol 106, Issue 21
Cal Sawyer
cal.sawyer at framestore.com
Wed Nov 18 12:18:57 GMT 2020
Hello
Not a Scale user per se (we run a 3rdparty offshoot of Scale). In a past
life managing Nexenta with OpenSolaris DR storage, I used nc/netcat for
bulk data sync, which is far more efficient than rsync. With a bit of
planning and analysis of directory structure on the target, nc runs could
be parallelised as well, although not quite in the same way as running
rsync via parallels. Of course, nc has to be available on Isilon but i have
no experience with that platform. The only caveat in using nc is the amount
of change to the target data as copying progresses (is the target datastore
static or still seeing changes?). nc has to be followed with rsync to apply
any changes and/or verify the integrity of the bulk copy.
https://nakkaya.com/2009/04/15/using-netcat-for-file-transfers/
Are your Isilon and Scale systems located in the same network space?
I'd also suggest that if possible, add a quad-port 10GbE (or larger:
25/100GbE) NIC to your servers to gain a wider data path and conduct your
copy operations on those interfaces
regards
[image: Framestore]
Cal Sawyer · Senior Systems Engineer London · New York · Los Angeles ·
Chicago · Montréal · Mumbai
28 Chancery Lane
London WC2A 1LB
[T] +44 (0)20 7344 8000
W3W: warm.soil.patio
On Wed, 18 Nov 2020 at 12:00, <gpfsug-discuss-request at spectrumscale.org>
wrote:
> Send gpfsug-discuss mailing list submissions to
> gpfsug-discuss at spectrumscale.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> or, via email, send a message with subject or body 'help' to
> gpfsug-discuss-request at spectrumscale.org
>
> You can reach the person managing the list at
> gpfsug-discuss-owner at spectrumscale.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of gpfsug-discuss digest..."
>
>
> Today's Topics:
>
> 1. Re: Migrate/syncronize data from Isilon to Scale over NFS?
> (Chris Schlipalius)
> 2. Re: Migrate/syncronize data from Isilon to Scale over NFS?
> (Jonathan Buzzard)
> 3. Re: Migrate/syncronize data from Isilon to Scale over NFS?
> (Andi Christiansen)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Wed, 18 Nov 2020 07:17:18 +0800
> From: Chris Schlipalius <chris.schlipalius at pawsey.org.au>
> To: <gpfsug-discuss at spectrumscale.org>
> Subject: Re: [gpfsug-discuss] Migrate/syncronize data from Isilon to
> Scale over NFS?
> Message-ID: <578BE691-DEE4-43AC-97D2-546AC406E14A at pawsey.org.au>
> Content-Type: text/plain; charset="utf-8"
>
> So at my last job we used to rsync data between isilons across campus, and
> isilon to Windows File Cluster (and back).
>
> I recommend using dry run to generate a list of files and then use this to
> run with rysnc.
>
> This allows you also to be able to break up the transfer into batches, and
> check if files have changed before sync (say if your isilon files are not
> RO.
>
> Also ensure you have a recent version of rsync that preserves extended
> attributes and check your ACLS.
>
>
>
> A dry run example:
>
> https://unix.stackexchange.com/a/261372
>
>
>
> I always felt more comfortable having a list of files before a sync?.
>
>
>
>
>
>
>
> Regards,
>
> Chris Schlipalius
>
>
>
> Team Lead, Data Storage Infrastructure, Supercomputing Platforms, Pawsey
> Supercomputing Centre (CSIRO)
>
> 1 Bryce Avenue
>
> Kensington WA 6151
>
> Australia
>
>
>
> Tel +61 8 6436 8815
>
> Email chris.schlipalius at pawsey.org.au
>
> Web www.pawsey.org.au
>
>
>
>
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20201118/c99c2fb1/attachment-0001.html
> >
>
> ------------------------------
>
> Message: 2
> Date: Wed, 18 Nov 2020 11:48:52 +0000
> From: Jonathan Buzzard <jonathan.buzzard at strath.ac.uk>
> To: gpfsug-discuss at spectrumscale.org
> Subject: Re: [gpfsug-discuss] Migrate/syncronize data from Isilon to
> Scale over NFS?
> Message-ID: <7983810e-f51c-8cf7-a750-5c3285870bd4 at strath.ac.uk>
> Content-Type: text/plain; charset=utf-8; format=flowed
>
> On 17/11/2020 23:17, Chris Schlipalius wrote:
> > So at my last job we used to rsync data between isilons across campus,
> > and isilon to Windows File Cluster (and back).
> >
> > I recommend using dry run to generate a list of files and then use this
> > to run with rysnc.
> >
> > This allows you also to be able to break up the transfer into batches,
> > and check if files have changed before sync (say if your isilon files
> > are not RO.
> >
> > Also ensure you have a recent version of rsync that preserves extended
> > attributes and check your ACLS.
> >
> > A dry run example:
> >
> > https://unix.stackexchange.com/a/261372
> >
> > I always felt more comfortable having a list of files before a sync?.
> >
>
> I would counsel in the strongest possible terms against that approach.
>
> Basically you have to be assured that none of your file names have
> "wacky" characters in them, because handling "wacky" characters in file
> names is exceedingly difficult. I cannot stress how hard it is and the
> above example does not handle all "wacky" characters in file names.
>
> So what do I mean by "wacky" characters. Well remember a file name can
> have just about anything in it on Linux with the exception of '/', and
> users especially when using a GUI, and even more so if they are Mac
> users can and do use what I will call "wacky" characters in their file
> names.
>
> The obvious ones are spaces, but it's not just ASCII 0x20, but tabs too.
> Then there is the use of the wildcard characters, especially '?' but
> also '*'.
>
> Not too difficult to handle you might say. Right now deal with a file
> name with a newline character in it :-) Don't ask me how or why you even
> do that but let me assure you that I have seen them on more than one
> occasion. And now your dry run list is broken...
>
> Not only that if you have a few hundred million files to move a list
> just becomes unwieldy anyway.
>
> One thing I didn't mention is that I would run anything with in a screen
> (or tmux if that is your poison) and turn on logging.
>
> For those interested I am in the process of cleaning up the script a bit
> and will post it somewhere in due course.
>
>
> JAB.
>
> --
> Jonathan A. Buzzard Tel: +44141-5483420
> HPC System Administrator, ARCHIE-WeSt.
> University of Strathclyde, John Anderson Building, Glasgow. G4 0NG
>
>
> ------------------------------
>
> Message: 3
> Date: Wed, 18 Nov 2020 12:54:47 +0100 (CET)
> From: Andi Christiansen <andi at christiansen.xxx>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>,
> Jonathan Buzzard <jonathan.buzzard at strath.ac.uk>
> Subject: Re: [gpfsug-discuss] Migrate/syncronize data from Isilon to
> Scale over NFS?
> Message-ID: <1947408989.293430.1605700487095 at privateemail.com>
> Content-Type: text/plain; charset=UTF-8
>
> Hi Jonathan,
>
> i would be very interested in seeing your scripts when they are posted.
> Let me know where to get them!
>
> Thanks a bunch!
> Andi Christiansen
>
> > On 11/18/2020 12:48 PM Jonathan Buzzard <jonathan.buzzard at strath.ac.uk>
> wrote:
> >
> >
> > On 17/11/2020 23:17, Chris Schlipalius wrote:
> > > So at my last job we used to rsync data between isilons across campus,
> > > and isilon to Windows File Cluster (and back).
> > >
> > > I recommend using dry run to generate a list of files and then use
> this
> > > to run with rysnc.
> > >
> > > This allows you also to be able to break up the transfer into batches,
> > > and check if files have changed before sync (say if your isilon files
> > > are not RO.
> > >
> > > Also ensure you have a recent version of rsync that preserves extended
> > > attributes and check your ACLS.
> > >
> > > A dry run example:
> > >
> > > https://unix.stackexchange.com/a/261372
> > >
> > > I always felt more comfortable having a list of files before a sync?.
> > >
> >
> > I would counsel in the strongest possible terms against that approach.
> >
> > Basically you have to be assured that none of your file names have
> > "wacky" characters in them, because handling "wacky" characters in file
> > names is exceedingly difficult. I cannot stress how hard it is and the
> > above example does not handle all "wacky" characters in file names.
> >
> > So what do I mean by "wacky" characters. Well remember a file name can
> > have just about anything in it on Linux with the exception of '/', and
> > users especially when using a GUI, and even more so if they are Mac
> > users can and do use what I will call "wacky" characters in their file
> > names.
> >
> > The obvious ones are spaces, but it's not just ASCII 0x20, but tabs too.
> > Then there is the use of the wildcard characters, especially '?' but
> > also '*'.
> >
> > Not too difficult to handle you might say. Right now deal with a file
> > name with a newline character in it :-) Don't ask me how or why you even
> > do that but let me assure you that I have seen them on more than one
> > occasion. And now your dry run list is broken...
> >
> > Not only that if you have a few hundred million files to move a list
> > just becomes unwieldy anyway.
> >
> > One thing I didn't mention is that I would run anything with in a screen
> > (or tmux if that is your poison) and turn on logging.
> >
> > For those interested I am in the process of cleaning up the script a bit
> > and will post it somewhere in due course.
> >
> >
> > JAB.
> >
> > --
> > Jonathan A. Buzzard Tel: +44141-5483420
> > HPC System Administrator, ARCHIE-WeSt.
> > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org
> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
> ------------------------------
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
> End of gpfsug-discuss Digest, Vol 106, Issue 21
> ***********************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20201118/2fc06f24/attachment-0001.htm>
More information about the gpfsug-discuss
mailing list