[gpfsug-discuss] GPFS cpoying huge batches of files from one stgpool to another- how do you do it?

[Jez Tucker]

Apologies should have read:

'This assumes restore.my.projectlist contains'..

> -----Original Message-----
> From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-
> bounces at gpfsug.org] On Behalf Of Jez Tucker
> Sent: 22 March 2012 09:28
> To: gpfsug main discussion list
> Subject: Re: [gpfsug-discuss] GPFS cpoying huge batches of files from one
> stgpool to another- how do you do it?
> 
> Here's a quick script to use GNU parallel:
> 
> We saturated 8 Gb FC with this.
> 
> 
> #!/bin/sh
> 
> DATADIR=/mnt/gpfs/srcfldr
> DESTDIR=/mnt/gpfs/destinationfldr
> 
> while read LINE; do
> 
>   PROJ=$(echo $LINE | awk '{ print $1; }');
>   DESTFLDR=$(echo $LINE | awk '{ print $2; }');
> 
>   echo "$PROJ -> $DEST";
>   mkdir -p "$DESTDIR/$DESTFLDR";
> 
>   find $PROJ/ | parallel  cp --parents -puv "{}" "$DESTDIR/$DESTFLDR/";
>   rsync -av $PROJ "$DESTDIR/$DESTFLDR/" 2>&1 >
> RESTORE_LOGS/$PROJ.restore.log;
> 
> done < restore.my.projectlist
> 
> 
> This assumes restore.nsd01.projectlist contains something such as:
> 
> <data folder> <destinationfldr subdir>
> <data folder> <destinationfldr  subdir>
> ...
> 
> 
> 
> > -----Original Message-----
> > From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-
> > bounces at gpfsug.org] On Behalf Of Orlando Richards
> > Sent: 20 March 2012 09:09
> > To: gpfsug-discuss at gpfsug.org
> > Subject: Re: [gpfsug-discuss] GPFS cpoying huge batches of files from
> > one stgpool to another- how do you do it?
> >
> > We don't do this often, so when we do we tend to do a "noddy"
> > parallelisation of rsync - manually divvy the folders up and spawn
> > multiple threads.
> >
> > We might have a requirement soon(ish) to do this on a much larger
> > scale though, with virtually no interruption to service - so I'm very
> > keen to see what how the AFM solution looks, since this should allow
> > us to present a continual and single view into the filesystem, whilst
> > migrating it all in the background to the new filesystem, with just a
> > brief wobble whilst we flip between the old and new views.
> >
> >
> > On 20/03/12 00:47, Jez Tucker wrote:
> > > In answer to my own question ...
> > >
> > > http://www.gnu.org/software/parallel
> > >
> > >
> >
> http://www.gnu.org/software/parallel/man.html#example__parallelizing_r
> > sync
> > >
> > > Or
> > >
> > > http://code.google.com/p/parallel-ssh/ (parallel versions of rsync etc).
> > >
> > > One for the bookmarks... hopefully you'll find it useful.
> > >
> > > Jez
> > >
> > > *From:*gpfsug-discuss-bounces at gpfsug.org
> > > [mailto:gpfsug-discuss-bounces at gpfsug.org] *On Behalf Of *Jez Tucker
> > > *Sent:* 19 March 2012 23:36
> > > *To:* gpfsug main discussion list
> > > *Subject:* [gpfsug-discuss] GPFS cpoying huge batches of files from
> > > one stgpool to another- how do you do it?
> > >
> > > Hello
> > >
> > > Just wondering how other people go about copying loads of files in a
> > > many, many deep directory path from one file system to another.
> > > Assume filenames are full UNICODE and can contain almost any
> character.
> > >
> > > r to use.
> > >
> > >
> > > Rushes Postproduction Limited, 66 Old Compton Street, London W1D
> 4UH
> > > tel: +44 (0)20 7437 8676
> > > web: http://www.rushes.co.uk
> > > The information contained in this e-mail is confidential and may be
> > > subject to legal privilege. If you are not the intended recipient,
> > > you must not use, copy, distribute or disclose the e-mail or any
> > > part of its contents or take any action in reliance on it. If you
> > > have received this e-mail in error, please e-mail the sender by replying
> to this message.
> > > All reasonable precautions have been taken to ensure no viruses are
> > > present in this e-mail. Rushes Postproduction Limited cannot accept
> > > responsibility for loss or damage arising from the use of this
> > > e-mail or attachments and recommend that you subject these to your
> > > virus checking procedures prior to use.
> > >
> > >
> > > _______________________________________________
> > > gpfsug-discuss mailing list
> > > gpfsug-discuss at gpfsug.org
> > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> >
> >
> > --
> >              --
> >     Dr Orlando Richards
> >    Information Services
> > IT Infrastructure Division
> >         Unix Section
> >      Tel: 0131 650 4994
> >
> > The University of Edinburgh is a charitable body, registered in
> > Scotland, with registration number SC005336.
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at gpfsug.org
> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 
> Rushes Postproduction Limited, 66 Old Compton Street, London W1D 4UH
> tel: +44 (0)20 7437 8676
> web: http://www.rushes.co.uk
> The information contained in this e-mail is confidential and may be subject
> to legal privilege. If you are not the intended recipient, you must not use,
> copy, distribute or disclose the e-mail or any part of its contents or take any
> action in reliance on it. If you have received this e-mail in error, please e-
> mail the sender by replying to this message. All reasonable precautions
> have been taken to ensure no viruses are present in this e-mail. Rushes
> Postproduction Limited cannot accept responsibility for loss or damage
> arising from the use of this e-mail or attachments and recommend that you
> subject these to your virus checking procedures prior to use.
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Rushes Postproduction Limited, 66 Old Compton Street, London W1D 4UH
tel: +44 (0)20 7437 8676
web: http://www.rushes.co.uk
The information contained in this e-mail is confidential and may be subject to legal privilege. If you are not the intended recipient, you must not use, copy, distribute or disclose the e-mail or any part of its contents or take any action in reliance on it. If you have received this e-mail in error, please e-mail the sender by replying to this message. All reasonable precautions have been taken to ensure no viruses are present in this e-mail. Rushes Postproduction Limited cannot accept responsibility for loss or damage arising from the use of this e-mail or attachments and recommend that you subject these to your virus checking procedures prior to use.