[gpfsug-discuss] GPFS cpoying huge batches of files from one stgpool to another- how do you do it?
Jez Tucker
Jez.Tucker at rushes.co.uk
Thu Mar 22 09:27:56 GMT 2012
Here's a quick script to use GNU parallel:
We saturated 8 Gb FC with this.
#!/bin/sh
DATADIR=/mnt/gpfs/srcfldr
DESTDIR=/mnt/gpfs/destinationfldr
while read LINE; do
PROJ=$(echo $LINE | awk '{ print $1; }');
DESTFLDR=$(echo $LINE | awk '{ print $2; }');
echo "$PROJ -> $DEST";
mkdir -p "$DESTDIR/$DESTFLDR";
find $PROJ/ | parallel cp --parents -puv "{}" "$DESTDIR/$DESTFLDR/";
rsync -av $PROJ "$DESTDIR/$DESTFLDR/" 2>&1 > RESTORE_LOGS/$PROJ.restore.log;
done < restore.my.projectlist
This assumes restore.nsd01.projectlist contains something such as:
<data folder> <destinationfldr subdir>
<data folder> <destinationfldr subdir>
...
> -----Original Message-----
> From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-
> bounces at gpfsug.org] On Behalf Of Orlando Richards
> Sent: 20 March 2012 09:09
> To: gpfsug-discuss at gpfsug.org
> Subject: Re: [gpfsug-discuss] GPFS cpoying huge batches of files from one
> stgpool to another- how do you do it?
>
> We don't do this often, so when we do we tend to do a "noddy"
> parallelisation of rsync - manually divvy the folders up and spawn
> multiple threads.
>
> We might have a requirement soon(ish) to do this on a much larger scale
> though, with virtually no interruption to service - so I'm very keen to
> see what how the AFM solution looks, since this should allow us to
> present a continual and single view into the filesystem, whilst
> migrating it all in the background to the new filesystem, with just a
> brief wobble whilst we flip between the old and new views.
>
>
> On 20/03/12 00:47, Jez Tucker wrote:
> > In answer to my own question ...
> >
> > http://www.gnu.org/software/parallel
> >
> >
> http://www.gnu.org/software/parallel/man.html#example__parallelizing_r
> sync
> >
> > Or
> >
> > http://code.google.com/p/parallel-ssh/ (parallel versions of rsync etc).
> >
> > One for the bookmarks... hopefully you'll find it useful.
> >
> > Jez
> >
> > *From:*gpfsug-discuss-bounces at gpfsug.org
> > [mailto:gpfsug-discuss-bounces at gpfsug.org] *On Behalf Of *Jez Tucker
> > *Sent:* 19 March 2012 23:36
> > *To:* gpfsug main discussion list
> > *Subject:* [gpfsug-discuss] GPFS cpoying huge batches of files from one
> > stgpool to another- how do you do it?
> >
> > Hello
> >
> > Just wondering how other people go about copying loads of files in a
> > many, many deep directory path from one file system to another. Assume
> > filenames are full UNICODE and can contain almost any character.
> >
> > r to use.
> >
> >
> > Rushes Postproduction Limited, 66 Old Compton Street, London W1D 4UH
> > tel: +44 (0)20 7437 8676
> > web: http://www.rushes.co.uk
> > The information contained in this e-mail is confidential and may be
> > subject to legal privilege. If you are not the intended recipient, you
> > must not use, copy, distribute or disclose the e-mail or any part of its
> > contents or take any action in reliance on it. If you have received this
> > e-mail in error, please e-mail the sender by replying to this message.
> > All reasonable precautions have been taken to ensure no viruses are
> > present in this e-mail. Rushes Postproduction Limited cannot accept
> > responsibility for loss or damage arising from the use of this e-mail or
> > attachments and recommend that you subject these to your virus checking
> > procedures prior to use.
> >
> >
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at gpfsug.org
> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
> --
> --
> Dr Orlando Richards
> Information Services
> IT Infrastructure Division
> Unix Section
> Tel: 0131 650 4994
>
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Rushes Postproduction Limited, 66 Old Compton Street, London W1D 4UH
tel: +44 (0)20 7437 8676
web: http://www.rushes.co.uk
The information contained in this e-mail is confidential and may be subject to legal privilege. If you are not the intended recipient, you must not use, copy, distribute or disclose the e-mail or any part of its contents or take any action in reliance on it. If you have received this e-mail in error, please e-mail the sender by replying to this message. All reasonable precautions have been taken to ensure no viruses are present in this e-mail. Rushes Postproduction Limited cannot accept responsibility for loss or damage arising from the use of this e-mail or attachments and recommend that you subject these to your virus checking procedures prior to use.
More information about the gpfsug-discuss
mailing list