[gpfsug-discuss] Migrate/syncronize data from Isilon to Scale over NFS?

Jan-Frode Myklebust janfrode at tanso.net
Tue Nov 17 12:07:30 GMT 2020


Nice to see it working well!

But, what about ACLs? Does you rsync pull in all needed metadata, or do you
also need to sync ACLs ? Any plans for how to solve that ?

On Tue, Nov 17, 2020 at 12:52 PM Andi Christiansen <andi at christiansen.xxx>
wrote:

> Hi all,
>
> thanks for all the information, there was some interesting things amount
> it..
>
> I kept on going with rsync and ended up making a file with all top level
> user directories and splitting them into chunks of 347 per rsync
> session(total 42000 ish folders). yesterday we had only 14 sessions with
> 3000 folders in each and that was too much work for one rsync session..
>
> i divided them out among all GPFS nodes to have them fetch an area each
> and actually doing that 3 times on each node and that has now boosted the
> bandwidth usage from 3Gbit to around 16Gbit in total..
>
> all nodes have been seing doing work above 7Gbit individual which is
> actually near to what i was expecting without any modifications to the NFS
> server or TCP tuning..
>
> CPU is around 30-50% on each server and mostly below or around 30% so it
> seems like it could have handled abit more sessions..
>
> Small files are really a killer but with all 96+ sessions we have now its
> not often all sessions are handling small files at the same time so we have
> an average of about 10-12Gbit bandwidth usage.
>
> Thanks all! ill keep you in mind if for some reason we see it slowing down
> again but for now i think we will try to see if it will go the last mile
> with a bit more sessions on each :)
>
> Best Regards
> Andi Christiansen
>
> > On 11/17/2020 9:57 AM Uwe Falke <uwefalke at de.ibm.com> wrote:
> >
> >
> > Hi, Andi, sorry I just took your 20Gbit for the sign of 2x10Gbps bons,
> but
> > it is over two nodes, so no bonding. But still, I'd expect to open
> several
> > TCP connections in parallel per source-target pair  (like with several
> > rsyncs per source node) would bear an advantage (and still I thing NFS
> > doesn't do that, but I can be wrong).
> > If more nodes have access to the Isilon data they could also participate
> > (and don't need NFS exports for that).
> >
> > Mit freundlichen Grüßen / Kind regards
> >
> > Dr. Uwe Falke
> > IT Specialist
> > Hybrid Cloud Infrastructure / Technology Consulting & Implementation
> > Services
> > +49 175 575 2877 Mobile
> > Rathausstr. 7, 09111 Chemnitz, Germany
> > uwefalke at de.ibm.com
> >
> > IBM Services
> >
> > IBM Data Privacy Statement
> >
> > IBM Deutschland Business & Technology Services GmbH
> > Geschäftsführung: Sven Schooss, Stefan Hierl
> > Sitz der Gesellschaft: Ehningen
> > Registergericht: Amtsgericht Stuttgart, HRB 17122
> >
> >
> >
> > From:   Uwe Falke/Germany/IBM
> > To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> > Date:   17/11/2020 09:50
> > Subject:        Re: [EXTERNAL] [gpfsug-discuss] Migrate/syncronize data
> > from Isilon to Scale over       NFS?
> >
> >
> > Hi Andi,
> >
> > what about leaving NFS completeley out and using rsync  (multiple rsyncs
> > in parallel, of course) directly between your source and target servers?
> > I am not sure how many TCP connections (suppose it is NFS4) in parallel
> > are opened between client and server, using a 2x bonded interface well
> > requires at least two.  That combined with the DB approach suggested by
> > Jonathan to control the activity of the rsync streams would be my best
> > guess.
> > If you have many small files, the overhead might still kill you. Tarring
> > them up into larger aggregates for transfer would help a lot, but then
> you
> > must be sure they won't change or you need to implement your own version
> > control for that class of files.
> >
> > Mit freundlichen Grüßen / Kind regards
> >
> > Dr. Uwe Falke
> > IT Specialist
> > Hybrid Cloud Infrastructure / Technology Consulting & Implementation
> > Services
> > +49 175 575 2877 Mobile
> > Rathausstr. 7, 09111 Chemnitz, Germany
> > uwefalke at de.ibm.com
> >
> > IBM Services
> >
> > IBM Data Privacy Statement
> >
> > IBM Deutschland Business & Technology Services GmbH
> > Geschäftsführung: Sven Schooss, Stefan Hierl
> > Sitz der Gesellschaft: Ehningen
> > Registergericht: Amtsgericht Stuttgart, HRB 17122
> >
> >
> >
> >
> > From:   Andi Christiansen <andi at christiansen.xxx>
> > To:     "gpfsug-discuss at spectrumscale.org"
> > <gpfsug-discuss at spectrumscale.org>
> > Date:   16/11/2020 20:44
> > Subject:        [EXTERNAL] [gpfsug-discuss] Migrate/syncronize data from
> > Isilon to Scale over    NFS?
> > Sent by:        gpfsug-discuss-bounces at spectrumscale.org
> >
> >
> >
> > Hi all,
> >
> > i have got a case where a customer wants 700TB migrated from isilon to
> > Scale and the only way for him is exporting the same directory on NFS
> from
> > two different nodes...
> >
> > as of now we are using multiple rsync processes on different parts of
> > folders within the main directory. this is really slow and will take
> > forever.. right now 14 rsync processes spread across 3 nodes fetching
> from
> > 2..
> >
> > does anyone know of a way to speed it up? right now we see from 1Gbit to
> > 3Gbit if we are lucky(total bandwidth) and there is a total of 30Gbit
> from
> > scale nodes and 20Gbits from isilon so we should be able to reach just
> > under 20Gbit...
> >
> >
> > if anyone have any ideas they are welcome!
> >
> >
> > Thanks in advance
> > Andi Christiansen _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org
> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org
> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20201117/1fba22bb/attachment-0002.htm>


More information about the gpfsug-discuss mailing list