<div><div dir="auto">It may be overkill for your use case but MPI file utils is very good for large datasets. </div></div><div dir="auto"><br></div><div dir="auto"><div><a href="https://github.com/hpc/mpifileutils">https://github.com/hpc/mpifileutils</a></div></div><div dir="auto"><br></div><div dir="auto">Cheers,</div><div dir="auto"><br></div><div dir="auto">Carl. </div><div><br><div class="gmail_quote"><div dir="ltr">On Fri, 19 Oct 2018 at 7:05 am, <<a href="mailto:Dwayne.Hart@med.mun.ca">Dwayne.Hart@med.mun.ca</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Thank you all for the responses. I'm currently using msrsync and things appear to be going very well.<br>
<br>
The data transfer is contained inside our DC. I'm transferring a user's home directory content from one GPFS file system to another. Our IBM Spectrum Scale Solution consists of 12 IO nodes connected to IB and the client node that I'm transferring the data from one fs to another is also connected to IB with a possible maximum of 2 hops. <br>
<br>
[root@client-system]# /gpfs/home/dwayne/bin/msrsync -P --stats -p 32 /gpfs/home/user/ /research/project/user/<br>
[64756/992397 entries] [30.1 T/239.6 T transferred] [81 entries/s] [39.0 G/s bw] [monq 0] [jq 62043]<br>
<br>
Best,<br>
Dwayne<br>
<br>
-----Original Message-----<br>
From: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a> [mailto:<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a>] On Behalf Of Christopher Black<br>
Sent: Thursday, October 18, 2018 4:43 PM<br>
To: gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a>><br>
Subject: Re: [gpfsug-discuss] Best way to migrate data<br>
<br>
Other tools and approaches that we've found helpful:<br>
msrsync: handles parallelizing rsync within a dir tree and can greatly speed up transfers on a single node with both filesystems mounted, especially when dealing with many small files<br>
Globus/GridFTP: set up one or more endpoints on each side, gridftp will auto parallelize and recover from disruptions<br>
<br>
msrsync is easier to get going but is limited to one parent dir per node. We've sometimes done an additional level of parallelization by running msrsync with different top level directories on different hpc nodes simultaneously.<br>
<br>
Best,<br>
Chris<br>
<br>
Refs:<br>
<a href="https://github.com/jbd/msrsync" rel="noreferrer" target="_blank">https://github.com/jbd/msrsync</a><br>
<a href="https://www.globus.org/" rel="noreferrer" target="_blank">https://www.globus.org/</a><br>
<br>
On 10/18/18, 2:54 PM, "<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a> on behalf of Sanchez, Paul" <<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a> on behalf of <a href="mailto:Paul.Sanchez@deshaw.com" target="_blank">Paul.Sanchez@deshaw.com</a>> wrote:<br>
<br>
Sharding can also work, if you have a storage-connected compute grid in your environment: If you enumerate all of the directories, then use a non-recursive rsync for each one, you may be able to parallelize the workload by using several clients simultaneously. It may still max out the links of these clients (assuming your source read throughput and target write throughput bottlenecks aren't encountered first) but it may run that way for 1/100th of the time if you can use 100+ machines.<br>
<br>
-Paul<br>
-----Original Message-----<br>
From: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a> <<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a>> On Behalf Of Buterbaugh, Kevin L<br>
Sent: Thursday, October 18, 2018 2:26 PM<br>
To: gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a>><br>
Subject: Re: [gpfsug-discuss] Best way to migrate data<br>
<br>
Hi Dwayne,<br>
<br>
I’m assuming you can’t just let an rsync run, possibly throttled in some way? If not, and if you’re just tapping out your network, then would it be possible to go old school? We have parts of the Medical Center here where their network connections are … um, less than robust. So they tar stuff up to a portable HD, sneaker net it to us, and we untar is from an NSD server.<br>
<br>
HTH, and I really hope that someone has a better idea than that!<br>
<br>
Kevin<br>
<br>
> On Oct 18, 2018, at 12:19 PM, <a href="mailto:Dwayne.Hart@med.mun.ca" target="_blank">Dwayne.Hart@med.mun.ca</a> wrote:<br>
><br>
> Hi,<br>
><br>
> Just wondering what the best recipe for migrating a user’s home directory content from one GFPS file system to another which hosts a larger research GPFS file system? I’m currently using rsync and it has maxed out the client system’s IB interface.<br>
><br>
> Best,<br>
> Dwayne<br>
> —<br>
> Dwayne Hart | Systems Administrator IV<br>
><br>
> CHIA, Faculty of Medicine<br>
> Memorial University of Newfoundland<br>
> 300 Prince Philip Drive<br>
> St. John’s, Newfoundland | A1B 3V6<br>
> Craig L Dobbin Building | 4M409<br>
> T 709 864 6631<br>
> _______________________________________________<br>
> gpfsug-discuss mailing list<br>
> gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
> <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253A-252F-252Fgpfsug.org-252Fmailman-252Flistinfo-252Fgpfsug-2Ddiscuss-26amp-3Bdata-3D02-257C01-257CKevin.Buterbaugh-2540vanderbilt.edu-257Ccca728d2d61f4be06bcd08d6351f3650-257Cba5a7f39e3be4ab3b45067fa80faecad-257C0-257C0-257C636754805507359478-26amp-3Bsdata-3D2YAiqgqKl4CerlyCn3vJ9v9u-252FrGzbfa7aKxJ0PYV-252Fhc-253D-26amp-3Breserved-3D0&d=DwIGaQ&c=C9X8xNkG_lwP_-eFHTGejw&r=DopWM-bvfskhBn2zeglfyyw5U2pumni6m_QzQFYFepU&m=e-U5zXflwxr0w9-5ia0FHn3tF1rwmM1qciZNrBLwFeg&s=NVJncSq-SKJSPgljdYqLDoy753jhxiKJNI2M8CexJME&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253A-252F-252Fgpfsug.org-252Fmailman-252Flistinfo-252Fgpfsug-2Ddiscuss-26amp-3Bdata-3D02-257C01-257CKevin.Buterbaugh-2540vanderbilt.edu-257Ccca728d2d61f4be06bcd08d6351f3650-257Cba5a7f39e3be4ab3b45067fa80faecad-257C0-257C0-257C636754805507359478-26amp-3Bsdata-3D2YAiqgqKl4CerlyCn3vJ9v9u-252FrGzbfa7aKxJ0PYV-252Fhc-253D-26amp-3Breserved-3D0&d=DwIGaQ&c=C9X8xNkG_lwP_-eFHTGejw&r=DopWM-bvfskhBn2zeglfyyw5U2pumni6m_QzQFYFepU&m=e-U5zXflwxr0w9-5ia0FHn3tF1rwmM1qciZNrBLwFeg&s=NVJncSq-SKJSPgljdYqLDoy753jhxiKJNI2M8CexJME&e=</a><br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=C9X8xNkG_lwP_-eFHTGejw&r=DopWM-bvfskhBn2zeglfyyw5U2pumni6m_QzQFYFepU&m=e-U5zXflwxr0w9-5ia0FHn3tF1rwmM1qciZNrBLwFeg&s=oM0Uo8pPSV5bUj2Hyjzvw1q12Oug_mH-aYsM_R4Zfv4&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=C9X8xNkG_lwP_-eFHTGejw&r=DopWM-bvfskhBn2zeglfyyw5U2pumni6m_QzQFYFepU&m=e-U5zXflwxr0w9-5ia0FHn3tF1rwmM1qciZNrBLwFeg&s=oM0Uo8pPSV5bUj2Hyjzvw1q12Oug_mH-aYsM_R4Zfv4&e=</a><br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=C9X8xNkG_lwP_-eFHTGejw&r=DopWM-bvfskhBn2zeglfyyw5U2pumni6m_QzQFYFepU&m=e-U5zXflwxr0w9-5ia0FHn3tF1rwmM1qciZNrBLwFeg&s=oM0Uo8pPSV5bUj2Hyjzvw1q12Oug_mH-aYsM_R4Zfv4&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=C9X8xNkG_lwP_-eFHTGejw&r=DopWM-bvfskhBn2zeglfyyw5U2pumni6m_QzQFYFepU&m=e-U5zXflwxr0w9-5ia0FHn3tF1rwmM1qciZNrBLwFeg&s=oM0Uo8pPSV5bUj2Hyjzvw1q12Oug_mH-aYsM_R4Zfv4&e=</a><br>
<br>
<br>
________________________________<br>
<br>
This message is for the recipient’s use only, and may contain confidential, privileged or protected information. Any unauthorized use or dissemination of this communication is prohibited. If you received this message in error, please immediately notify the sender and destroy all copies of this message. The recipient should check this email and any attachments for the presence of viruses, as we accept no liability for any damage caused by any virus transmitted by this email.<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
</blockquote></div></div>