<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">This is not the answer to not writing it yourself:<div class=""><br class=""></div><div class="">However, be aware that GNU xargs has the -P <i class="">x</i> option, which will try to keep <i class="">x</i> batches running. It’s a good way to optimize the number of threads for anything you’re multiprocessing in the shell. So you can build a list and have xargs fork <i class="">x</i> copies of rsync or cp at a time (with -n <i class="">y</i> items in each batch). Not waiting to start the next batch when one finishes can add up to lots of MB*s very quickly.<div class=""><br class=""></div><div class=""><br class=""></div><div class="">This is not the answer to anything, and is probably a waste of your time:</div><div class=""><br class=""></div><div class="">I started to comment that if GPFS did provide such a “data path shortcut”, I think I’d want it to work between any two allocation areas — even two independent filesets in the same file system. Then I started working though the possibilities for just doing that… and it devolved into the realization that we’ve got to copy the data most of the time (unless it’s in the same filesystem *and* the same storage pool — and maybe even then depending on how the allocator works). Realizing that I decide that sometimes it just sucks to have data in the wrong (old) place. :)</div><div class=""><br class=""></div><div class="">Maybe what we want is to be able to split an independent fileset (if it is 1:1 with a storage pool) from a filesystem and graft it onto another one — that’s probably easier and it almost mirrors vgsplit, et al.</div><div class=""><br class=""></div><div class="">I should go do actual work...</div><div class=""><br class=""></div><div class="">Liberty,</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div><blockquote type="cite" class=""><div class="">On Dec 5, 2016, at 9:09 AM, Brian Marshall <<a href="mailto:mimarsh2@vt.edu" class="">mimarsh2@vt.edu</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">All,<div class=""><br class=""></div><div class="">I am in the same boat. I'd like to copy ~500 TB from one filesystem to another. Both are being served by the same NSD servers.</div><div class=""><br class=""></div><div class="">We've done the multiple rsync script method in the past (and yes it's a bit of a pain). Would love to have an easier utility. </div><div class=""><br class=""></div><div class="">Best,</div><div class="">Brian Marshall</div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Mon, Dec 5, 2016 at 5:26 AM, Heiner Billich <span dir="ltr" class=""><<a href="mailto:Heiner.Billich@psi.ch" target="_blank" class="">Heiner.Billich@psi.ch</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello,<br class="">
<br class="">
I heard about some gpfs optimized bulk(?) copy command named 'mmcp' or 'mmcopy' but couldn't find it in either /user/lpp/mmfs/samples/ or by asking google. Can please somebody point me to the source? I wonder whether it allows incremental copies as rsync does.<br class="">
<br class="">
We need to copy a few 100TB of data and simple rsync provides just about 100MB/s. I know about the possible workarounds - write a wrapper script, run several rsyncs in parallel, distribute the rsync jobs on several nodes, use a special rsync versions that knows about gpfs ACLs, ... or try mmfind, which requires me to write a custom wrapper for cp ....<br class="">
<br class="">
I really would prefer some ready-to-use script or program.<br class="">
<br class="">
Thank you and kind regards,<br class="">
Heiner Billich<br class="">
______________________________<wbr class="">_________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="http://spectrumscale.org/" rel="noreferrer" target="_blank" class="">spectrumscale.org</a><br class="">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank" class="">http://gpfsug.org/mailman/list<wbr class="">info/gpfsug-discuss</a><br class="">
</blockquote></div><br class=""></div>
_______________________________________________<br class="">gpfsug-discuss mailing list<br class="">gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a><br class=""><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class=""></div></blockquote></div><br class=""><div id="com.littleknownsoftware.SigProSignature" class=""><div class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">-- <br class="">Stephen<br class=""><br class=""><br class=""></div></div>
</div></div></div></div></body></html>