<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title></title>
</head>
<body>
<div name="messageBodySection" style="font-size: 14px; font-family: -apple-system, BlinkMacSystemFont, sans-serif;">Thank you Marc. I was just trying to suggest another approach to this email thread.
<div><br /></div>
<div>However i believe, we cannot run mmfind/mmapplypolicy with remote filesystems and can only be run on the owning cluster? In our clusters - All the gpfs clients are generally in there own compute clusters and mount filesystems from other storage clusters - which i thought is one of the recommended designs.</div>
<div>The scripts in the /usr/lpp/mmfs/samples/util folder do work with remote filesystems, and thus on the compute nodes.</div>
<div><br /></div>
<div>I was also trying to find something that could be used by users and not by superuser… but i guess none of these tools are meant to be run by a user without superuser privileges.</div>
</div>
<div name="messageSignatureSection" style="font-size: 14px; font-family: -apple-system, BlinkMacSystemFont, sans-serif;"><br />
Regards,<br />
Lohit</div>
<div name="messageReplySection" style="font-size: 14px; font-family: -apple-system, BlinkMacSystemFont, sans-serif;"><br />
On Mar 8, 2019, 3:54 PM -0600, Marc A Kaplan <makaplan@us.ibm.com>, wrote:<br />
<blockquote type="cite" style="margin: 5px 5px; padding-left: 10px; border-left: thin solid #1abc9c;"><span style="font-size:10pt;font-family:sans-serif">Lohit... Any and all of those commands and techniques should still work with newer version of GPFS.</span><br />
<span style="font-size:10pt;font-family:sans-serif"><br />
But mmapplypolicy is the supported command for generating file lists. It uses the GPFS APIs and some parallel processing tricks.</span><br />
<br />
<span style="font-size:10pt;font-family:sans-serif">mmfind is a script that make it easier to write GPFS "policy rules" and runs mmapplypolicy for you.</span><br />
<br />
<span style="font-size:10pt;font-family:sans-serif">mmxcp can be used with mmfind (and/or mmapplypolicy) to make it easy to run a cp (or other command) in parallel on those filelists ...<br /></span><br />
<span style="font-size:10pt;font-family:sans-serif">--marc K of GPFS</span><br />
<br />
<br />
<br />
<span style="font-size:9pt;color:#5f5f5f;font-family:sans-serif">From: </span><span style="font-size:9pt;font-family:sans-serif">valleru@cbio.mskcc.org</span><br />
<span style="font-size:9pt;color:#5f5f5f;font-family:sans-serif">To: </span><span style="font-size:9pt;font-family:sans-serif">""gpfsug-discuss<""gpfsug-discuss@spectrumscale.org "<gpfsug-discuss@spectrumscale.org>, gpfsug main discussion list <gpfsug-discuss@spectrumscale.org></span><br />
<span style="font-size:9pt;color:#5f5f5f;font-family:sans-serif">Date: </span><span style="font-size:9pt;font-family:sans-serif">03/08/2019 10:13 AM</span><br />
<span style="font-size:9pt;color:#5f5f5f;font-family:sans-serif">Subject: </span><span style="font-size:9pt;font-family:sans-serif">Re: [gpfsug-discuss] Follow-up: migrating billions of files</span><br />
<span style="font-size:9pt;color:#5f5f5f;font-family:sans-serif">Sent by: </span><span style="font-size:9pt;font-family:sans-serif">gpfsug-discuss-bounces@spectrumscale.org</span><br />
<hr noshade="noshade" />
<br />
<br />
<br />
<span style="font-size:9pt;font-family:sans-serif">I had to do this twice too. Once i had to copy a 4 PB filesystem as fast as possible when NSD disk descriptors were corrupted and shutting down GPFS would have led to me loosing those files forever, and the other was a regular maintenance but had to copy similar data in less time.</span><br />
<br />
<span style="font-size:9pt;font-family:sans-serif">In both the cases, i just used GPFS provided util scripts in /usr/lpp/mmfs/samples/util/ . These could be run only as root i believe. I wish i could give them to users to use.</span><br />
<br />
<span style="font-size:9pt;font-family:sans-serif">I had used few of those scripts like tsreaddir which used to be really fast in listing all the paths in the directories. It prints full paths of all files along with there inodes etc. I had modified it to print just the full file paths.</span><br />
<br />
<span style="font-size:9pt;font-family:sans-serif">I then use these paths and group them up in different groups which gets fed into a array jobs to the SGE/LSF cluster.</span><br />
<span style="font-size:9pt;font-family:sans-serif">Each array jobs basically uses GNU parallel and running something similar to rsync -avR . The “-R” option basically creates the directories as given.</span><br />
<span style="font-size:9pt;font-family:sans-serif">Of course this worked because i was using the fast private network to transfer between the storage systems. Also i know that cp or tar might be better than rsync with respect to speed, but rsync was convenient and i could always start over again without checkpointing or remembering where i left off previously.</span><br />
<br />
<span style="font-size:9pt;font-family:sans-serif">Similar to how Bill mentioned in the previous email, but i used gpfs util scripts and basic GNU parallel/rsync, SGE/LSF to submit jobs to the cluster as superuser. It used to work pretty well.</span><br />
<br />
<span style="font-size:9pt;font-family:sans-serif">Since then - I constantly use parallel and rsync to copy large directories.</span><br />
<br />
<span style="font-size:9pt;font-family:sans-serif">Thank you,</span><br />
<span style="font-size:9pt;font-family:sans-serif">Lohit</span><br />
<span style="font-size:9pt;font-family:sans-serif"><br />
On Mar 8, 2019, 7:43 AM -0600, William Abbott <babbott@rutgers.edu>, wrote:</span>
<p style="margin-top:5px;margin-Bottom:5px"><span style="font-size:9pt;font-family:sans-serif">We had a similar situation and ended up using parsyncfp, which generates<br />
multiple parallel rsyncs based on file lists. If they're on the same IB<br />
fabric (as ours were) you can use that instead of ethernet, and it<br />
worked pretty well. One caveat is that you need to follow the parallel<br />
transfers with a final single rsync, so you can use --delete.<br />
<br />
For the initial transfer you can also use bbcp. It can get very good<br />
performance but isn't nearly as convenient as rsync for subsequent<br />
transfers. The performance isn't good with small files but you can use<br />
tar on both ends to deal with that, in a similar way to what Uwe<br />
suggests below. The bbcp documentation outlines how to do that.<br />
<br />
Bill<br />
<br />
On 3/6/19 8:13 AM, Uwe Falke wrote:</span></p>
<p style="margin-top:5px;margin-Bottom:5px"><span style="font-size:9pt;font-family:sans-serif">Hi, in that case I'd open several tar pipes in parallel, maybe using<br />
directories carefully selected, like<br />
<br />
tar -c <source_dir> | ssh <target_host> "tar -x"<br />
<br />
I am not quite sure whether "-C /" for tar works here ("tar -C / -x"), but<br />
along these lines might be a good efficient method. target_hosts should be<br />
all nodes haveing the target file system mounted, and you should start<br />
those pipes on the nodes with the source file system.<br />
It is best to start with the largest directories, and use some<br />
masterscript to start the tar pipes controlled by semaphores to not<br />
overload anything.<br />
<br />
<br />
<br />
Mit freundlichen Grüßen / Kind regards<br />
<br />
<br />
Dr. Uwe Falke<br />
<br />
IT Specialist<br />
High Performance Computing Services / Integrated Technology Services /<br />
Data Center Services<br />
-------------------------------------------------------------------------------------------------------------------------------------------<br />
IBM Deutschland<br />
Rathausstr. 7<br />
09111 Chemnitz<br />
Phone: +49 371 6978 2165<br />
Mobile: +49 175 575 2877<br />
E-Mail: uwefalke@de.ibm.com<br />
-------------------------------------------------------------------------------------------------------------------------------------------<br />
IBM Deutschland Business & Technology Services GmbH / Geschäftsführung:<br />
Thomas Wolter, Sven Schooß<br />
Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart,<br />
HRB 17122<br />
<br />
<br />
<br />
<br />
From: "Oesterlin, Robert" <Robert.Oesterlin@nuance.com<br />
To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org<br />
Date: 06/03/2019 13:44<br />
Subject: [gpfsug-discuss] Follow-up: migrating billions of files<br />
Sent by: gpfsug-discuss-bounces@spectrumscale.org<br />
<br />
<br />
<br />
Some of you had questions to my original post. More information:<br />
<br />
Source:<br />
- Files are straight GPFS/Posix - no extended NFSV4 ACLs<br />
- A solution that requires $?s to be spent on software (ie, Aspera) isn?t<br />
a very viable option<br />
- Both source and target clusters are in the same DC<br />
- Source is stand-alone NSD servers (bonded 10g-E) and 8gb FC SAN storage<br />
- Approx 40 file systems, a few large ones with 300M-400M files each,<br />
others smaller<br />
- no independent file sets<br />
- migration must pose minimal disruption to existing users<br />
<br />
Target architecture is a small number of file systems (2-3) on ESS with<br />
independent filesets<br />
- Target (ESS) will have multiple 40gb-E links on each NSD server (GS4)<br />
<br />
My current thinking is AFM with a pre-populate of the file space and<br />
switch the clients over to have them pull data they need (most of the data<br />
is older and less active) and them let AFM populate the rest in the<br />
background.<br />
<br />
<br />
Bob Oesterlin<br />
Sr Principal Storage Engineer, Nuance<br />
_______________________________________________<br />
gpfsug-discuss mailing list<br />
gpfsug-discuss at spectrumscale.org<br /></span><a href="https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttp-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss%26d%3DDwICAg%26c%3Djf_iaSHvJObTbx-siA1ZOg%26r%3DfTuVGtgq6A14KiNeaGfNZzOOgtHW5Lm4crZU6lJxtB8%26m%3DJ5RpIj-EzFyU_dM9I4P8SrpHMikte_pn9sbllFcOvyM%26s%3DfEwDQyDSL7hvOVPbg_n8o_LDz-cLqSI6lQtSzmhaSoI%26e&data=02%7C01%7Cbabbott%40rutgers.edu%7C8cbda3d651584119393808d6a2358544%7Cb92d2b234d35447093ff69aca6632ffe%7C1%7C0%7C636874748092821399&sdata=W06i8IWqrxgEmdp3htxad0euiRhA6%2Bexd3YAziSrUhg%3D&reserved=0="><span style="font-size:9pt;font-family:sans-serif">https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttp-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss%26d%3DDwICAg%26c%3Djf_iaSHvJObTbx-siA1ZOg%26r%3DfTuVGtgq6A14KiNeaGfNZzOOgtHW5Lm4crZU6lJxtB8%26m%3DJ5RpIj-EzFyU_dM9I4P8SrpHMikte_pn9sbllFcOvyM%26s%3DfEwDQyDSL7hvOVPbg_n8o_LDz-cLqSI6lQtSzmhaSoI%26e&data=02%7C01%7Cbabbott%40rutgers.edu%7C8cbda3d651584119393808d6a2358544%7Cb92d2b234d35447093ff69aca6632ffe%7C1%7C0%7C636874748092821399&sdata=W06i8IWqrxgEmdp3htxad0euiRhA6%2Bexd3YAziSrUhg%3D&reserved=0=</span></a><span style="font-size:9pt;font-family:sans-serif"><br />
<br />
<br />
<br />
<br />
<br />
_______________________________________________<br />
gpfsug-discuss mailing list<br />
gpfsug-discuss at spectrumscale.org<br /></span><a href="https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7Cbabbott%40rutgers.edu%7C8cbda3d651584119393808d6a2358544%7Cb92d2b234d35447093ff69aca6632ffe%7C1%7C0%7C636874748092821399&sdata=Pjf4RhUchThoFvWI7hLJO4eWhoTXnIYd9m7Mvf809iE%3D&reserved=0"><span style="font-size:9pt;font-family:sans-serif">https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7Cbabbott%40rutgers.edu%7C8cbda3d651584119393808d6a2358544%7Cb92d2b234d35447093ff69aca6632ffe%7C1%7C0%7C636874748092821399&sdata=Pjf4RhUchThoFvWI7hLJO4eWhoTXnIYd9m7Mvf809iE%3D&reserved=0</span></a><span style="font-size:9pt;font-family:sans-serif"><br /></span></p>
<p style="margin-top:5px;margin-Bottom:5px"><span style="font-size:9pt;font-family:sans-serif">_______________________________________________<br />
gpfsug-discuss mailing list<br />
gpfsug-discuss at spectrumscale.org<br /></span><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><span style="font-size:9pt;font-family:sans-serif">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></a><tt><span style="font-size:10pt">_______________________________________________<br />
gpfsug-discuss mailing list<br />
gpfsug-discuss at spectrumscale.org<br /></span></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><span style="font-size:10pt">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></tt></a><tt><span style="font-size:10pt"><br /></span></tt></p>
<p style="margin-top:5px;margin-Bottom:5px"></p>
<p style="margin-top:5px;margin-Bottom:5px"></p>
<br />
_______________________________________________<br />
gpfsug-discuss mailing list<br />
gpfsug-discuss at spectrumscale.org<br />
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<br /></blockquote>
</div>
</body>
</html>