[gpfsug-discuss] AFM Alternative? Aspera?

[Venkateswara R Puvvada]

Transferring the small files with AFM  + NFS over high latency networks is 
 always a challenge. For example,  for each small file replication AFM 
performs a lookup, create, write and set mtime operation. If the latency 
is 10ms,  replication of each file takes minimum (10 * 4 = 40 ms)  amount 
of time. AFM is not a network acceleration tool and also it does not use 
compression.  If the file sizes are big, AFM parallel IO and parallel 
mounts feature can be used.  Aspera can be used to transfer the small 
files over high latency network with better utilization of the network 
bandwidth.

https://www.ibm.com/support/knowledgecenter/no/STXKQY_5.0.4/com.ibm.spectrum.scale.v5r04.doc/b1lins_afmparalleldatatransferwithremotemounts.htm
https://www.ibm.com/support/knowledgecenter/no/STXKQY_5.0.4/com.ibm.spectrum.scale.v5r04.doc/bl1ins_paralleldatatransfersafm.htm

~Venkat (vpuvvada at in.ibm.com)



From:   Chris Schlipalius <chris.schlipalius at pawsey.org.au>
To:     <gpfsug-discuss at spectrumscale.org>
Date:   02/27/2020 05:54 AM
Subject:        [EXTERNAL] Re: [gpfsug-discuss] AFM Alternative? Aspera?
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Maybe the following would assist? I do think tarring up files first is 
best, but you could always check out:
http://www.redbooks.ibm.com/redpapers/pdfs/redp5527.pdf
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.spectrumscaleug.org_wp-2Dcontent_uploads_2019_05_SSSD19DE-2DDay-2D2-2DB02-2DIntegration-2Dof-2DSpectrum-2DScale-2Dand-2DAspera-2DSync.pdf&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=1pVcjKeZ7gCaDtLoJFbfKCETe1XOmol6d2ryoccqC1A&s=tRCxd4SimJH_eycqekhzM0Qp3TB3NtaIYWBvyQnrIiM&e= 


Aspera sync integration

(non html links added for your use – how they don’t get scrubbed:
www.spectrumscaleug.org/wp-content/uploads/2019/05/SSSD19DE-Day-2-B02-Integration-of-Spectrum-Scale-and-Aspera-Sync.pdf 

www.redbooks.ibm.com/redpapers/pdfs/redp5527.pdf
)
Regards,
Chris Schlipalius
 
Team Lead, Data Storage Infrastructure, Data & Visualisation, Pawsey 
Supercomputing Centre (CSIRO)
1 Bryce Avenue
Kensington  WA  6151
Australia
 
Tel  +61 8 6436 8815 
Email  chris.schlipalius at pawsey.org.au
Web  www.pawsey.org.au <
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.pawsey.org.au_&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=1pVcjKeZ7gCaDtLoJFbfKCETe1XOmol6d2ryoccqC1A&s=Xkm8VFy3l6nyD40yhONihsKcqmwRhy4SZyd0lwHf1GA&e= 
>
 
 
 


On 26/2/20, 9:39 pm, "gpfsug-discuss-bounces at spectrumscale.org on behalf 
of gpfsug-discuss-request at spectrumscale.org" 
<gpfsug-discuss-bounces at spectrumscale.org on behalf of 
gpfsug-discuss-request at spectrumscale.org> wrote:

    Re: AFM Alternative?



_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=1pVcjKeZ7gCaDtLoJFbfKCETe1XOmol6d2ryoccqC1A&s=mYK1ZsVgtsM6HntRMLPS49tKvEhhgGAdWF2qniyn9Ko&e= 






-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20200228/0cb27772/attachment-0002.htm>