<font size=2 face="sans-serif">I would recommend opening a case, collect
the default traces from both gateway and application (or protocol) nodes
to check the RPC overhead. There should not be difference between mmap
IO and regular IO for AFM filesets. Also note that refresh intervals are
stored as part of inode and for the large number of file access it is possible
that inodes are evicted as part of dcache shrinkage and next access to
the same files might go to home for the revalidation. </font><font size=3 face="Calibri">afmRefreshAsync
</font><font size=2 face="sans-serif"> option can be set at fleset
level also. Looks like it is missing from the documentation, this will
be corrected.</font><br><br><font size=2 face="sans-serif">~Venkat (vpuvvada@in.ibm.com)</font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Andreas Mattsson <andreas.mattsson@maxiv.lu.se></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug main discussion
list <gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">10/03/2019 07:25 PM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">[EXTERNAL] Re:
[gpfsug-discuss] afmRefreshAsync questions</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><font size=3 face="Calibri">After further investigaion, it seems like
this XDS software is using memory mapped io when operating on the files.</font><br><font size=3 face="Calibri">Is it possible that MMAP IO has a higher
performance hit by AFM than regular file access?</font><br><br><font size=3 face="Calibri">/Andreas</font><br><font size=3 face="Times New Roman">____________________________________________</font><br><img src=cid:_4_DBA3DA28DBA3D7A00018C6B56525848A style="border:0px solid;"><p><br><font size=2 face="Calibri"><b>Andreas Mattsson</b></font><br><font size=1 face="Calibri">Systems Engineer</font><br><font size=3 face="Calibri"> </font><br><font size=1 face="Calibri">MAX IV Laboratory<br>Lund University</font><font size=1 face="Calibri"><br>P.O. Box 118, SE-221 00 Lund, Sweden<br>Visiting address: Fotongatan 2, 224 84 Lund</font><font size=1 face="Calibri"><br>Mobile: +46 706 64 95 44</font><font size=1 color=blue face="Calibri"><u><br></u></font><a href="mailto:andreas.mattsson@maxiv.se" target="_blank"></a><a href="mailto:andreas.mattsson@maxiv.lu.se"><font size=1 color=blue face="Calibri"><u>andreas.mattsson@maxiv.lu.se</u></font></a><font size=1 color=blue face="Calibri"><u><br></u></font><a href="http://www.maxiv.se/" target="_blank"><font size=1 color=blue face="Calibri"><u>www.maxiv.se</u></font></a><p><br><hr><br><font size=2 face="Calibri"><b>Från:</b> gpfsug-discuss-bounces@spectrumscale.org
<gpfsug-discuss-bounces@spectrumscale.org> för Andreas Mattsson <andreas.mattsson@maxiv.lu.se><b><br>Skickat:</b> den 1 oktober 2019 08:33:35<b><br>Till:</b> gpfsug main discussion list<b><br>Ämne:</b> Re: [gpfsug-discuss] afmRefreshAsync questions</font><font size=3></font><br><font size=3> </font><br><font size=3 face="Calibri">Hi,</font><br><font size=3 face="Calibri">I've tried increasing all the refresh intervals,
but even at 300 seconds, there is very little performance increase.</font><br><font size=3 face="Calibri">The job runs in several steps, and gets
held up at two places, as far as I can see.</font><br><font size=3 face="Calibri">First at a kind of parallelisation step
where about 1000-3000 files are created in the current working folder on
a single compute node, and then at a step where lots of small output files
are written on each of the compute nodes involved in the job. </font><br><font size=3 face="Calibri">Comparing with running the same data set
on a non-AFM cache fileset in the same storage system, it runs at least
a factor 5 slower, even with really high refresh intervals.</font><br><font size=3 face="Calibri">In the Scale documentation, it states that
the afmRefreshAsync is only configurable cluster wide. Is it also configurable
on a per-fileset level?</font><br><a href="https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.3/com.ibm.spectrum.scale.v5r03.doc/bl1adm_configurationparametersAFM.htm"><font size=3 color=blue face="Calibri"><u>https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.3/com.ibm.spectrum.scale.v5r03.doc/bl1adm_configurationparametersAFM.htm</u></font></a><br><br><br><font size=3 face="Calibri">The software is XDS, </font><a href="http://xds.mpimf-heidelberg.mpg.de/"><font size=3 color=blue face="Calibri"><u>http://xds.mpimf-heidelberg.mpg.de/</u></font></a><br><font size=3 face="Calibri">Unfortunately it is a closed source software,
so it is not possible to adapt the software.</font><br><br><font size=3 face="Calibri">Regards,</font><br><font size=3 face="Calibri">Andreas Mattsson</font><br><br><font size=3 face="Times New Roman">____________________________________________</font><br><img src=cid:_4_DBA2473CDBA244B40018C6B56525848A style="border:0px solid;"><p><br><font size=2 face="Calibri"><b>Andreas Mattsson</b></font><br><font size=1 face="Calibri">Systems Engineer</font><br><font size=3 face="Calibri"> </font><br><font size=1 face="Calibri">MAX IV Laboratory<br>Lund University</font><font size=1 face="Calibri"><br>P.O. Box 118, SE-221 00 Lund, Sweden<br>Visiting address: Fotongatan 2, 224 84 Lund</font><font size=1 face="Calibri"><br>Mobile: +46 706 64 95 44</font><font size=1 color=blue face="Calibri"><u><br></u></font><a href="mailto:andreas.mattsson@maxiv.se" target="_blank"></a><a href="mailto:andreas.mattsson@maxiv.lu.se"><font size=1 color=blue face="Calibri"><u>andreas.mattsson@maxiv.lu.se</u></font></a><font size=1 color=blue face="Calibri"><u><br></u></font><a href="http://www.maxiv.se/" target="_blank"><font size=1 color=blue face="Calibri"><u>www.maxiv.se</u></font></a><p><br><hr><br><font size=2 face="Calibri"><b>Från:</b> gpfsug-discuss-bounces@spectrumscale.org
<gpfsug-discuss-bounces@spectrumscale.org> för Venkateswara R Puvvada
<vpuvvada@in.ibm.com><b><br>Skickat:</b> den 27 september 2019 10:23:13<b><br>Till:</b> gpfsug main discussion list<b><br>Ämne:</b> Re: [gpfsug-discuss] afmRefreshAsync questions</font><font size=3></font><br><font size=3> </font><br><font size=2 face="sans-serif">Hi,</font><font size=3><br></font><font size=2 face="sans-serif"><br>Both storage and client clusters have to be on 5.0.3.x to get the
AFM revalidation performance with </font><font size=3 face="Calibri">afmRefreshAsync</font><font size=2 face="sans-serif">.
What are the refresh intervals ?, you could also try increasing them. Is
this config option set at fileset level or cluster level ?</font><font size=3><br></font><font size=2 face="sans-serif"><br>~Venkat (vpuvvada@in.ibm.com)</font><font size=3><br><br><br></font><font size=1 color=#5f5f5f face="sans-serif"><br>From: </font><font size=1 face="sans-serif">Andreas
Mattsson <andreas.mattsson@maxiv.lu.se></font><font size=1 color=#5f5f5f face="sans-serif"><br>To: </font><font size=1 face="sans-serif">GPFS
User Group <gpfsug-discuss@spectrumscale.org></font><font size=1 color=#5f5f5f face="sans-serif"><br>Date: </font><font size=1 face="sans-serif">09/26/2019
03:26 PM</font><font size=1 color=#5f5f5f face="sans-serif"><br>Subject: </font><font size=1 face="sans-serif">[EXTERNAL]
[gpfsug-discuss] afmRefreshAsync questions</font><font size=1 color=#5f5f5f face="sans-serif"><br>Sent by: </font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><font size=3><br></font><hr noshade><font size=3><br><br></font><font size=3 face="Calibri"><br>Hi,<br>Due to having a data analysis software that isn't running well at all in
our AFM caches, it runs 4-6 times slower on an AFM cache than on a non-AFM
fileset on the same storage system, I wanted to try out the afmRefreshAsync
feature that came with 5.0.3 to see if it is the cache data refresh that
is holding things up.<br>Enabling this feature has had zero impact on performance of the software
though.</font><font size=3><br></font><font size=3 face="Calibri"><br>The storage cluster is running 5.0.3.x, and afmRefreshAsync has been set
there, but at the moment the remote-mounting client cluster is still running
5.0.2.x.<br>Would this feature still have any effect in this setup?</font><font size=3><br></font><font size=3 face="Calibri"><br>Regards,<br>Andreas Mattsson</font><font size=3><br></font><font size=3 face="Times New Roman"><br>____________________________________________</font><font size=3><br></font><img src=cid:_4_DBA27C04DBA26E040018C6B56525848A width=121 height=65 style="border:0px solid;"><br><font size=2 face="Calibri"><b><br>Andreas Mattsson</b></font><font size=1 face="Calibri"><br>Systems Engineer</font><font size=3 face="Calibri"><br> </font><font size=1 face="Calibri"><br>MAX IV Laboratory<br>Lund University<br>P.O. Box 118, SE-221 00 Lund, Sweden<br>Visiting address: Fotongatan 2, 224 84 Lund<br>Mobile: +46 706 64 95 44</font><font size=3 color=blue><u><br></u></font><a href="mailto:andreas.mattsson@maxiv.se" target="_blank"></a><a href="mailto:andreas.mattsson@maxiv.lu.se"><font size=1 color=blue face="Calibri"><u>andreas.mattsson@maxiv.lu.se</u></font></a><font size=3 color=blue><u><br></u></font><a href="http://www.maxiv.se/" target="_blank"><font size=1 color=blue face="Calibri"><u>www.maxiv.se</u></font></a><br><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org</font></tt><font size=3 color=blue><u><br></u></font><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></tt></a><br><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br></font></tt><br><br><BR>