<div dir="ltr">Hi Sven<div>yes, it's primarily reads.</div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><span style="font-size:small"><font color="#888888">Kind regards</font></span><span style="font-size:small"><br><div><p>Ray Coetzee<br>Mob: +44 759 704 7060</p></div><div><p>Skype: ray.coetzee<u></u><u></u></p></div><div><p>Email: <a href="mailto:coetzee.ray@gmail.com" style="color:rgb(17,85,204)" target="_blank">coetzee.ray@gmail.com</a></p></div></span><br></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Fri, Jan 12, 2018 at 8:57 PM, Sven Oehme <span dir="ltr"><<a href="mailto:oehmes@gmail.com" target="_blank">oehmes@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>is this primary read or write ?</span><div class="HOEnZb"><div class="h5"><br><br><div class="gmail_quote"><div dir="ltr">On Fri, Jan 12, 2018, 12:51 PM Ray Coetzee <<a href="mailto:coetzee.ray@gmail.com" target="_blank">coetzee.ray@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex" id="m_-5984835922597167433gmail_block_quote0"><div dir="ltr">Hey Sven, the latest clients I've tested with is 4.2.3-6 on RHEL7.2. (Without the meltdown patch)<div> </div><div>Hey Bryan, I remember that quote from Yuri, that's why I hoped some "magic" may have been done since then. <br><div><br><div>Other attempts to improve performance I've tried include:</div><div><ul><li>Using LROC to have a larger chance of a cache hit (Unfortunately the entire dataset is multiple TB) <br></li><li>Built an NVMe based scratch filesystem (18x 1.8TB NVMe) just for this purpose (Job runs halved in duration but nowhere near what NFS can give)</li><li>Made changes to prefecthPct, PrefetchAgressiveness, DisableDIO, and some others with little improvement.</li></ul><div>For those interested, as a performance comparison. The same job when run on an aging Isilon takes 1m30s, while GPFS will take ~38min on the all NVMe scratch filesystem and over 60min on spindle based filesystem.</div></div></div></div><div class="gmail_extra"><br clear="all"><div><div class="m_-5984835922597167433m_-6329203610002665801gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><span style="font-size:small"><font color="#888888">Kind regards</font></span></div></div></div></div></div></div></div><div dir="ltr"><div class="gmail_extra"><div><div class="m_-5984835922597167433m_-6329203610002665801gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><span style="font-size:small"><br><div><p>Ray Coetzee<br>Email: <a href="mailto:coetzee.ray@gmail.com" target="_blank">coetzee.ray@gmail.com</a><br></p></div></span><br></div></div></div></div></div></div></div><div dir="ltr"><div class="gmail_extra">
<br><div class="gmail_quote">On Fri, Jan 12, 2018 at 4:12 PM, Bryan Banister <span dir="ltr"><<a href="mailto:bbanister@jumptrading.com" target="_blank">bbanister@jumptrading.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div class="m_-5984835922597167433m_-6329203610002665801m_-3713892279031986925WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">You could put all of your data onto SSDs in a RAID1 configuration so that you don’t have insane read-modify-write penalties on writes (RAID1) and avoid horrible
seek thrashing that spinning rust requires (SSD random access medium) for your 4K I/O operations.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">One of my favorite Yuri quotes, “The mmap code is like asbestos… best not to touch it”. He gave many reasons why mmap operations on a distributed file system
is incredibly hard and not recommended.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">-Bryan<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@<wbr>spectrumscale.org</a> [mailto:<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-<wbr>bounces@spectrumscale.org</a>]
<b>On Behalf Of </b>Sven Oehme<br>
<b>Sent:</b> Friday, January 12, 2018 8:45 AM<br>
<b>To:</b> <a href="mailto:coetzee.ray@gmail.com" target="_blank">coetzee.ray@gmail.com</a>; gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.<wbr>org</a>><br>
<b>Subject:</b> Re: [gpfsug-discuss] mmap performance against Spectrum Scale<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><em><span style="font-size:8.5pt;font-family:"Verdana",sans-serif;color:#333333">Note: External Email</span></em><u></u><u></u></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" align="center">
</div><div><div class="m_-5984835922597167433m_-6329203610002665801h5">
<div>
<p class="MsoNormal">what version of Scale are you using right now ?<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Fri, Jan 12, 2018 at 2:29 AM Ray Coetzee <<a href="mailto:coetzee.ray@gmail.com" target="_blank">coetzee.ray@gmail.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal">I'd like to ask the group of their experiences in improving the performance of applications that use mmap calls against files on Spectrum Scale.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Besides using an NFS export from CES instead of a native GPFS mount, or precaching the dataset into the pagepool, what other approaches are there to offset the performance hit of the 4K IO size?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="color:#888888">Kind regards</span><u></u><u></u></p>
<div>
<p>Ray Coetzee<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal">______________________________<wbr>_________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</a><u></u><u></u></p>
</blockquote>
</div>
</div></div></div>
<br>
<hr>
<font face="Arial" color="Gray" size="1"><br>
Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this
email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness
or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial
product.<br>
</font>
</div>
</blockquote></div><br></div></div>
______________________________<wbr>_________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</a><br>
</blockquote></div>
</div></div></blockquote></div><br></div>