<div dir="ltr">Hi Lohit, <div><br></div><div>i am working with ray on a mmap performance improvement right now, which most likely has the same root cause as yours , see --> <a href="http://gpfsug.org/pipermail/gpfsug-discuss/2018-January/004411.html">http://gpfsug.org/pipermail/gpfsug-discuss/2018-January/004411.html</a></div><div>the thread above is silent after a couple of back and rorth, but ray and i have active communication in the background and will repost as soon as there is something new to share. </div><div>i am happy to look at this issue after we finish with ray's workload if there is something missing, but first let's finish his, get you try the same fix and see if there is something missing. </div><div><br></div><div>btw. if people would share their use of MMAP , what applications they use (home grown, just use lmdb which uses mmap under the cover, etc) please let me know so i get a better picture on how wide the usage is with GPFS. i know a lot of the ML/DL workloads are using it, but i would like to know what else is out there i might not think about. feel free to drop me a personal note, i might not reply to it right away, but eventually. </div><div><br></div><div>thx. sven<br></div><div><br></div><div><br><div class="gmail_quote"><div dir="ltr">On Thu, Feb 22, 2018 at 12:33 PM <<a href="mailto:valleru@cbio.mskcc.org" target="_blank">valleru@cbio.mskcc.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div name="messageBodySection" style="font-size:14px;font-family:-apple-system,BlinkMacSystemFont,sans-serif">Hi all,
<div><br></div>
<div>I wanted to know, how does mmap interact with GPFS pagepool with respect to filesystem block-size? </div>
<div>Does the efficiency depend on the mmap read size and the block-size of the filesystem even if all the data is cached in pagepool?</div>
<div><br></div>
<div>GPFS 4.2.3.2 and CentOS7.</div>
<div><br></div>
<div>Here is what i observed:</div>
<div><br></div>
<div>I was testing a user script that uses mmap to read from 100M to 500MB files.</div>
<div><br></div>
<div>The above files are stored on 3 different filesystems.</div>
<div><br></div>
<div>Compute nodes - 10G pagepool and 5G seqdiscardthreshold.</div>
<div><br></div>
<div>1. 4M block size GPFS filesystem, with separate metadata and data. Data on Near line and metadata on SSDs</div>
<div>2. 1M block size GPFS filesystem as a AFM cache cluster, "with all the required files fully cached" from the above GPFS cluster as home. Data and Metadata together on SSDs</div>
<div>3. 16M block size GPFS filesystem, with separate metadata and data. Data on Near line and metadata on SSDs</div>
<div><br></div>
<div>When i run the script first time for “each" filesystem:</div>
<div>I see that GPFS reads from the files, and caches into the pagepool as it reads, from mmdiag -- iohist</div>
<div><br></div>
<div>When i run the second time, i see that there are no IO requests from the compute node to GPFS NSD servers, which is expected since all the data from the 3 filesystems is cached.</div>
<div><br></div>
<div>However - the time taken for the script to run for the files in the 3 different filesystems is different - although i know that they are just "mmapping"/reading from pagepool/cache and not from disk.</div>
<div><br></div>
<div>Here is the difference in time, for IO just from pagepool:</div>
<div><br></div>
<div>20s 4M block size</div>
<div>15s 1M block size</div>
<div>40S 16M block size.</div>
<div><br></div>
<div>Why do i see a difference when trying to mmap reads from different block-size filesystems, although i see that the IO requests are not hitting disks and just the pagepool?</div>
<div><br></div>
<div>I am willing to share the strace output and mmdiag outputs if needed.</div>
</div>
<div name="messageSignatureSection" style="font-size:14px;font-family:-apple-system,BlinkMacSystemFont,sans-serif"><br>
Thanks,
<div>Lohit</div>
</div>
<div name="messageReplySection" style="font-size:14px;font-family:-apple-system,BlinkMacSystemFont,sans-serif"><br>
<div></div>
</div>
</div>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
</blockquote></div></div></div>