<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div dir="ltr" >Hi Zachary,</div>
<div dir="ltr" > </div>
<div dir="ltr" >When a compressed file is mmapped, each 4K read in your tests causes the accessed part of the file to be decompressed (in the granularity of 10 GPFS blocks). For usual file sizes, the parts being accessed will be decompressed and IOs speed will be normal except for the first 4K IO in each 10-GPFS-block group. For very large files, a large percentage of small random IOs may keep getting amplified to 10-block decompression IO for a long time. This is probably what happened in your mmap application run.</div>
<div dir="ltr" > </div>
<div dir="ltr" >The suggestion is to not compress files until they have become cold (not likely to be accessed any time soon) and avoid compressing very large files that may be accessed through mmap later. The product already has a built-in protection preventing compression of files that are mmapped at compression time. You can add an exclude rule in the compression policy run for files that are identified to have mmap performance issues (in case they get mmapped after being compressed in a periodical policy run).</div>
<div dir="ltr" > </div>
<div dir="ltr" >Leo Luan</div>
<div dir="ltr" > </div>
<div dir="ltr" ><font color="#5F5F5F" size="2" style="outline: none; font-family: "Helvetica Neue", Helvetica, Arial, "Lucida Grande", sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" >From: </font><font size="2" style="outline: none; color: rgb(18, 18, 18); font-family: "Helvetica Neue", Helvetica, Arial, "Lucida Grande", sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" >Zachary Giles <zgiles@gmail.com></font><br style="outline: none; color: rgb(18, 18, 18); font-family: "Helvetica Neue", Helvetica, Arial, "Lucida Grande", sans-serif; font-size: 12px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" ><font color="#5F5F5F" size="2" style="outline: none; font-family: "Helvetica Neue", Helvetica, Arial, "Lucida Grande", sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" >To: </font><font size="2" style="outline: none; color: rgb(18, 18, 18); font-family: "Helvetica Neue", Helvetica, Arial, "Lucida Grande", sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" >gpfsug main discussion list <gpfsug-discuss@spectrumscale.org></font><br style="outline: none; color: rgb(18, 18, 18); font-family: "Helvetica Neue", Helvetica, Arial, "Lucida Grande", sans-serif; font-size: 12px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" ><font color="#5F5F5F" size="2" style="outline: none; font-family: "Helvetica Neue", Helvetica, Arial, "Lucida Grande", sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" >Date: </font><font size="2" style="outline: none; color: rgb(18, 18, 18); font-family: "Helvetica Neue", Helvetica, Arial, "Lucida Grande", sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" >02/10/2017 01:57 PM</font><br style="outline: none; color: rgb(18, 18, 18); font-family: "Helvetica Neue", Helvetica, Arial, "Lucida Grande", sans-serif; font-size: 12px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" ><font color="#5F5F5F" size="2" style="outline: none; font-family: "Helvetica Neue", Helvetica, Arial, "Lucida Grande", sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" >Subject: </font><font size="2" style="outline: none; color: rgb(18, 18, 18); font-family: "Helvetica Neue", Helvetica, Arial, "Lucida Grande", sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" >[gpfsug-discuss] Questions about mmap GPFS and compression</font><br style="outline: none; color: rgb(18, 18, 18); font-family: "Helvetica Neue", Helvetica, Arial, "Lucida Grande", sans-serif; font-size: 12px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" ><font color="#5F5F5F" size="2" style="outline: none; font-family: "Helvetica Neue", Helvetica, Arial, "Lucida Grande", sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" >Sent by: </font><font size="2" style="outline: none; color: rgb(18, 18, 18); font-family: "Helvetica Neue", Helvetica, Arial, "Lucida Grande", sans-serif; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" >gpfsug-discuss-bounces@spectrumscale.org</font></div>
<hr align="left" dir="ltr" size="2" style="outline: none; box-sizing: border-box; font-family: "Helvetica Neue", Helvetica, Arial, "Lucida Grande", sans-serif; font-size: 12px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); color: rgb(128, 145, 165);" width="100%" ><div dir="ltr" >
<div>
<div><br style="outline: none; color: rgb(18, 18, 18); font-family: "Helvetica Neue", Helvetica, Arial, "Lucida Grande", sans-serif; font-size: 12px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" ><tt style="outline: none; color: rgb(18, 18, 18); font-size: 12px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" ><font face="" size="3" style="outline: none;" >Hello All,<br style="outline: none;" ><br style="outline: none;" >I've been seeing some less than desirable behavior with mmap and<br style="outline: none;" >compression in GPFS. Curious if others see similar or have any ideas<br style="outline: none;" >if this is accurate..<br style="outline: none;" >The guys here want me to open an IBM ticket, but I figured I'd see if<br style="outline: none;" >anyone has had this experience before.<br style="outline: none;" ><br style="outline: none;" >We have an internally developed app that runs on our cluster<br style="outline: none;" >referencing data sitting in GPFS. It is using mmap to access the files<br style="outline: none;" >due to a library we're using that requires it.<br style="outline: none;" ><br style="outline: none;" >If we run the app against some data on GPFS, it performs well..<br style="outline: none;" >finishing in a few minutes time -- Great. However, if we compress the<br style="outline: none;" >file (in GPFS), the app is still running after 2 days time.<br style="outline: none;" >stracing the app shows that is polling on a file descriptor, forever..<br style="outline: none;" >as if a data block is still pending.<br style="outline: none;" ><br style="outline: none;" >I know mmap is supported with compression according to the manual<br style="outline: none;" >(with some stipulations), and that performance is expected to be much<br style="outline: none;" >less since it's more large-block oriented due to decompressed in<br style="outline: none;" >groups.. no problem. But it seems like some data should get returned.<br style="outline: none;" ><br style="outline: none;" >I'm surprised to find that a very small amount of data is sitting in<br style="outline: none;" >the buffers (mmfsadm dump buffers) in reference to the inodes. The<br style="outline: none;" >decompression thread is running continuously, while the app is still<br style="outline: none;" >polling for data from memory and sleeping, retrying, sleeping, repeat.<br style="outline: none;" ><br style="outline: none;" >What I believe is happening is that the 4k pages are being pulled out<br style="outline: none;" >of large decompression groups from an mmap read request, put in the<br style="outline: none;" >buffer, then the compression group data is thrown away since it has<br style="outline: none;" >the result it wants, only to need another piece of data that would<br style="outline: none;" >have been in that group slightly later, which is recalled, put in the<br style="outline: none;" >buffer.. etc. Thus an infinite slowdown. Perhaps also the data is<br style="outline: none;" >expiring out of the buffer before the app has a chance to read it. I<br style="outline: none;" >can't tell. In any case, the app makes zero progress.<br style="outline: none;" ><br style="outline: none;" >I tried without our app, using fio.. mmap on an uncompressed file with<br style="outline: none;" >1 thread 1 iodepth, random read, 4k blocks, yields ~76MB/s (not<br style="outline: none;" >impressive). However, on a compressed file it is only 20KB/s max. (<br style="outline: none;" >far less impressive ). Reading a file using aio etc is over 3GB/s on a<br style="outline: none;" >single thread without even trying.<br style="outline: none;" ><br style="outline: none;" >What do you think?<br style="outline: none;" >Anyone see anything like this? Perhaps there are some tunings to waste<br style="outline: none;" >a bit more memory on cached blocks rather than make decompression<br style="outline: none;" >recycle?<br style="outline: none;" ><br style="outline: none;" >I've searched back the archives a bit. There's a May 2013 thread about<br style="outline: none;" >slowness as well. I think we're seeing much much less than that. Our<br style="outline: none;" >page pools are of decent size. Its not just slowness, it's as if the<br style="outline: none;" >app never gets a block back at all. ( We could handle slowness .. )<br style="outline: none;" ><br style="outline: none;" >Thanks. Open to ideas..<br style="outline: none;" ><br style="outline: none;" >-Zach Giles</font></tt></div></div></div></div><BR>