<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">This sounds like a bug to me... (I wouldn't expect mmchattr works on different node than other file access). I would check "mmdiag --iohist verbose" during these slow reads, to see if it gives a hint at what it's doing, versus what it shows during "mmchattr". Maybe one is triggering prefetch, while the other is some kind of random IO ? Also might be worth to try a mmtrace. Compare the traces for<div><br></div><div><div style="color:rgb(0,0,0)">mmtrace start trace="all 0 vnode 1 vnop 1 io 1"</div><div style="color:rgb(0,0,0)">cat compressedLargeFile</div><div style="color:rgb(0,0,0)">mmtrace stop</div></div><div><br></div><div>vs.:</div><div><br></div><div><div style="color:rgb(0,0,0)"><div>mmtrace start trace="all 0 vnode 1 vnop 1 io 1"</div><div>mmchattr --compress no someLargeFile</div><div>mmtrace stop</div></div><div style="color:rgb(0,0,0)"><br></div></div><div style="color:rgb(0,0,0)">(but please make sure that the file wasn't already uncompressed in pagepool in this second run).</div><div style="color:rgb(0,0,0)"><br></div><div style="color:rgb(0,0,0)"><br></div><div style="color:rgb(0,0,0)"> -jf</div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jan 20, 2021 at 12:59 PM Daniel Kidger <<a href="mailto:daniel.kidger@uk.ibm.com">daniel.kidger@uk.ibm.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:9pt"><div dir="ltr">I think you need to think about which node the file is being decompressed on (and if that node has plenty of space in the page pool.)</div>
<div dir="ltr">iirc mmchattr works on one of the 'manager' nodes not necessarily the node you typed the command on?</div>
<div dir="ltr"><div dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10pt"><div dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10pt"><div dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10pt"><div dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10pt"><div dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10pt"><div dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10pt"><div dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10pt"><div dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10.5pt"><div dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10.5pt"><div dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10.5pt"><div dir="ltr" style="font-family:Arial,Helvetica,sans-serif;font-size:10.5pt"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr" style="margin-top:20px"><div style="margin-top:10px"><div style="font-family:sans-serif;font-size:8pt;margin-bottom:0.0001pt;line-height:normal"><font face="Verdana,Arial,Helvetica,sans-serif" size="2"><font face="Verdana,Arial,Helvetica,sans-serif" size="2"><span style="font-size:1em">Daniel</span></font></font></div>
<div style="font-family:sans-serif;font-size:8pt;margin-bottom:0.0001pt;line-height:normal"><font face="Verdana,Arial,Helvetica,sans-serif" size="2"><font face="Verdana,Arial,Helvetica,sans-serif" size="2"> </font></font></div>
<div style="margin-bottom:0.0001pt;line-height:normal"><font face="Verdana,Arial,Helvetica,sans-serif" size="2"><font face="Verdana,Arial,Helvetica,sans-serif" size="2"><font color="#19A9F3" size="3" style="font-family:"Helvetica Neue",Helvetica,Arial,"Lucida Grande",sans-serif;font-size:8pt;outline:none">_________________________________________________________</font><br style="outline:none"><font color="#808080" face="Arial" size="2"><span><b>Daniel Kidger Ph.D.</b></span></font></font></font></div>
<div style="margin-bottom:0.0001pt;line-height:normal"><font face="Verdana,Arial,Helvetica,sans-serif" size="2"><font face="Verdana,Arial,Helvetica,sans-serif" size="2"><span style="outline:none;font-size:10.666666984558105px"><font color="#121212"><span style="outline:none;font-size:7.5pt">IBM Technical Sales Specialist</span></font></span></font></font></div>
<div style="margin-bottom:0.0001pt;line-height:normal"><font face="Verdana,Arial,Helvetica,sans-serif" size="2"><font face="Verdana,Arial,Helvetica,sans-serif" size="2"><span style="outline:none;font-size:10.666666984558105px"><font color="#121212"><span style="outline:none;font-size:7.5pt">Spectrum Scale, Spectrum Discover and IBM Cloud Object Storage<br style="outline:none"><br style="outline:none"><span style="font-size:0.857em">+</span></span></font></span><span style="font-size:0.857em"><span style="outline:none;font-family:sans-serif"><font color="#5f5f5f"><span style="outline:none;font-family:Verdana,Geneva,sans-serif"><span style="outline:none">44-(0)7818 522 266 </span></span></font></span></span><br style="outline:none;font-family:sans-serif;font-size:10.666666984558105px"><span style="outline:none;font-family:sans-serif;font-size:10.666666984558105px"><font color="#5f5f5f"><span style="outline:none;font-family:Arial,Helvetica,sans-serif"><span style="outline:none;font-size:8pt"><a href="mailto:daniel.kidger@uk.ibm.com" target="_blank">daniel.kidger@uk.ibm.com</a></span></span></font></span> </font></font>
<table border="0" style="border-collapse:collapse;border-color:rgb(105,105,105);table-layout:fixed;width:360px;word-wrap:break-word"> <tbody> <tr> <td style="overflow:hidden;width:66px;border-color:rgb(105,105,105)"><font face="Verdana,Arial,Helvetica,sans-serif" size="2"><font face="Verdana,Arial,Helvetica,sans-serif" size="2"><a href="https://www.youracclaim.com/badges/687cf790-fe65-4a92-b129-d23ae41862ac/public_url" target="_blank"><img alt="" src="https://images.youracclaim.com/size/110x110/images/c49300ae-d13e-4071-90f5-15f59d199c9e/IBM_Volunteers_Gold_v6.png" style="width: 90px; height: 90px;"></a></font></font></td> <td style="overflow:hidden;width:66px;border-color:rgb(105,105,105)"><a href="https://www.youracclaim.com/badges/8dac4bc0-7b3a-4035-b127-daec8dce9200" target="_blank"><img src="https://images.youracclaim.com/size/110x110/images/c3275eba-a1a8-4602-ab7a-c857877733fb/Virtual_Collabortor_Enterprise.png" style="width: 90px; height: 90px;"></a></td> <td style="overflow:hidden;width:66px;border-color:rgb(105,105,105)"><font face="Verdana,Arial,Helvetica,sans-serif" size="2"><font face="Verdana,Arial,Helvetica,sans-serif" size="2"><a href="https://www.youracclaim.com/badges/8153c6a7-3e02-40be-87ee-24e27ae9459c/public_url" target="_blank"><img alt="" src="https://images.youracclaim.com/size/110x110/images/f2539224-f951-46b4-b376-b88f21c2be98/IBM-Selling-Certification---Level-1.png" style="width: 90px; height: 90px;"></a></font></font></td> <td style="overflow:hidden;width:66px;border-color:rgb(105,105,105)"><font face="Verdana,Arial,Helvetica,sans-serif" size="2"><font face="Verdana,Arial,Helvetica,sans-serif" size="2"><a href="https://www.youracclaim.com/badges/78197e2c-4277-4ec9-808b-ad6abe1e1b16/public_url" target="_blank"><img alt="" src="https://images.youracclaim.com/size/110x110/images/ea52b12f-97ac-4e72-8d24-b0ced8054e7d/Storage_Technical_V1.png" style="width: 90px; height: 90px;"></a></font></font></td> </tr> </tbody></table><font face="Verdana,Arial,Helvetica,sans-serif" size="2"><font face="Verdana,Arial,Helvetica,sans-serif" size="2"> </font></font>
<div><font face="Verdana,Arial,Helvetica,sans-serif" size="2"><font face="Verdana,Arial,Helvetica,sans-serif" size="2"> </font></font></div><font face="Verdana,Arial,Helvetica,sans-serif" size="2"><font face="Verdana,Arial,Helvetica,sans-serif" size="2"> </font></font></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
<div dir="ltr"> </div>
<div dir="ltr"> </div>
<blockquote dir="ltr" style="border-left-width:2px;border-left-style:solid;border-left-color:rgb(170,170,170);margin-left:5px;padding-left:5px;direction:ltr;margin-right:0px">----- Original message -----<br>From: Alec <<a href="mailto:anacreo@gmail.com" target="_blank">anacreo@gmail.com</a>><br>Sent by: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a><br>To: <a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a><br>Cc:<br>Subject: [EXTERNAL] [gpfsug-discuss] Spectrum Scale 5 and Reading Compressed Data<br>Date: Wed, Jan 20, 2021 11:10<br> <br>
<div dir="ltr">We have AIX and Spectrum Scale 5.1 and are compressing older data.
<div> </div>
<div>We can compress data at about 10GB/minute and decompress data wicked fast using mmchattr, when a user reads data from a compressed file via application open / read calls.... it moves at about 5MB/s. Normally our I/O pipeline allows for 2400MB/s on a single file read.</div>
<div> </div>
<div>What can we look at to speed up the read of the compressed data, are there any tunables that might affect this?</div>
<div> </div>
<div>As it is now if the backup daemon is backing up a compressed file, it can get stuck for hours, I will go and mmchattr to decompress the file, within a minute the file is decompressed, and backed up, then I simply recompress the file once backup has moved on.</div>
<div> </div>
<div>Any advice on how to improve the compressed reads under AIX would be very helpful.</div>
<div> </div>
<div>Alec</div></div>
<div><font face="Default Monospace,Courier New,Courier,monospace" size="2">_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=USgQqOp8HDCg0DYjdjSVFvVOwq1rMgRYPP_hoZqgUyI&s=_hdEB3EvWW-8ZzdS1D1roh92-AicdrVMywJwQGlKTIQ&e=" target="_blank">https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=USgQqOp8HDCg0DYjdjSVFvVOwq1rMgRYPP_hoZqgUyI&s=_hdEB3EvWW-8ZzdS1D1roh92-AicdrVMywJwQGlKTIQ&e=</a> </font></div></blockquote>
<div dir="ltr"> </div></div>Unless stated otherwise above:<br>
IBM United Kingdom Limited - Registered in England and Wales with number 741598. <br>
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU<br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
</blockquote></div>