<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div dir="ltr" >Thanks Olaf!!! I've sent an email to mramos open a PMR for this case.</div>
<div dir="ltr" > </div>
<div dir="ltr" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div dir="ltr" ><div> </div>
<table cellpadding="0" cellspacing="0" style="border:0;table-layout:auto;white-space:normal;color:#5f5f5f;width:650px;font-size:7.2pt" width="650px" > <tbody> <tr> <td style="vertical-align: bottom;" > <div style="font-size:9pt" ><span style="font-size:1.000em;" >Abraços / Regards / Saludos,</span></div>
<p> </p>
<div class="vcard" style="margin:0 0 8px 0" ><span style="font-size:1.000em;" ><b class="fn n" style="color:#888888;font-size:12pt" ><span class="given-name" >Anderson</span> <span class="family-name" >Nobre</span></b><br> AIX & Power Consultant<br> Master Certified IT Specialist<br> IBM Systems Hardware Client Technical Team – IBM Systems Lab Services<br> <br> <img alt="community_general_lab_services" class="org" height="120" src="http://esign.webmaster.ibm.com/img/signature_lab_services.jpg" width="333" ></span></div> </td> <td> </td> </tr> </tbody></table>
<table cellpadding="0" cellspacing="0" style="border:0;table-layout:auto;white-space:normal;color:#5f5f5f;width:650px;font-size:7.2pt" width="650px" > <tbody> <tr> <td colspan="3" > <hr style="border:0;color:#696969;background-color:#696969;height:1px;" ></td> </tr> <tr> <td style="vertical-align: top;font-family:arial,helvetica,sans-serif" width="400px" ><span style="font-size:1.000em;" ><span class="tel" ><span style="color:#466BB0;font-weight:bold" ><abbr class="type" title="work" >Phone:</abbr></span> <span class="value" >55-19-2132-4317</span></span><br> <span style="color:#466BB0;font-weight:bold;font-style:7.2pt" >E-mail:</span> <a class="email" href="mailto:anobre@br.ibm.com" target="_blank"><span class="value" style="color:#5f5f5f" >anobre@br.ibm.com</span></a></span></td> <td align="right" style="vertical-align: top;font-family:arial,helvetica,sans-serif" width="150px" ><span style="font-size:1.000em;" ><img alt="IBM" class="org" src="http://esign.webmaster.ibm.com/img/ibm3.gif" ></span></td> </tr> </tbody></table></div></div></div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Original message -----<br>From: "Olaf Weiser" <olaf.weiser@de.ibm.com><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Cc:<br>Subject: Re: [gpfsug-discuss] Top files on GPFS filesystem<br>Date: Wed, Sep 5, 2018 12:02 PM<br> <br><font face="sans-serif" size="2" >hmm ...limiting IO resources per file / file based . .is really not an easy thing to do ... </font><br><font face="sans-serif" size="2" >please consider open a PMR and tell me the number.... there is an undocumented parameter , that one could limit the buffer per file .. so indirectly , you can influence the #threads working per file .. </font><br><font face="sans-serif" size="2" >but this change will affect all files on that node ... </font><br><font face="sans-serif" size="2" >I made some good experiences with that for SAS, which is here mostly read driven ( running on RHEL )</font>
<div><br><br><br><br><br><font color="#5f5f5f" face="sans-serif" size="1" >From: </font><font face="sans-serif" size="1" >"Anderson Ferreira Nobre" <anobre@br.ibm.com></font><br><font color="#5f5f5f" face="sans-serif" size="1" >To: </font><font face="sans-serif" size="1" >gpfsug-discuss@spectrumscale.org</font><br><font color="#5f5f5f" face="sans-serif" size="1" >Cc: </font><font face="sans-serif" size="1" >gpfsug-discuss@spectrumscale.org</font><br><font color="#5f5f5f" face="sans-serif" size="1" >Date: </font><font face="sans-serif" size="1" >09/04/2018 09:40 PM</font><br><font color="#5f5f5f" face="sans-serif" size="1" >Subject: </font><font face="sans-serif" size="1" >Re: [gpfsug-discuss] Top files on GPFS filesystem</font><br><font color="#5f5f5f" face="sans-serif" size="1" >Sent by: </font><font face="sans-serif" size="1" >gpfsug-discuss-bounces@spectrumscale.org</font>
<hr noshade="noshade" ><br><br><br><font face="Arial" size="2" >Hi Olaf,</font><br><font face="Arial" size="2" > </font><br><font face="Arial" size="2" >Thanks and sorry for reply so long. We've been testing several ways to provide this information for the user. Let me give you more details about that.</font><br><font face="Arial" size="2" >There's a corporate SASGRID consolidating several SAS applications from several business areas. All of them are using the same saswork filesystem. So the idea is to provide a way to identify the top processes or users that are doing more I/O in terms of throughput or IOPS. We have tested the following:</font><br><font face="Arial" size="2" >- fileheat and policy engine to identify most active files: We first activated fileheat by executing the command</font><br><font face="Arial" size="2" > # mmchconfig fileHeatLossPercent=25,fileHeatPeriodMinutes=720</font><br><font face="Arial" size="2" > After that the SAS admin started to run a job and the we created the following policy to see if we could detect the corresponding SAS file:</font><br><font face="Arial" size="2" > rule 'fileheatlist' list 'hotfiles' weight(FILE_HEAT)<br> SHOW( HEX( XATTR( 'gpfs.FileHeat' )) ||<br> ' A=' || varchar(ACCESS_TIME) ||<br> ' K=' || varchar(KB_ALLOCATED) ||<br> ' H=' || varchar(FILE_HEAT) ||<br> ' U=' || varchar(USER_ID) ||<br> ' G=' || varchar(GROUP_ID) ||<br> ' FZ=' || varchar(FILE_SIZE) ||<br> ' CT=' || varchar(CREATION_TIME) ||<br> ' CHT=' || varchar(CHANGE_TIME) ||<br> ' M=' || varchar(MODIFICATION_TIME) )</font><br><font face="Arial" size="2" > where FILE_HEAT != 0.0</font><br><font face="Arial" size="2" > Then, we executed the command:</font><br><font face="Arial" size="2" > # mmapplypolicy -P policy-file-heat.txt -I defer -f test1</font><br><font face="Arial" size="2" > I don't know why, but always was reporting that zero files were selected. I don't know what´s missing or if that's the way it is.</font><br><font face="Arial" size="2" >- Combine mmdiag with a list of files generated by ILM engine: For we get busiest files we executed the following command:</font><br><font face="Arial" size="2" > # mmdiag --iohist verbose > mmdiag--iohist_verbose.out</font><br><font face="Arial" size="2" > One way to list the top files was this:</font><br><font face="Arial" size="2" > # cat mmdiag--iohist_verbose.out | grep data | awk '{print $10}' | uniq -c | sort -nr | head<br> 7 135003<br> 5 135003<br> 3 135003<br> 2 134985<br> 2 134985<br> 1 64171<br> 1 64094<br> 1 64013<br> 1 46465<br> 1 46465</font><br><font face="Arial" size="2" > Another one was executing the following command:</font><br><font face="Arial" size="2" > # cat mmdiag--iohist_verbose.out | grep data | sort -k6 -nr | head<br>03:12:11.911813 W data 2:132768 8 11.782 cli 0AC3C23C:58AEDD53 10.195.194.60 451799 0 Sync SyncFSWorkerThread<br>03:12:10.927003 W data 1:5410160 8 11.086 cli 0AC3C23C:58091F75 10.195.194.60 46465 1319 Sync SyncFSWorkerThread<br>03:12:11.927521 W data 2:113995072 8 7.602 cli 0AC3C23C:58AEDD53 10.195.194.60 451776 1 Sync SyncFSWorkerThread<br>03:12:10.999507 W data 2:149912432 24 3.830 cli 0AC3C23C:58AEDC8D 10.195.194.60 134985 4 Sync SyncFSWorkerThread<br>03:12:20.190427 W data 1:40854976 8 3.058 cli 0AC3C23C:58091F75 10.195.194.60 64013 0 Sync SyncFSWorkerThread<br>03:12:11.923742 W data 2:182741840 8 3.036 cli 0AC3C23C:58AEDD53 10.195.194.60 385976 0 Sync SyncFSWorkerThread<br>03:12:20.186045 W data 1:41352672 16 2.451 cli 0AC3C23C:58091F1B 10.195.194.60 451774 2 Sync SyncFSWorkerThread<br>03:12:16.139833 W data 2:149912416 24 1.595 cli 0AC3C23C:58AEDC8D 10.195.194.60 134985 4 Cleaner CleanBufferThread<br>03:12:21.544674 W data 3:146654840 8 0.873 cli 0AC3C23C:592334F8 10.195.194.60 451780 0 Sync SyncFSWorkerThread<br>03:12:10.998636 W data 2:149912352 8 0.833 cli 0AC3C23C:58AEDC8D 10.195.194.60 134985 4 Sync SyncFSWorkerThread</font><br><font face="Arial" size="2" > For we discover which filesystem that inode number belongs:</font><br><font face="Arial" size="2" > # mmlsnsd -L | grep 58AEDD53<br>sasconfig nsdconfig0001 0AC3C23C58AEDD53 host1,host2</font><br><font face="Arial" size="2" > Then we could run a policy rule to just list the files, here is the policy:</font><br><font face="Arial" size="2" > rule 'fileheatlist' list 'hotfiles' weight(FILE_HEAT))<br> show( ' U=' || varchar(USER_ID) ||<br> ' G=' || varchar(GROUP_ID) ||<br> ' A=' || varchar(ACCESS_TIME) ||<br> ' K=' || varchar(KB_ALLOCATED) ||<br> ' H=' || varchar(computeFileHeat(CURRENT_TIMESTAMP-ACCESS_TIME,xattr('gpfs.FileHeat'),KB_ALLOCATED)) ||<br> ' FZ=' || varchar(FILE_SIZE) ||<br> ' CT=' || varchar(CREATION_TIME) ||<br> ' CHT=' || varchar(CHANGE_TIME) ||<br> ' M=' || varchar(MODIFICATION_TIME) )</font><br><font face="Arial" size="2" > # mmapplypolicy sasconfig -P policy-file-heat3.txt -I defer -f teste6</font><br><font face="Arial" size="2" > Then we could grep by inode number and see which file it is:</font><br><font face="Arial" size="2" > # grep "^451799 " teste6.list.hotfiles</font><br><font face="Arial" size="2" > For privacy reasons I won't show the result but it found the file. The good thing this list also provides the UID and GID of the file. We still waiting a feedback from SAS admin to see it's acceptable.</font><br><font face="Arial" size="2" >- dstat with --gpfs-ops --top-io-adv|--top-bio|--top-io: The problem is it only shows one process. That's not enough.</font><br><font face="Arial" size="2" >- Systemtap: It didn't work. I think it's because there's no GPFS symbols. If somebody know how to add GPFS symbols that can be very handy.</font><br><font face="Arial" size="2" >- QOS: We first enabled QOS to just collect filesystem statistics:</font><br><font face="Arial" size="2" > # mmchqos saswork --enable --fine-stats 60 --pid-stats yes</font><br><font face="Arial" size="2" > The the SAS admin started another SAS job and got the PID. Then we run the following command:</font><br><font face="Arial" size="2" > # mmlsqos saswork --fine-stats 2 --seconds 60 | grep SASPID</font><br><font face="Arial" size="2" > We never matched the PIDs. When you get from ps -ef | grep nodms, it return a PID of 5 digits and mmlsqos gives PIDs of 8 digits. We have a ticket opended to understand what's happening.</font><br><font face="Arial" size="2" > </font><br><font face="Arial" size="2" >After all this time trying to figure out a way to generate this report, I think the problem is more complex. Even if we get this information what we could do to put a limit in those processes? I think the best option would have AIX servers running WLM and the saswork filesystems would need to be local on each server. In that way we not only could monitor but define classes, shares and limits for I/O. I think RedHat or Linux in general doesn't have a workload manager like in AIX.</font><br><font face="Arial" size="2" > </font><br><font face="Arial" size="2" > </font>
<table style="border-collapse:collapse;" width="650" > <tbody> <tr height="8" > <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="645" ><font color="#5f5f5f" size="1" >Abraços / Regards / Saludos,</font>
<p><font color="#5f5f5f" size="1" > </font><br> <font color="#8f8f8f" size="3" ><b>Anderson Nobre</b></font><br> <font color="#5f5f5f" size="1" >AIX & Power Consultant<br> Master Certified IT Specialist<br> IBM Systems Hardware Client Technical Team – IBM Systems Lab Services</font><br> <br> <img alt="community_general_lab_services" height="120" src="cid:_2_DDEEC03CDDEEBB8C0051A029C12582FF" style="border:0px solid;" width="333" ></p> </td> <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="4" ><font color="#5f5f5f" size="1" > </font></td> </tr> </tbody></table>
<table style="border-collapse:collapse;" width="650" > <tbody> <tr height="8" > <td colspan="2" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="650" > <hr></td> </tr> <tr height="8" valign="top" > <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="472" ><font color="#4181c0" face="Arial" size="1" ><b>Phone:</b></font><font color="#5f5f5f" face="Arial" size="1" >55-19-2132-4317</font><br> <font color="#4181c0" face="Arial" size="1" ><b>E-mail:</b></font><font color="#5f5f5f" face="Arial" size="1" > </font><a href="mailto:anobre@br.ibm.com" target="_blank"><font color="#5f5f5f" face="Arial" size="1" ><u>anobre@br.ibm.com</u></font></a></td> <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="177" > <div align="right" ><img align="bottom" alt="IBM" height="30" src="cid:_1_DDE8F8F0DDEEE35C0051A029C12582FF" style="border:0px solid;" width="83" ></div> </td> </tr> </tbody></table><br><font face="Arial" size="2" > </font><br><font face="Arial" size="2" > </font><br><font face="Arial" size="2" >----- Original message -----<br>From: "Olaf Weiser" <olaf.weiser@de.ibm.com><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Cc:<br>Subject: Re: [gpfsug-discuss] Top files on GPFS filesystem<br>Date: Mon, Aug 13, 2018 3:10 AM</font><br><br><font face="sans-serif" size="2" >there's no mm* command to get it cluster wide..<br>you can use <i>fileheat </i>and policy engine to identify most active files .. and further more... combine it with migration rules ... to replace those files ..<br>please note.. files, that are accessed very heavily but all requests answered out of pagepol (cached files) .. fileheat does'nt get increased for cache hits... fileheat is only counted for real IOs to the disk... as intended ...</font><br><br><br><br><br><br><br><font color="#5f5f5f" face="sans-serif" size="1" >From: </font><font face="sans-serif" size="1" >"Anderson Ferreira Nobre" <anobre@br.ibm.com></font><br><font color="#5f5f5f" face="sans-serif" size="1" >To: </font><font face="sans-serif" size="1" >gpfsug-discuss@spectrumscale.org</font><br><font color="#5f5f5f" face="sans-serif" size="1" >Date: </font><font face="sans-serif" size="1" >08/10/2018 08:10 PM</font><br><font color="#5f5f5f" face="sans-serif" size="1" >Subject: </font><font face="sans-serif" size="1" >[gpfsug-discuss] Top files on GPFS filesystem</font><br><font color="#5f5f5f" face="sans-serif" size="1" >Sent by: </font><font face="sans-serif" size="1" >gpfsug-discuss-bounces@spectrumscale.org</font>
<hr noshade="noshade" ><br><br><br><font face="Arial" size="2" >Hi all,<br><br>Does anyone know how to list the top files by throughput and IOPS in a single GPFS filesystem like filemon in AIX?</font><br><br><font face="Arial" size="2" > </font>
<table style="border-collapse:collapse;" width="650" > <tbody> <tr height="8" > <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;" width="641" ><font color="#5f5f5f" size="1" >Abraços / Regards / Saludos,</font><font size="3" > </font>
<p><font color="#5f5f5f" size="1" > </font><br> <font color="#8f8f8f" size="3" ><b>Anderson Nobre</b></font><br> <font color="#5f5f5f" size="1" >AIX & Power Consultant<br> Master Certified IT Specialist<br> IBM Systems Hardware Client Technical Team – IBM Systems Lab Services</font><br> <br> <img alt="community_general_lab_services" height="120" src="cid:_2_DDE9189CDDE913540051A029C12582FF" style="border:0px solid;" width="333" ></p> </td> <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;" width="5" ><font color="#5f5f5f" size="1" > </font></td> </tr> </tbody></table><br><font face="Arial" size="2" > </font>
<table style="border-collapse:collapse;" width="650" > <tbody> <tr height="8" > <td colspan="2" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;" width="648" > <hr></td> </tr> <tr height="8" valign="top" > <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;" width="469" ><font color="#4181c0" face="Arial" size="1" ><b>Phone:</b></font><font color="#5f5f5f" face="Arial" size="1" >55-19-2132-4317</font><br> <font color="#4181c0" face="Arial" size="1" ><b>E-mail:</b></font><font color="#5f5f5f" face="Arial" size="1" > </font><a href="mailto:anobre@br.ibm.com" target="_blank"><font color="#5f5f5f" face="Arial" size="1" ><u>anobre@br.ibm.com</u></font></a></td> <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;" width="176" > <div align="right" ><img align="bottom" alt="IBM" src="cid:_1_DDE93E04DDE939E80051A02AC12582FF" style="border:0px solid;" ></div> </td> </tr> </tbody></table><br><br><tt><font face="" size="3" >_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org</font></tt><br><tt><font color="blue" face="" size="3" ><u><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></u></font></tt><br><br><br><font face="Arial" size="2" > </font><br><tt><font face="" size="3" >_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org</font></tt><br><tt><font color="blue" face="" size="3" ><u><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></u></font></tt><br><font face="Arial" size="2" > </font><br><img alt="Image._2_DBC5F19CDBC5ECBC00214F54C12582E8.jpg" height="100" src="cid:_2_DDE95060DDE94DD80051A02AC12582FF" style="border:0px solid;" width="386" ><img alt="Image._1_DBCF2504DBCF20E800214F54C12582E8.gif" height="30" src="cid:_1_DDE95288DDE94DD80051A02AC12582FF" style="border:0px solid;" width="83" ><br><tt><font face="" size="3" >_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org</font></tt><br><tt><font face="" size="3" ><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></tt><br><br> </div>
<div><font face="Default Monospace,Courier New,Courier,monospace" size="2" >_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></div></blockquote>
<div dir="ltr" > </div></div>
<BR>