<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div dir="ltr" >Hi Olaf,</div>
<div dir="ltr" > </div>
<div dir="ltr" >Thanks and sorry for reply so long. We've been testing several ways to provide this information for the user. Let me give you more details about that.</div>
<div dir="ltr" >There's a corporate SASGRID consolidating several SAS applications from several business areas. All of them are using the same saswork filesystem. So the idea is to provide a way to identify the top processes or users that are doing more I/O in terms of throughput or IOPS. We have tested the following:</div>
<div dir="ltr" >- fileheat and policy engine to identify most active files: We first activated fileheat by executing the command</div>
<div dir="ltr" > # mmchconfig fileHeatLossPercent=25,fileHeatPeriodMinutes=720</div>
<div dir="ltr" > After that the SAS admin started to run a job and the we created the following policy to see if we could detect the corresponding SAS file:</div>
<div dir="ltr" ><div> rule 'fileheatlist' list 'hotfiles' weight(FILE_HEAT)<br> SHOW( HEX( XATTR( 'gpfs.FileHeat' )) ||<br> ' A=' || varchar(ACCESS_TIME) ||<br> ' K=' || varchar(KB_ALLOCATED) ||<br> ' H=' || varchar(FILE_HEAT) ||<br> ' U=' || varchar(USER_ID) ||<br> ' G=' || varchar(GROUP_ID) ||<br> ' FZ=' || varchar(FILE_SIZE) ||<br> ' CT=' || varchar(CREATION_TIME) ||<br> ' CHT=' || varchar(CHANGE_TIME) ||<br> ' M=' || varchar(MODIFICATION_TIME) )</div>
<div> where FILE_HEAT != 0.0</div>
<div> Then, we executed the command:</div>
<div> # mmapplypolicy -P policy-file-heat.txt -I defer -f test1</div>
<div> I don't know why, but always was reporting that zero files were selected. I don't know what´s missing or if that's the way it is.</div>
<div>- Combine mmdiag with a list of files generated by ILM engine: For we get busiest files we executed the following command:</div>
<div> # mmdiag --iohist verbose > mmdiag--iohist_verbose.out</div>
<div> One way to list the top files was this:</div>
<div> # cat mmdiag--iohist_verbose.out | grep data | awk '{print $10}' | uniq -c | sort -nr | head<br> 7 135003<br> 5 135003<br> 3 135003<br> 2 134985<br> 2 134985<br> 1 64171<br> 1 64094<br> 1 64013<br> 1 46465<br> 1 46465</div>
<div> Another one was executing the following command:</div>
<div> # cat mmdiag--iohist_verbose.out | grep data | sort -k6 -nr | head<br>03:12:11.911813 W data 2:132768 8 11.782 cli 0AC3C23C:58AEDD53 10.195.194.60 451799 0 Sync SyncFSWorkerThread<br>03:12:10.927003 W data 1:5410160 8 11.086 cli 0AC3C23C:58091F75 10.195.194.60 46465 1319 Sync SyncFSWorkerThread<br>03:12:11.927521 W data 2:113995072 8 7.602 cli 0AC3C23C:58AEDD53 10.195.194.60 451776 1 Sync SyncFSWorkerThread<br>03:12:10.999507 W data 2:149912432 24 3.830 cli 0AC3C23C:58AEDC8D 10.195.194.60 134985 4 Sync SyncFSWorkerThread<br>03:12:20.190427 W data 1:40854976 8 3.058 cli 0AC3C23C:58091F75 10.195.194.60 64013 0 Sync SyncFSWorkerThread<br>03:12:11.923742 W data 2:182741840 8 3.036 cli 0AC3C23C:58AEDD53 10.195.194.60 385976 0 Sync SyncFSWorkerThread<br>03:12:20.186045 W data 1:41352672 16 2.451 cli 0AC3C23C:58091F1B 10.195.194.60 451774 2 Sync SyncFSWorkerThread<br>03:12:16.139833 W data 2:149912416 24 1.595 cli 0AC3C23C:58AEDC8D 10.195.194.60 134985 4 Cleaner CleanBufferThread<br>03:12:21.544674 W data 3:146654840 8 0.873 cli 0AC3C23C:592334F8 10.195.194.60 451780 0 Sync SyncFSWorkerThread<br>03:12:10.998636 W data 2:149912352 8 0.833 cli 0AC3C23C:58AEDC8D 10.195.194.60 134985 4 Sync SyncFSWorkerThread</div>
<div> For we discover which filesystem that inode number belongs:</div>
<div> # mmlsnsd -L | grep 58AEDD53<br> sasconfig nsdconfig0001 0AC3C23C58AEDD53 host1,host2</div>
<div> Then we could run a policy rule to just list the files, here is the policy:</div>
<div> rule 'fileheatlist' list 'hotfiles' weight(FILE_HEAT))<br> show( ' U=' || varchar(USER_ID) ||<br> ' G=' || varchar(GROUP_ID) ||<br> ' A=' || varchar(ACCESS_TIME) ||<br> ' K=' || varchar(KB_ALLOCATED) ||<br> ' H=' || varchar(computeFileHeat(CURRENT_TIMESTAMP-ACCESS_TIME,xattr('gpfs.FileHeat'),KB_ALLOCATED)) ||<br> ' FZ=' || varchar(FILE_SIZE) ||<br> ' CT=' || varchar(CREATION_TIME) ||<br> ' CHT=' || varchar(CHANGE_TIME) ||<br> ' M=' || varchar(MODIFICATION_TIME) )</div>
<div> # mmapplypolicy sasconfig -P policy-file-heat3.txt -I defer -f teste6</div>
<div> Then we could grep by inode number and see which file it is:</div>
<div> # grep "^451799 " teste6.list.hotfiles</div>
<div> For privacy reasons I won't show the result but it found the file. The good thing this list also provides the UID and GID of the file. We still waiting a feedback from SAS admin to see it's acceptable.</div>
<div>- dstat with <span style="color: rgb(18, 18, 18); font-family: Arial, Helvetica, sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;" >--gpfs-ops --top-io-adv|--top-bio|--top-io: The problem is it only shows one process. That's not enough.</span></div>
<div><span style="color: rgb(18, 18, 18); font-family: Arial, Helvetica, sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;" >- Systemtap: It didn't work. I think it's because there's no GPFS symbols. If somebody know how to add GPFS symbols that can be very handy.</span></div>
<div><span style="color: rgb(18, 18, 18); font-family: Arial, Helvetica, sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;" >- QOS: We first enabled QOS to just collect filesystem statistics:</span></div>
<div><span style="color: rgb(18, 18, 18); font-family: Arial, Helvetica, sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;" > # mmchqos saswork --enable --fine-stats 60 --pid-stats yes</span></div>
<div><span style="color: rgb(18, 18, 18); font-family: Arial, Helvetica, sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;" > The the SAS admin started another SAS job and got the PID. Then we run the following command:</span></div>
<div><span style="color: rgb(18, 18, 18); font-family: Arial, Helvetica, sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;" > # mmlsqos saswork --fine-stats 2 --seconds 60 | grep SASPID</span></div>
<div><span style="color: rgb(18, 18, 18); font-family: Arial, Helvetica, sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;" > We never matched the PIDs. When you get from ps -ef | grep nodms, it return a PID of 5 digits and mmlsqos gives PIDs of 8 digits. We have a ticket opended to understand what's happening.</span></div>
<div> </div>
<div>After all this time trying to figure out a way to generate this report, I think the problem is more complex. Even if we get this information what we could do to put a limit in those processes? I think the best option would have AIX servers running WLM and the saswork filesystems would need to be local on each server. In that way we not only could monitor but define classes, shares and limits for I/O. I think RedHat or Linux in general doesn't have a workload manager like in AIX.</div>
<div> </div>
<div> </div></div>
<div dir="ltr" ><div dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div dir="ltr" ><table cellpadding="0" cellspacing="0" style="border:0;table-layout:auto;white-space:normal;color:#5f5f5f;width:650px;font-size:7.2pt" width="650px" > <tbody> <tr> <td style="vertical-align: bottom;" > <div style="font-size:9pt" ><span style="font-size:1.000em;" >Abraços / Regards / Saludos,</span></div>
<p> </p>
<div style="margin:0 0 8px 0" ><span style="font-size:1.000em;" ><b style="color:#888888;font-size:12pt" ><span>Anderson</span> <span>Nobre</span></b><br> AIX & Power Consultant<br> Master Certified IT Specialist<br> IBM Systems Hardware Client Technical Team – IBM Systems Lab Services<br> <br> <img alt="community_general_lab_services" height="120" src="http://esign.webmaster.ibm.com/img/signature_lab_services.jpg" width="333" ></span></div> </td> <td> </td> </tr> </tbody></table>
<table cellpadding="0" cellspacing="0" style="border:0;table-layout:auto;white-space:normal;color:#5f5f5f;width:650px;font-size:7.2pt" width="650px" > <tbody> <tr> <td colspan="3" > <hr style="border:0;color:#696969;background-color:#696969;height:1px;" ></td> </tr> <tr> <td style="vertical-align: top;font-family:arial,helvetica,sans-serif" width="400px" ><span style="font-size:1.000em;" ><span><span style="color:#466BB0;font-weight:bold" ><abbr title="work" >Phone:</abbr></span> <span>55-19-2132-4317</span></span><br> <span style="color:#466BB0;font-weight:bold;font-style:7.2pt" >E-mail:</span> <a href="mailto:anobre@br.ibm.com" target="_blank"><span style="color:#5f5f5f" >anobre@br.ibm.com</span></a></span></td> <td align="right" style="vertical-align: top;font-family:arial,helvetica,sans-serif" width="150px" ><span style="font-size:1.000em;" ><img alt="IBM" src="http://esign.webmaster.ibm.com/img/ibm3.gif" ></span></td> </tr> </tbody></table></div></div></div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Original message -----<br>From: "Olaf Weiser" <olaf.weiser@de.ibm.com><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Cc:<br>Subject: Re: [gpfsug-discuss] Top files on GPFS filesystem<br>Date: Mon, Aug 13, 2018 3:10 AM<br> <br><font face="sans-serif" size="2" >there's no mm* command to get it cluster wide.. </font><br><font face="sans-serif" size="2" >you can use <i>fileheat </i>and policy engine to identify most active files .. and further more... combine it with migration rules ... to replace those files .. </font><br><font face="sans-serif" size="2" >please note.. files, that are accessed very heavily but all requests answered out of pagepol (cached files) .. fileheat does'nt get increased for cache hits... fileheat is only counted for real IOs to the disk... as intended ...</font><br><br><br><br><br><br><br><font color="#5f5f5f" face="sans-serif" size="1" >From: </font><font face="sans-serif" size="1" >"Anderson Ferreira Nobre" <anobre@br.ibm.com></font><br><font color="#5f5f5f" face="sans-serif" size="1" >To: </font><font face="sans-serif" size="1" >gpfsug-discuss@spectrumscale.org</font><br><font color="#5f5f5f" face="sans-serif" size="1" >Date: </font><font face="sans-serif" size="1" >08/10/2018 08:10 PM</font><br><font color="#5f5f5f" face="sans-serif" size="1" >Subject: </font><font face="sans-serif" size="1" >[gpfsug-discuss] Top files on GPFS filesystem</font><br><font color="#5f5f5f" face="sans-serif" size="1" >Sent by: </font><font face="sans-serif" size="1" >gpfsug-discuss-bounces@spectrumscale.org</font>
<hr noshade="noshade" ><br><br><br><font face="Arial" size="2" >Hi all,</font><br><font face="Arial" size="2" > </font><br><font face="Arial" size="2" >Does anyone know how to list the top files by throughput and IOPS in a single GPFS filesystem like filemon in AIX?</font><br><font face="Arial" size="2" > </font><br><font face="Arial" size="2" > </font>
<table style="border-collapse:collapse;" width="650" > <tbody> <tr height="8" > <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="645" ><font color="#5f5f5f" size="1" >Abraços / Regards / Saludos,</font>
<p><font color="#5f5f5f" size="1" > </font><br> <font color="#8f8f8f" size="3" ><b>Anderson Nobre</b></font><br> <font color="#5f5f5f" size="1" >AIX & Power Consultant<br> Master Certified IT Specialist<br> IBM Systems Hardware Client Technical Team – IBM Systems Lab Services</font><br> <br> <img alt="community_general_lab_services" height="120" src="cid:_2_DBC5F19CDBC5ECBC00214F54C12582E8" style="border:0px solid;" width="333" ></p> </td> <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="4" ><font color="#5f5f5f" size="1" > </font></td> </tr> </tbody></table>
<table style="border-collapse:collapse;" width="650" > <tbody> <tr height="8" > <td colspan="2" style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="650" > <hr></td> </tr> <tr height="8" valign="top" > <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="472" ><font color="#4181c0" face="Arial" size="1" ><b>Phone:</b></font><font color="#5f5f5f" face="Arial" size="1" >55-19-2132-4317</font><br> <font color="#4181c0" face="Arial" size="1" ><b>E-mail:</b></font><font color="#5f5f5f" face="Arial" size="1" > </font><a href="mailto:anobre@br.ibm.com" target="_blank"><font color="#5f5f5f" face="Arial" size="1" ><u>anobre@br.ibm.com</u></font></a></td> <td style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;" width="177" > <div align="right" ><img align="bottom" alt="IBM" height="30" src="cid:_1_DBCF2504DBCF20E800214F54C12582E8" style="border:0px solid;" width="83" ></div> </td> </tr> </tbody></table><br><tt><font face="" size="3" >_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org</font></tt><br><tt><font face="" size="3" ><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></tt><br><br><br>
<div><font face="Default Monospace,Courier New,Courier,monospace" size="2" >_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></div></blockquote>
<div dir="ltr" > </div></div></div>
<div dir="ltr" ><img alt="Image._2_DBC5F19CDBC5ECBC00214F54C12582E8.jpg" border="0" src="cid:1536071547526146" ><img alt="Image._1_DBCF2504DBCF20E800214F54C12582E8.gif" border="0" src="cid:1536071547526147" ></div></div>
<BR>