<font size=2 face="sans-serif">hmm ...limiting IO resources per file /
file based . .is really not an easy thing to do ... </font><br><font size=2 face="sans-serif">please consider open a PMR and
tell me the number.... there is an undocumented parameter , that one could
limit the buffer per file .. so indirectly , you can influence the #threads
working per file .. </font><br><font size=2 face="sans-serif">but this change will affect all files
on that node ... </font><br><font size=2 face="sans-serif">I made some good experiences with that
for SAS, which is here mostly read driven ( running on RHEL )</font><br><div><br><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">"Anderson Ferreira
Nobre" <anobre@br.ibm.com></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug-discuss@spectrumscale.org</font><br><font size=1 color=#5f5f5f face="sans-serif">Cc:
</font><font size=1 face="sans-serif">gpfsug-discuss@spectrumscale.org</font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">09/04/2018 09:40 PM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
Top files on GPFS filesystem</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><font size=2 face="Arial">Hi Olaf,</font><br><font size=2 face="Arial"> </font><br><font size=2 face="Arial">Thanks and sorry for reply so long. We've
been testing several ways to provide this information for the user. Let
me give you more details about that.</font><br><font size=2 face="Arial">There's a corporate SASGRID consolidating
several SAS applications from several business areas. All of them are using
the same saswork filesystem. So the idea is to provide a way to identify
the top processes or users that are doing more I/O in terms of throughput
or IOPS. We have tested the following:</font><br><font size=2 face="Arial">- fileheat and policy engine to identify
most active files: We first activated fileheat by executing the command</font><br><font size=2 face="Arial"> # mmchconfig fileHeatLossPercent=25,fileHeatPeriodMinutes=720</font><br><font size=2 face="Arial"> After that the SAS admin started
to run a job and the we created the following policy to see if we could
detect the corresponding SAS file:</font><br><font size=2 face="Arial"> rule 'fileheatlist' list 'hotfiles'
weight(FILE_HEAT)<br> SHOW( HEX( XATTR( 'gpfs.FileHeat' )) ||<br> ' A=' || varchar(ACCESS_TIME) ||<br> ' K=' || varchar(KB_ALLOCATED) ||<br> ' H=' || varchar(FILE_HEAT) ||<br> ' U=' || varchar(USER_ID) ||<br> ' G=' || varchar(GROUP_ID) ||<br> ' FZ=' || varchar(FILE_SIZE) ||<br> ' CT=' || varchar(CREATION_TIME) ||<br> ' CHT=' || varchar(CHANGE_TIME) ||<br> ' M=' || varchar(MODIFICATION_TIME) )</font><br><font size=2 face="Arial"> where FILE_HEAT != 0.0</font><br><font size=2 face="Arial"> Then, we executed the command:</font><br><font size=2 face="Arial"> # mmapplypolicy -P policy-file-heat.txt
-I defer -f test1</font><br><font size=2 face="Arial"> I don't know why, but always was reporting
that zero files were selected. I don't know what´s missing or if that's
the way it is.</font><br><font size=2 face="Arial">- Combine mmdiag with a list of files generated
by ILM engine: For we get busiest files we executed the following command:</font><br><font size=2 face="Arial"> # mmdiag --iohist verbose > mmdiag--iohist_verbose.out</font><br><font size=2 face="Arial"> One way to list the top files was
this:</font><br><font size=2 face="Arial"> # cat mmdiag--iohist_verbose.out |
grep data | awk '{print $10}' | uniq -c | sort -nr | head<br> 7 135003<br> 5 135003<br> 3 135003<br> 2 134985<br> 2 134985<br> 1 64171<br> 1 64094<br> 1 64013<br> 1 46465<br> 1 46465</font><br><font size=2 face="Arial"> Another one was executing the following
command:</font><br><font size=2 face="Arial"> # cat mmdiag--iohist_verbose.out |
grep data | sort -k6 -nr | head<br>03:12:11.911813 W data 2:132768
8 11.782 cli 0AC3C23C:58AEDD53
10.195.194.60 451799
0 Sync SyncFSWorkerThread<br>03:12:10.927003 W data 1:5410160
8 11.086 cli 0AC3C23C:58091F75
10.195.194.60 46465 1319
Sync SyncFSWorkerThread<br>03:12:11.927521 W data 2:113995072
8 7.602 cli 0AC3C23C:58AEDD53
10.195.194.60 451776
1 Sync SyncFSWorkerThread<br>03:12:10.999507 W data 2:149912432
24 3.830 cli 0AC3C23C:58AEDC8D
10.195.194.60 134985
4 Sync SyncFSWorkerThread<br>03:12:20.190427 W data 1:40854976
8 3.058 cli 0AC3C23C:58091F75
10.195.194.60 64013
0 Sync SyncFSWorkerThread<br>03:12:11.923742 W data 2:182741840
8 3.036 cli 0AC3C23C:58AEDD53
10.195.194.60 385976
0 Sync SyncFSWorkerThread<br>03:12:20.186045 W data 1:41352672
16 2.451 cli 0AC3C23C:58091F1B
10.195.194.60 451774
2 Sync SyncFSWorkerThread<br>03:12:16.139833 W data 2:149912416
24 1.595 cli 0AC3C23C:58AEDC8D
10.195.194.60 134985
4 Cleaner CleanBufferThread<br>03:12:21.544674 W data 3:146654840
8 0.873 cli 0AC3C23C:592334F8
10.195.194.60 451780
0 Sync SyncFSWorkerThread<br>03:12:10.998636 W data 2:149912352
8 0.833 cli 0AC3C23C:58AEDC8D
10.195.194.60 134985
4 Sync SyncFSWorkerThread</font><br><font size=2 face="Arial"> For we discover which filesystem that
inode number belongs:</font><br><font size=2 face="Arial"> # mmlsnsd -L | grep 58AEDD53<br> sasconfig nsdconfig0001 0AC3C23C58AEDD53 host1,host2</font><br><font size=2 face="Arial"> Then we could run a policy rule to
just list the files, here is the policy:</font><br><font size=2 face="Arial"> rule 'fileheatlist' list 'hotfiles'
weight(FILE_HEAT))<br> show( ' U=' || varchar(USER_ID) ||<br> ' G=' || varchar(GROUP_ID) ||<br> ' A=' || varchar(ACCESS_TIME)
||<br> ' K=' || varchar(KB_ALLOCATED)
||<br> ' H=' || varchar(computeFileHeat(CURRENT_TIMESTAMP-ACCESS_TIME,xattr('gpfs.FileHeat'),KB_ALLOCATED))
||<br> ' FZ=' || varchar(FILE_SIZE)
||<br> ' CT=' || varchar(CREATION_TIME)
||<br> ' CHT=' || varchar(CHANGE_TIME)
||<br> ' M=' || varchar(MODIFICATION_TIME)
)</font><br><font size=2 face="Arial"> # mmapplypolicy sasconfig -P policy-file-heat3.txt
-I defer -f teste6</font><br><font size=2 face="Arial"> Then we could grep by inode number
and see which file it is:</font><br><font size=2 face="Arial"> # grep "^451799 " teste6.list.hotfiles</font><br><font size=2 face="Arial"> For privacy reasons I won't show the
result but it found the file. The good thing this list also provides the
UID and GID of the file. We still waiting a feedback from SAS admin to
see it's acceptable.</font><br><font size=2 face="Arial">- dstat with --gpfs-ops --top-io-adv|--top-bio|--top-io:
The problem is it only shows one process. That's not enough.</font><br><font size=2 face="Arial">- Systemtap: It didn't work. I think it's
because there's no GPFS symbols. If somebody know how to add GPFS symbols
that can be very handy.</font><br><font size=2 face="Arial">- QOS: We first enabled QOS to just collect
filesystem statistics:</font><br><font size=2 face="Arial"> # mmchqos saswork --enable --fine-stats
60 --pid-stats yes</font><br><font size=2 face="Arial"> The the SAS admin started another
SAS job and got the PID. Then we run the following command:</font><br><font size=2 face="Arial"> # mmlsqos saswork --fine-stats 2 --seconds
60 | grep SASPID</font><br><font size=2 face="Arial"> We never matched the PIDs. When you
get from ps -ef | grep nodms, it return a PID of 5 digits and mmlsqos gives
PIDs of 8 digits. We have a ticket opended to understand what's happening.</font><br><font size=2 face="Arial"> </font><br><font size=2 face="Arial">After all this time trying to figure out
a way to generate this report, I think the problem is more complex. Even
if we get this information what we could do to put a limit in those processes?
I think the best option would have AIX servers running WLM and the saswork
filesystems would need to be local on each server. In that way we not only
could monitor but define classes, shares and limits for I/O. I think RedHat
or Linux in general doesn't have a workload manager like in AIX.</font><br><font size=2 face="Arial"> </font><br><font size=2 face="Arial"> </font><table width=650 style="border-collapse:collapse;"><tr height=8><td width=645 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#5f5f5f>Abraços
/ Regards / Saludos,</font><p><font size=1 color=#5f5f5f> </font><br><font size=3 color=#8f8f8f><b>Anderson Nobre</b></font><font size=1 color=#5f5f5f><br>AIX & Power Consultant<br>Master Certified IT Specialist<br>IBM Systems Hardware Client Technical Team – IBM Systems Lab Services<br><br></font><img src=cid:_2_DDEEC03CDDEEBB8C0051A029C12582FF width=333 height=120 alt=community_general_lab_services style="border:0px solid;"><td width=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#5f5f5f> </font></table><br><table width=650 style="border-collapse:collapse;"><tr height=8><td width=650 colspan=2 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><hr><tr valign=top height=8><td width=472 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#4181c0 face="Arial"><b>Phone:</b></font><font size=1 color=#5f5f5f face="Arial">55-19-2132-4317</font><font size=1 color=#4181c0 face="Arial"><b><br>E-mail:</b></font><font size=1 color=#5f5f5f face="Arial"> </font><a href="mailto:anobre@br.ibm.com" target="_blank"><font size=1 color=#5f5f5f face="Arial"><u>anobre@br.ibm.com</u></font></a><td width=177 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><div align=right><img align=bottom src=cid:_1_DDE8F8F0DDEEE35C0051A029C12582FF width=83 height=30 alt=IBM style="border:0px solid;"></div></table><br><font size=2 face="Arial"> </font><br><font size=2 face="Arial"> </font><br><font size=2 face="Arial">----- Original message -----<br>From: "Olaf Weiser" <olaf.weiser@de.ibm.com><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Cc:<br>Subject: Re: [gpfsug-discuss] Top files on GPFS filesystem<br>Date: Mon, Aug 13, 2018 3:10 AM<br> </font><font size=2 face="sans-serif"><br>there's no mm* command to get it cluster wide.. <br>you can use <i>fileheat </i>and policy engine to identify most active files
.. and further more... combine it with migration rules ... to replace
those files .. <br>please note.. files, that are accessed very heavily but all requests answered
out of pagepol (cached files) .. fileheat does'nt get increased for cache
hits... fileheat is only counted for real IOs to the disk... as intended
...</font><font size=2 face="Arial"><br><br><br><br><br><br></font><font size=1 color=#5f5f5f face="sans-serif"><br>From: </font><font size=1 face="sans-serif">"Anderson
Ferreira Nobre" <anobre@br.ibm.com></font><font size=1 color=#5f5f5f face="sans-serif"><br>To: </font><font size=1 face="sans-serif">gpfsug-discuss@spectrumscale.org</font><font size=1 color=#5f5f5f face="sans-serif"><br>Date: </font><font size=1 face="sans-serif">08/10/2018
08:10 PM</font><font size=1 color=#5f5f5f face="sans-serif"><br>Subject: </font><font size=1 face="sans-serif">[gpfsug-discuss]
Top files on GPFS filesystem</font><font size=1 color=#5f5f5f face="sans-serif"><br>Sent by: </font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><font size=2 face="Arial"></font><hr noshade><font size=2 face="Arial"><br><br></font><font size=2 face="Arial"><br>Hi all,<br> <br>Does anyone know how to list the top files by throughput and IOPS in a
single GPFS filesystem like filemon in AIX?<br> <br> </font><font size=2 face="Arial"> </font><table width=650 style="border-collapse:collapse;"><tr height=8><td width=641 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><font size=1 color=#5f5f5f>Abraços
/ Regards / Saludos,</font><font size=3> </font><p><font size=1 color=#5f5f5f> </font><font size=3 color=#8f8f8f><b><br>Anderson Nobre</b></font><font size=1 color=#5f5f5f><br>AIX & Power Consultant<br>Master Certified IT Specialist<br>IBM Systems Hardware Client Technical Team – IBM Systems Lab Services</font><font size=3><br><br></font><img src=cid:_2_DDE9189CDDE913540051A029C12582FF width=333 height=120 alt=community_general_lab_services style="border:0px solid;"><td width=5 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><font size=1 color=#5f5f5f> </font></table><br><font size=2 face="Arial"> </font><table width=650 style="border-collapse:collapse;"><tr height=8><td width=648 colspan=2 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><hr><tr valign=top height=8><td width=469 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><font size=1 color=#4181c0 face="Arial"><b>Phone:</b></font><font size=1 color=#5f5f5f face="Arial">55-19-2132-4317</font><font size=1 color=#4181c0 face="Arial"><b><br>E-mail:</b></font><font size=1 color=#5f5f5f face="Arial"> </font><a href="mailto:anobre@br.ibm.com" target="_blank"><font size=1 color=#5f5f5f face="Arial"><u>anobre@br.ibm.com</u></font></a><td width=176 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><div align=right><img align=bottom src=cid:_1_DDE93E04DDE939E80051A02AC12582FF alt=IBM style="border:0px solid;"></div></table><br><tt><font size=3><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org</font></tt><tt><font size=3 color=blue><u><br></u></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><tt><font size=3 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></tt></a><font size=2 face="Arial"><br><br><br> </font><br><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org</font></tt><tt><font size=2 color=blue><u><br></u></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><tt><font size=2 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></tt></a><br><font size=2 face="Arial"> </font><br><img src=cid:_2_DDE95060DDE94DD80051A02AC12582FF width=386 height=100 alt=Image._2_DBC5F19CDBC5ECBC00214F54C12582E8.jpg style="border:0px solid;"><img src=cid:_1_DDE95288DDE94DD80051A02AC12582FF width=83 height=30 alt=Image._1_DBCF2504DBCF20E800214F54C12582E8.gif style="border:0px solid;"><br><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br></font></tt><br><br></div><BR>