[gpfsug-discuss] Fileheat - does work! Complete test/example provided here.
Jan-Frode Myklebust
janfrode at tanso.net
Tue Sep 3 14:07:44 BST 2019
Thanks for this example, very userful, but I'm still struggeling a bit at a
customer..
We're doing heat daily based rebalancing, with fileheatlosspercent=20 and
fileheatperiodminutes=720:
RULE "defineTiers" GROUP POOL 'Tiers'
IS 'ssdpool' LIMIT(70)
then 'saspool'
RULE 'Rebalance' MIGRATE FROM POOL 'Tiers' TO POOL 'Tiers'
WEIGHT(FILE_HEAT) WHERE FILE_SIZE<10000000000
but are seeing too many files moved down to the saspool and too few are
staying in the ssdpool. Right now we ran a test of this policy, and saw
that it wanted to move 130k files / 300 GB down to the saspool, and a
single small file up to the ssdpool -- even though the ssdpool is only 50%
utilized.
Running your listing policy reveals lots of files with zero heat:
<7> /gpfs/gpfs0/file1 RULE 'fh2' LIST 'fh' WEIGHT(0.000000) SHOW(
_NULL_ _NULL_ _NULL_ +0.00000000000000E+000 _NULL_ 720 25 server.locale)
<7> /gpfs/gpfs0/file2 RULE 'fh2' LIST 'fh' WEIGHT(0.000000) SHOW(
_NULL_ _NULL_ _NULL_ +0.00000000000000E+000 _NULL_ 720 25 server.locale)
<7> /gpfs/gpfs0/file3/HM_WVS_8P41017_1/HM_WVS_8P41017_1.S2206 RULE
'fh2' LIST 'fh' WEIGHT(0.000000) SHOW( _NULL_ _NULL_ _NULL_
+0.00000000000000E+000 _NULL_ 720 25 server.locale)
and others with heat:
<5> /gpfs/gpfs0/file4 RULE 'fh2' LIST 'fh' WEIGHT(0.004246) SHOW(
300401047 0 0 +4.24600492924153E-003 11E7C19700000000 720 25 server.locale)
<5> /gpfs/gpfs0/file5 RULE 'fh2' LIST 'fh' WEIGHT(0.001717) SHOW(
120971793 1 0 +1.71725239616613E-003 0735E21100010000 720 25 server.locale)
These are not new files -- so we're wondering if maybe the fileheat is
reduced to zero/NULL after a while (how many times can it shrink by 25%
before it's zero??).
Would it make sense to increase fileheatperiodeminutes and/or decrease
fileheatlosspercentage? What would be good values? (BTW: we have relatime
enabled)
Any other ideas for why it won't fill up our ssdpool to close to LIMIT(70) ?
-jf
On Tue, Aug 13, 2019 at 3:33 PM Marc A Kaplan <makaplan at us.ibm.com> wrote:
> Yes, you are correct. It should only be necessary to set
> fileHeatPeriodMinutes, since the loss percent does have a default value.
> But IIRC (I implemented part of this!) you must restart the daemon to get
> those fileheat parameter(s) "loaded"and initialized into the daemon
> processes.
>
> Not fully trusting my memory... I will now "prove" this works today as
> follows:
>
> To test, create and re-read a large file with dd...
>
> [root@/main/gpfs-git]$mmchconfig fileHeatPeriodMinutes=60
> mmchconfig: Command successfully completed
> ...
> [root@/main/gpfs-git]$mmlsconfig | grep -i heat
> fileHeatPeriodMinutes 60
>
> [root@/main/gpfs-git]$mmshutdown
> ...
> [root@/main/gpfs-git]$mmstartup
> ...
> [root@/main/gpfs-git]$mmmount c23
> ...
> [root@/main/gpfs-git]$ls -l /c23/10g
> -rw-r--r--. 1 root root 10737418240 May 16 15:09 /c23/10g
>
> [root@/main/gpfs-git]$mmlsattr -d -X /c23/10g
> file name: /c23/10g
> security.selinux
>
> (NO fileheat attribute yet...)
>
> [root@/main/gpfs-git]$dd if=/c23/10g bs=1M of=/dev/null
> ...
> After the command finishes, you may need to wait a while for the metadata
> to flush to the inode on disk ... or you can force that with an unmount or
> a mmfsctl...
>
> Then the fileheat attribute will appear (I just waited by answering
> another email... No need to do any explicit operations on the file system..)
>
> [root@/main/gpfs-git]$mmlsattr -d -X /c23/10g
> file name: /c23/10g
> security.selinux
> gpfs.FileHeat
>
> To see its hex string value:
>
> [root@/main/gpfs-git]$mmlsattr -d -X -L /c23/10g
> file name: /c23/10g
> ...
> security.selinux:
> 0x756E636F6E66696E65645F753A6F626A6563745F723A756E6C6162656C65645F743A733000
> gpfs.FileHeat: 0x000000EE42A40400
>
> Which will be interpreted by mmapplypolicy...
>
> YES, the interpretation is relative to last access time and current time,
> and done by a policy/sql function "computeFileHeat"
> (You could find this using m4 directives in your policy file...)
>
>
> define([FILE_HEAT],[computeFileHeat(CURRENT_TIMESTAMP-ACCESS_TIME,xattr('gpfs.FileHeat'),KB_ALLOCATED)])
>
> Well gone that far, might as well try mmapplypolicy too....
>
> [root@/main/gpfs-git]$cat /gh/policies/fileheat.policy
> define(DISPLAY_NULL,[CASE WHEN ($1) IS NULL THEN '_NULL_' ELSE varchar($1)
> END])
>
> rule fh1 external list 'fh' exec ''
> rule fh2 list 'fh' weight(FILE_HEAT)
> show(DISPLAY_NULL(xattr_integer('gpfs.FileHeat',1,4,'B')) || ' ' ||
> DISPLAY_NULL(xattr_integer('gpfs.FileHeat',5,2,'B')) || ' ' ||
> DISPLAY_NULL(xattr_integer('gpfs.FileHeat',7,2,'B')) || ' ' ||
> DISPLAY_NULL(FILE_HEAT) || ' ' ||
> DISPLAY_NULL(hex(xattr('gpfs.FileHeat'))) || ' ' ||
> getmmconfig('fileHeatPeriodMinutes') || ' ' ||
> getmmconfig('fileHeatLossPercent') || ' ' ||
> getmmconfig('clusterName') )
>
>
> [root@/main/gpfs-git]$mmapplypolicy /c23 --maxdepth 1 -P
> /gh/policies/fileheat.policy -I test -L 3
> ...
> <1> /c23/10g RULE 'fh2' LIST 'fh' WEIGHT(0.022363) SHOW( 238 17060 1024
> +2.23632812500000E-002 000000EE42A40400 60 10 makaplan.sl.cloud9.ibm.com)
> ...
> WEIGHT(0.022363) LIST 'fh' /c23/10g SHOW(238 17060 1024
> +2.23632812500000E-002 000000EE42A40400 60 10 makaplan.sl.cloud9.ibm.com)
>
>
>
>
> [image: Inactive hide details for Jan-Frode Myklebust ---08/13/2019
> 06:22:46 AM---What about filesystem atime updates. We recently chan]Jan-Frode
> Myklebust ---08/13/2019 06:22:46 AM---What about filesystem atime updates.
> We recently changed the default to «relatime». Could that maybe
>
> From: Jan-Frode Myklebust <janfrode at tanso.net>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Date: 08/13/2019 06:22 AM
> Subject: [EXTERNAL] Re: [gpfsug-discuss] Fileheat
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> ------------------------------
>
>
>
>
> What about filesystem atime updates. We recently changed the default to
> «relatime». Could that maybe influence heat tracking?
>
>
>
> -jf
>
>
> tir. 13. aug. 2019 kl. 11:29 skrev Ulrich Sibiller <
> *u.sibiller at science-computing.de* <u.sibiller at science-computing.de>>:
>
> On 12.08.19 15:38, Marc A Kaplan wrote:
> > My Admin guide says:
> >
> > The loss percentage and period are set via the configuration
> > variables *fileHeatLossPercent *and *fileHeatPeriodMinutes*. By
> default, the file access temperature
> > is not
> > tracked. To use access temperature in policy, the tracking must
> first be enabled. To do this, set
> > the two
> > configuration variables as follows:*
>
> Yes, I am aware of that.
>
> > fileHeatLossPercent*
> > The percentage (between 0 and 100) of file access temperature
> dissipated over the*
> > fileHeatPeriodMinutes *time. The default value is 10.
> > Chapter 25. Information lifecycle management for IBM Spectrum Scale
> *361**
> > fileHeatPeriodMinutes*
> > The number of minutes defined for the recalculation of file access
> temperature. To turn on
> > tracking, *fileHeatPeriodMinutes *must be set to a nonzero value.
> The default value is 0
> >
> >
> > SO Try setting both!
>
> Well, I have not because the documentation explicitly mentions a
> default. What's the point of a
> default if I have to explicitly configure it?
>
> > ALSO to take effect you may have to mmshutdown and mmstartup, at
> least on the (client gpfs) nodes
> > that are accessing the files of interest.
>
> I have now configured both parameters and restarted GPFS. Ran a tar
> over a directory - still no
> change. I will wait for 720minutes and retry (tomorrow).
>
> Thanks
>
> Uli
>
> --
> Science + Computing AG
> Vorstandsvorsitzender/Chairman of the board of management:
> Dr. Martin Matzke
> Vorstand/Board of Management:
> Matthias Schempp, Sabine Hohenstein
> Vorsitzender des Aufsichtsrats/
> Chairman of the Supervisory Board:
> Philippe Miltin
> Aufsichtsrat/Supervisory Board:
> Martin Wibbe, Ursula Morgenstern
> Sitz/Registered Office: Tuebingen
> Registergericht/Registration Court: Stuttgart
> Registernummer/Commercial Register No.: HRB 382196
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at *spectrumscale.org* <http://spectrumscale.org>
> *http://gpfsug.org/mailman/listinfo/gpfsug-discuss*
> <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190903/d0e482ad/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190903/d0e482ad/attachment-0001.gif>
More information about the gpfsug-discuss
mailing list