[gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat"

Alex Chekholko chekh at stanford.edu
Fri Jun 24 19:09:59 BST 2016


On 06/21/2016 10:25 PM, Gaurang Tapase wrote:
> Hello,
>
> Based on inputs from some of the Spectrum Scale users, sharing a blog
> link for "Automatic tiering of object data based on object heat" by Smita.
> https://smitaraut.wordpress.com/2016/06/16/hot-cakes-or-hot-objects-they-better-be-served-fast/
>

Thanks for the blog post.  I wanted to post my experiences:

Note "sata0" is really NL-SAS.

[root at scg-gs0 ~]# df -h|grep gsfs
/dev/gsfs0            4.4P  3.1P  1.4P  70% /srv/gsfs0
[root at scg-gs0 ~]# df -i|grep gsfs
/dev/gsfs0           314572800 234237984 80334816   75% /srv/gsfs0

[root at scg-gs0 ~]# mmlspool gsfs0
Storage pools in file system at '/srv/gsfs0':
Name                    Id   BlkSize Data Meta Total Data in (KB)   Free 
Data in (KB)   Total Meta in (KB)    Free Meta in (KB)
system                   0    256 KB  yes  yes     1887436800 
118287616 (  6%)     1887436800      328248832 ( 17%)
sas0                 65537    256 KB  yes   no   259795189760 
59495012864 ( 23%)              0              0 (  0%)
sata0                65538    256 KB  yes   no  4523875631104 
1405369561600 ( 31%)              0              0 (  0%)

[root at scg-gs0 ~]# mmlspolicy gsfs0 -L
RULE 'scratch01' SET POOL 'sata0' WHERE FILESET_NAME LIKE 'scratch'
RULE 'default' SET POOL 'sas0'

[root at scg-gs0 ~]# cat policy_gpool.txt
rule 'grpdef' GROUP POOL 'gpool' IS 'system' LIMIT(80) THEN 'sas0' 
LIMIT(80) THEN 'sata0'
rule 'repack' MIGRATE FROM POOL 'gpool' TO POOL 'gpool' WEIGHT(FILE_HEAT)

[root at scg-gs0 ~]# mmlsconfig|grep -i heat
fileHeatPeriodMinutes 1440
fileHeatLossPercent 1


The first thing to know is that the MIGRATE rules only run when you run 
'mmapplypolicy', even is you have the migrate policy loaded.  So you 
have to decide when/how to do that.

The filesystem scan takes a while; here are the parameters that let a 
full scan of our filesystem complete in under 10mins (8 I/O servers, 
~300M inodes):

mmapplypolicy gsfs0 -P policy.txt -N 
scg-gs0,scg-gs1,scg-gs2,scg-gs3,scg-gs4,scg-gs5,scg-gs6,scg-gs7 -L 2 -g 
/srv/gsfs0/admin_stuff/ -I yes -B 500 -A 61 -a 4


The issues we have are:
1) the pools are not sized correctly.  We have 3 tiers: 1.8TB SSD, 259TB 
10K SAS, 4523 TB 7.2K NL-SAS (0.03% ssd, 5.4% “fast”, 94.57% “slow”)
2) the client I/O can max out the IOPS in the "slow" pool and in the 
"fast" pool

The original idea was to have all client I/O go to the 10K pool, then 
migrate least-used data to slow pool.  Unfortunately our fast pool fills 
up too quickly and then GPFS can't empty it quickly enough into the slow 
pool.  So I end up switching the placement policy to put files into the 
slow pool.  And then emptying the fast pool into the slow pool doesn't 
work too well because the slow pool is already maxed out on IOPS just 
handling the workload.  So moving the data from fast to slow takes weeks 
and impacts performance.

The original intention was:
[root at scg-gs0 ~]# cat policy_placement.txt
RULE 'faster01' SET POOL 'system' WHERE FILESET_NAME LIKE 'home.%'
RULE 'scratch01' SET POOL 'sata0' WHERE FILESET_NAME LIKE 'scratch'
RULE 'default' SET POOL 'sas0'


I would recommend that your total workload fits in both capacity and 
performance into your "primary" aka "fast" pool.

I also just learned recently of the LIMIT parameter for placement, e.g.
RULE 'scratch01' SET POOL 'sata0' WHERE FILESET_NAME LIKE 'scratch'
RULE 'default' SET POOL 'sas0' LIMIT (95)
RULE 'spillover' SET POOL 'sata0'

So that would help with the pool not filling, but won't help with not 
having enough I/O for migration+clients.

Regards,
-- 
Alex Chekholko chekh at stanford.edu




More information about the gpfsug-discuss mailing list