<font size=2 face="sans-serif">I'd like to see comments and/or questions
and/or suggestions from any others who have implemented, deployed or seriously
planned using FILE_HEAT and/or GROUP POOL.<br>We implemented this a few years ago. Since we haven't had much feedback,
it's either perfect or ... ? ;-)</font><br><br><font size=2 face="sans-serif">Alex, my reading of your story is that
your hardware does not match your requirements, capacity, speeds, feeds.
</font><br><font size=2 face="sans-serif"><br>One new possibly helpful improvement is that SS release 4.2 includes QOSio
which allows one to control the IOPs consumed by migration.<br>So you can control the performance "hit" of running mmapplypolicy.
You can even control it on-the-fly without killing a long running
job </font><br><font size=2 face="sans-serif">-- for example you can throttle it back
to 100iops during prime-time day shift operations<br></font><br><font size=2 face="sans-serif">-- marc of GPFS.</font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Alex Chekholko <chekh@stanford.edu></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug-discuss@spectrumscale.org</font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">06/24/2016 02:10 PM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
Blog on "Automatic tiering of object data based on object heat"</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><tt><font size=2>On 06/21/2016 10:25 PM, Gaurang Tapase wrote:<br>> Hello,<br>><br>> Based on inputs from some of the Spectrum Scale users, sharing a blog<br>> link for "Automatic tiering of object data based on object heat"
by Smita.<br>> </font></tt><a href="https://smitaraut.wordpress.com/2016/06/16/hot-cakes-or-hot-objects-they-better-be-served-fast/"><tt><font size=2>https://smitaraut.wordpress.com/2016/06/16/hot-cakes-or-hot-objects-they-better-be-served-fast/</font></tt></a><tt><font size=2><br>><br><br>Thanks for the blog post. I wanted to post my experiences:<br><br>Note "sata0" is really NL-SAS.<br><br>[root@scg-gs0 ~]# df -h|grep gsfs<br>/dev/gsfs0 4.4P 3.1P 1.4P
70% /srv/gsfs0<br>[root@scg-gs0 ~]# df -i|grep gsfs<br>/dev/gsfs0 314572800 234237984 80334816
75% /srv/gsfs0<br><br>[root@scg-gs0 ~]# mmlspool gsfs0<br>Storage pools in file system at '/srv/gsfs0':<br>Name Id
BlkSize Data Meta Total Data in (KB) Free <br>Data in (KB) Total Meta in (KB) Free Meta in (KB)<br>system 0
256 KB yes yes 1887436800 <br>118287616 ( 6%) 1887436800 328248832
( 17%)<br>sas0 65537
256 KB yes no 259795189760 <br>59495012864 ( 23%) 0
0 ( 0%)<br>sata0 65538
256 KB yes no 4523875631104 <br>1405369561600 ( 31%) 0
0 ( 0%)<br><br>[root@scg-gs0 ~]# mmlspolicy gsfs0 -L<br>RULE 'scratch01' SET POOL 'sata0' WHERE FILESET_NAME LIKE 'scratch'<br>RULE 'default' SET POOL 'sas0'<br><br>[root@scg-gs0 ~]# cat policy_gpool.txt<br>rule 'grpdef' GROUP POOL 'gpool' IS 'system' LIMIT(80) THEN 'sas0' <br>LIMIT(80) THEN 'sata0'<br>rule 'repack' MIGRATE FROM POOL 'gpool' TO POOL 'gpool' WEIGHT(FILE_HEAT)<br><br>[root@scg-gs0 ~]# mmlsconfig|grep -i heat<br>fileHeatPeriodMinutes 1440<br>fileHeatLossPercent 1<br><br><br>The first thing to know is that the MIGRATE rules only run when you run
<br>'mmapplypolicy', even is you have the migrate policy loaded. So you
<br>have to decide when/how to do that.<br><br>The filesystem scan takes a while; here are the parameters that let a <br>full scan of our filesystem complete in under 10mins (8 I/O servers, <br>~300M inodes):<br><br>mmapplypolicy gsfs0 -P policy.txt -N <br>scg-gs0,scg-gs1,scg-gs2,scg-gs3,scg-gs4,scg-gs5,scg-gs6,scg-gs7 -L 2 -g
<br>/srv/gsfs0/admin_stuff/ -I yes -B 500 -A 61 -a 4<br><br><br>The issues we have are:<br>1) the pools are not sized correctly. We have 3 tiers: 1.8TB SSD,
259TB <br>10K SAS, 4523 TB 7.2K NL-SAS (0.03% ssd, 5.4% “fast”, 94.57% “slow”)<br>2) the client I/O can max out the IOPS in the "slow" pool and
in the <br>"fast" pool<br><br>The original idea was to have all client I/O go to the 10K pool, then <br>migrate least-used data to slow pool. Unfortunately our fast pool
fills <br>up too quickly and then GPFS can't empty it quickly enough into the slow
<br>pool. So I end up switching the placement policy to put files into
the <br>slow pool. And then emptying the fast pool into the slow pool doesn't
<br>work too well because the slow pool is already maxed out on IOPS just <br>handling the workload. So moving the data from fast to slow takes
weeks <br>and impacts performance.<br><br>The original intention was:<br>[root@scg-gs0 ~]# cat policy_placement.txt<br>RULE 'faster01' SET POOL 'system' WHERE FILESET_NAME LIKE 'home.%'<br>RULE 'scratch01' SET POOL 'sata0' WHERE FILESET_NAME LIKE 'scratch'<br>RULE 'default' SET POOL 'sas0'<br><br><br>I would recommend that your total workload fits in both capacity and <br>performance into your "primary" aka "fast" pool.<br><br>I also just learned recently of the LIMIT parameter for placement, e.g.<br>RULE 'scratch01' SET POOL 'sata0' WHERE FILESET_NAME LIKE 'scratch'<br>RULE 'default' SET POOL 'sas0' LIMIT (95)<br>RULE 'spillover' SET POOL 'sata0'<br><br>So that would help with the pool not filling, but won't help with not <br>having enough I/O for migration+clients.<br><br>Regards,<br>-- <br>Alex Chekholko chekh@stanford.edu<br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br><br></font></tt><br><BR>