<div dir="ltr"><div dir="ltr"><div dir="ltr"><div>I don't think it's going to figure this out automatically between the two rules.. I believe you will need to do something like the below (untested, and definitely not perfect!!) rebalancing:</div><div><br></div><div>define(<br> weight_junkedness,<br> (CASE<br> /* Create 3 classes of files */<br> /* ReallyOldStuff */<br> WHEN ((access_age > 365) AND (KB_ALLOCATED > 3584)) THEN 1000 <br> /* OldStuff */<br> WHEN ((access_age > 90) AND (KB_ALLOCATED > 3584)) THEN 100<br> /* everything else */<br> ELSE 0<br> END)<br>)<br><br>RULE 'defineTiers' GROUP POOL 'TIERS'<br> IS 'data' LIMIT(98)<br> THEN 'capacity' LIMIT(98)<br> THEN 'oc'<br><br>RULE 'Rebalance' MIGRATE FROM POOL 'TIERS' TO POOL 'TIERS' WEIGHT(weight_junkedness)<br></div><div><br></div><div>Based on /usr/lpp/mmfs/samples/ilm/mmpolicy-fileheat.sample.<br></div><div><br></div><div><br></div><div> -jf<br></div></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Dec 18, 2018 at 9:10 AM Buterbaugh, Kevin L <<a href="mailto:Kevin.Buterbaugh@vanderbilt.edu">Kevin.Buterbaugh@vanderbilt.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="overflow-wrap: break-word;">
Hi All,
<div><br>
</div>
<div>As those of you who suffered thru my talk at SC18 already know, we’re really short on space on one of our GPFS filesystems as the output of mmdf piped to grep pool shows:</div>
<div><br>
</div>
<div>
<div>Disks in storage pool: system (Maximum disk size allowed is 24 TB)</div>
<div>(pool total) 4.318T 1.078T ( 25%) 79.47G ( 2%)</div>
<div>Disks in storage pool: data (Maximum disk size allowed is 262 TB)</div>
<div>(pool total) 494.7T 38.15T ( 8%) 4.136T ( 1%)</div>
<div>Disks in storage pool: capacity (Maximum disk size allowed is 519 TB)</div>
<div>(pool total) 640.2T 14.56T ( 2%) 716.4G ( 0%)</div>
<div><br>
</div>
The system pool is metadata only. The data pool is the default pool. The capacity pool is where files with an atime (yes, atime) > 90 days get migrated. The capacity pool is comprised of NSDs that are 8+2P RAID 6 LUNs of 8 TB drives, so roughly 58.2 TB usable
space per NSD.</div>
<div><br>
</div>
<div>We have the new storage we purchased, but that’s still being tested and held in reserve for after the first of the year when we create a new GPFS 5 formatted filesystem and start migrating everything to the new filesystem.</div>
<div><br>
</div>
<div>In the meantime, we have also purchased a 60-bay JBOD and 30 x 12 TB drives and will be hooking it up to one of our existing storage arrays on Wednesday. My plan is to create another 3 8+2P RAID 6 LUNs and present those to GPFS as NSDs. They
will be about 88 TB usable space each (because … beginning rant … a 12 TB drive is < 11 TB is size … and don’t get me started on so-called “4K” TV’s … end rant).</div>
<div><br>
</div>
<div>A very wise man who used to work at IBM but now hangs out with people in red polos (<grin>) once told me that it’s OK to mix NSDs of slightly different sizes in the same pool, but you don’t want to put NSDs of vastly different sizes in the same
pool because the smaller ones will fill first and then the larger ones will have to take all the I/O. I consider 58 TB and 88 TB to be pretty significantly different and am therefore planning on creating yet another pool called “oc” (over capacity if a user
asks, old crap internally!) and migrating files with an atime greater than, say, 1 year to that pool. But since ALL of the files in the capacity pool haven’t even been looked at in at least 90 days already, does it really matter? I.e. should I just add the
NSDs to the capacity pool and be done with it?</div>
<div><br>
</div>
<div>If it’s a good idea to create another pool, then I have a question about mmapplypolicy and migrations. I believe I understand how things work, but after spending over an hour looking at the documentation I cannot find anything that explicitly
confirms my understanding … so if I have another pool called oc that’s ~264 TB in size and I write a policy file that looks like:</div>
<div><br>
</div>
<div>
<div>define(access_age,(DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME)))</div>
<div><br>
</div>
<div>RULE 'ReallyOldStuff'</div>
<div> MIGRATE FROM POOL 'capacity'</div>
<div> TO POOL 'oc'</div>
<div> LIMIT(98)</div>
<div> SIZE(KB_ALLOCATED/NLINK)</div>
<div> WHERE ((access_age > 365) AND (KB_ALLOCATED > 3584))</div>
<div><br>
</div>
<div>RULE 'OldStuff'</div>
<div> MIGRATE FROM POOL 'data'</div>
<div> TO POOL 'capacity'</div>
<div> LIMIT(98)</div>
<div> SIZE(KB_ALLOCATED/NLINK)</div>
<div> WHERE ((access_age > 90) AND (KB_ALLOCATED > 3584))</div>
</div>
<div><br>
</div>
<div>Keeping in mind that my capacity pool is already 98% full, is mmapplypolicy smart enough to calculate how much space it’s going to free up in the capacity pool by the “ReallyOldStuff” rule and therefore be able to potentially also move a ton of
stuff from the data pool to the capacity pool via the 2nd rule with just one invocation of mmapplypolicy? That’s what I expect that it will do. I’m hoping I don’t have to run the mmapplypolicy twice … the first to move stuff from capacity to oc and then
a second time for it to realize, oh, I’ve got a much of space free in the capacity pool now.</div>
<div><br>
</div>
<div>Thanks in advance...</div>
<div><br>
</div>
<div>Kevin</div>
<div><br>
</div>
<div>P.S. In case you’re scratching your head over the fact that we have files that people haven’t even looked at for months and months (more than a year in some cases) sitting out there … we sell quota in 1 TB increments … once they’ve bought the
quota, it’s theirs. As long as they’re paying us the monthly fee if they want to keep files relating to research they did during the George Bush Presidency out there … and I mean Bush 41, not Bush 43 ….then that’s their choice. We do not purge files.</div>
<div><br>
<div>
<div>—</div>
<div>Kevin Buterbaugh - Senior System Administrator</div>
<div>Vanderbilt University - Advanced Computing Center for Research and Education</div>
<div><a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" target="_blank">Kevin.Buterbaugh@vanderbilt.edu</a> - (615)875-9633</div>
<div><br>
</div>
<br class="gmail-m_-1284207655023731153Apple-interchange-newline">
</div>
<br>
</div>
</div>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
</blockquote></div>