<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Hi Marc,
<div class=""><br class="">
</div>
<div class="">I do understand what you’re saying about mmapplypolicy deciding it only needed to move ~1.8 million files to fill the capacity pool to ~98% full. However, it is now more than 24 hours since the mmapplypolicy finished “successfully” and:</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">Disks in storage pool: gpfs23capacity (Maximum disk size allowed is 519 TB)</div>
<div class="">eon35Ansd 58.2T 35 No Yes 29.66T ( 51%) 64.16G ( 0%) </div>
<div class="">eon35Dnsd 58.2T 35 No Yes 29.66T ( 51%) 64.61G ( 0%) </div>
<div class=""> ------------- -------------------- -------------------</div>
<div class="">(pool total) 116.4T 59.33T ( 51%) 128.8G ( 0%)</div>
</div>
<div class=""><br class="">
</div>
<div class="">And yes, I did run the mmapplypolicy with “-I yes” … here’s the partially redacted command line:</div>
<div class=""><br class="">
</div>
<div class="">/usr/lpp/mmfs/bin/mmapplypolicy gpfs23 -A 75 -a 4 -g <some folder on another gpfs filesystem> -I yes -L 1 -P ~/gpfs/gpfs23_migration.policy -N some,list,of,NSD,server,nodes</div>
<div class=""><br class="">
</div>
<div class="">And here’s that policy file:</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">define(access_age,(DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME)))</div>
<div class="">define(GB_ALLOCATED,(KB_ALLOCATED/1048576.0))</div>
<div class=""><br class="">
</div>
<div class="">RULE 'OldStuff'</div>
<div class=""> MIGRATE FROM POOL 'gpfs23data'</div>
<div class=""> TO POOL 'gpfs23capacity'</div>
<div class=""> LIMIT(98)</div>
<div class=""> WHERE ((access_age > 14) AND (KB_ALLOCATED > 3584))</div>
<div class=""><br class="">
</div>
<div class="">RULE 'INeedThatAfterAll'</div>
<div class=""> MIGRATE FROM POOL 'gpfs23capacity'</div>
<div class=""> TO POOL 'gpfs23data'</div>
<div class=""> LIMIT(75)</div>
<div class=""> WHERE (access_age < 14)</div>
</div>
<div class=""><br class="">
</div>
<div class="">The one thing that has changed is that formerly I only ran the migration in one direction at a time … i.e. I used to have those two rules in two separate files and would run an mmapplypolicy using the OldStuff rule the 1st weekend of the month
and run the other rule the other weekends of the month. This is the 1st weekend that I attempted to run an mmapplypolicy that did both at the same time. Did I mess something up with that?</div>
<div class=""><br class="">
</div>
<div class="">I have not run it again yet because we also run migrations on the other filesystem that we are still in the process of migrating off of. So gpfs23 goes 1st and as soon as it’s done the other filesystem migration kicks off. I don’t like to run
two migrations simultaneously if at all possible. The 2nd migration ran until this morning, when it was unfortunately terminated by a network switch crash that has also had me tied up all morning until now. :-(</div>
<div class=""><br class="">
</div>
<div class="">And yes, there is something else going on … well, was going on - the network switch crash killed this too … I have been running an rsync on one particular ~80TB directory tree from the old filesystem to gpfs23. I understand that the migration
wouldn’t know about those files and that’s fine … I just don’t understand why mmapplypolicy said it was going to fill the capacity pool to 98% but didn’t do it … wait, mmapplypolicy hasn’t gone into politics, has it?!? ;-)</div>
<div class=""><br class="">
</div>
<div class="">Thanks - and again, if I should open a PMR for this please let me know...</div>
<div class=""><br class="">
</div>
<div class="">Kevin</div>
<div class=""><br class="">
</div>
<div class="">
<div>
<blockquote type="cite" class="">
<div class="">On Apr 16, 2017, at 2:15 PM, Marc A Kaplan <<a href="mailto:makaplan@us.ibm.com" class="">makaplan@us.ibm.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class=""><font size="2" face="sans-serif" class="">Let's look at how mmapplypolicy does the reckoning.</font><br class="">
<font size="2" face="sans-serif" class="">Before it starts, it see your pools as:</font><br class="">
<br class="">
<font size="3" class="">[I] GPFS Current Data Pool Utilization in KB and %</font><br class="">
<font size="3" class="">Pool_Name KB_Occupied KB_Total Percent_Occupied</font><br class="">
<font size="3" class="">gpfs23capacity 55365193728 124983549952 44.297984614%</font><br class="">
<font size="3" class="">gpfs23data 166747037696 343753326592 48.507759721%</font><br class="">
<font size="3" class="">system 0 0 0.000000000% (no user data)</font><br class="">
<font size="3" class="">[I] 75142046 of 209715200 inodes used: 35.830520%.</font><br class="">
<font size="2" face="sans-serif" class=""><br class="">
Your rule says you want to migrate data to gpfs23capacity, up to 98% full:</font><br class="">
<br class="">
<font size="3" class="">RULE 'OldStuff'</font><br class="">
<font size="3" class=""> MIGRATE FROM POOL 'gpfs23data'</font><br class="">
<font size="3" class=""> TO POOL 'gpfs23capacity'</font><br class="">
<font size="3" class=""> LIMIT(98) WHERE ...</font><br class="">
<br class="">
<font size="3" class="">We scan your files and find and reckon...</font><br class="">
<font size="3" class="">[I] Summary of Rule Applicability and File Choices:</font><br class="">
<font size="3" class=""> Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule</font><br class="">
<font size="3" class=""> 0 5255960 237675081344 1868858 67355430720 0 RULE 'OldStuff' MIGRATE FROM POOL 'gpfs23data' TO POOL 'gpfs23capacity' LIMIT(98.000000) WHERE(.)</font><br class="">
<br class="">
<font size="3" class="">So yes, 5.25Million files match the rule, but the utility chooses 1.868Million files that add up to 67,355GB and figures that if it migrates those to gpfs23capacity,</font><br class="">
<font size="3" class="">(and also figuring the other migrations by your second rule)then gpfs23 will end up 97.9999% full.</font><br class="">
<font size="3" class="">We show you that with our "predictions" message.</font><br class="">
<br class="">
<font size="3" class="">Predicted Data Pool Utilization in KB and %:</font><br class="">
<font size="3" class="">Pool_Name KB_Occupied KB_Total Percent_Occupied</font><br class="">
<font size="3" class="">gpfs23capacity 122483878944 124983549952 97.999999993%</font><br class="">
<font size="3" class="">gpfs23data 104742360032 343753326592 30.470209865%</font><br class="">
<br class="">
<font size="3" class="">So that's why it chooses to migrate "only" 67GB....</font><br class="">
<br class="">
<font size="3" class="">See? Makes sense to me.</font><br class="">
<br class="">
<font size="3" class="">Questions:</font><br class="">
<font size="3" class="">Did you run with -I yes or -I defer ?</font><br class="">
<br class="">
<font size="3" class="">Were some of the files illreplicated or illplaced?</font><br class="">
<br class="">
<font size="3" class="">Did you give the cluster-wide space reckoning protocols time to see the changes? mmdf is usually "behind" by some non-neglible amount of time.</font><br class="">
<br class="">
<font size="3" class="">What else is going on?</font><br class="">
<font size="2" face="sans-serif" class="">If you're moving or deleting or creating data by other means while mmapplypolicy is running -- it doesn't "know" about that! </font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">Run it again!</font><br class="">
<br class="">
<span id="cid:_1_0F8E5E800F8E5C400069CCD985258104"><ATT00001.gif></span><br class="">
<br class="">
<br class="">
<br class="">
<font size="1" color="#5f5f5f" face="sans-serif" class="">From: </font><font size="1" face="sans-serif" class="">"Buterbaugh, Kevin L" <<a href="mailto:Kevin.Buterbaugh@Vanderbilt.Edu" class="">Kevin.Buterbaugh@Vanderbilt.Edu</a>></font><br class="">
<font size="1" color="#5f5f5f" face="sans-serif" class="">To: </font><font size="1" face="sans-serif" class="">gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" class="">gpfsug-discuss@spectrumscale.org</a>></font><br class="">
<font size="1" color="#5f5f5f" face="sans-serif" class="">Date: </font><font size="1" face="sans-serif" class="">04/16/2017 09:47 AM</font><br class="">
<font size="1" color="#5f5f5f" face="sans-serif" class="">Subject: </font><font size="1" face="sans-serif" class="">[gpfsug-discuss] mmapplypolicy didn't migrate everything it should have - why not?</font><br class="">
<font size="1" color="#5f5f5f" face="sans-serif" class="">Sent by: </font><font size="1" face="sans-serif" class=""><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" class="">gpfsug-discuss-bounces@spectrumscale.org</a></font><br class="">
<hr noshade="" class="">
<br class="">
<br class="">
<br class="">
<font size="3" class="">Hi All, </font><br class="">
<br class="">
<font size="3" class="">First off, I can open a PMR for this if I need to. Second, I am far from an mmapplypolicy guru. With that out of the way … I have an mmapplypolicy job that didn’t migrate anywhere close to what it could / should have. From the log
file I have it create, here is the part where it shows the policies I told it to invoke:</font><br class="">
<br class="">
<font size="3" class="">[I] Qos 'maintenance' configured as inf</font><br class="">
<font size="3" class="">[I] GPFS Current Data Pool Utilization in KB and %</font><br class="">
<font size="3" class="">Pool_Name KB_Occupied KB_Total Percent_Occupied</font><br class="">
<font size="3" class="">gpfs23capacity 55365193728 124983549952 44.297984614%</font><br class="">
<font size="3" class="">gpfs23data 166747037696 343753326592 48.507759721%</font><br class="">
<font size="3" class="">system 0 0 0.000000000% (no user data)</font><br class="">
<font size="3" class="">[I] 75142046 of 209715200 inodes used: 35.830520%.</font><br class="">
<font size="3" class="">[I] Loaded policy rules from /root/gpfs/gpfs23_migration.policy.</font><br class="">
<font size="3" class="">Evaluating policy rules with CURRENT_TIMESTAMP = 2017-04-15@01:13:02 UTC</font><br class="">
<font size="3" class="">Parsed 2 policy rules.</font><br class="">
<br class="">
<font size="3" class="">RULE 'OldStuff'</font><br class="">
<font size="3" class=""> MIGRATE FROM POOL 'gpfs23data'</font><br class="">
<font size="3" class=""> TO POOL 'gpfs23capacity'</font><br class="">
<font size="3" class=""> LIMIT(98)</font><br class="">
<font size="3" class=""> WHERE (((DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME)) > 14) AND (KB_ALLOCATED > 3584))</font><br class="">
<br class="">
<font size="3" class="">RULE 'INeedThatAfterAll'</font><br class="">
<font size="3" class=""> MIGRATE FROM POOL 'gpfs23capacity'</font><br class="">
<font size="3" class=""> TO POOL 'gpfs23data'</font><br class="">
<font size="3" class=""> LIMIT(75)</font><br class="">
<font size="3" class=""> WHERE ((DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME)) < 14)</font><br class="">
<br class="">
<font size="3" class="">And then the log shows it scanning all the directories and then says, "OK, here’s what I’m going to do":</font><br class="">
<br class="">
<font size="3" class="">[I] Summary of Rule Applicability and File Choices:</font><br class="">
<font size="3" class=""> Rule# Hit_Cnt KB_Hit Chosen KB_Chosen KB_Ill Rule</font><br class="">
<font size="3" class=""> 0 5255960 237675081344 1868858 67355430720 0 RULE 'OldStuff' MIGRATE FROM POOL 'gpfs23data' TO POOL 'gpfs23capacity' LIMIT(98.000000) WHERE(.)</font><br class="">
<font size="3" class=""> 1 611 236745504 611 236745504 0 RULE 'INeedThatAfterAll' MIGRATE FROM POOL 'gpfs23capacity' TO POOL 'gpfs23data' LIMIT(75.000000) WHERE(.)</font><br class="">
<br class="">
<font size="3" class="">[I] Filesystem objects with no applicable rules: 414911602.</font><br class="">
<br class="">
<font size="3" class="">[I] GPFS Policy Decisions and File Choice Totals:</font><br class="">
<font size="3" class=""> Chose to migrate 67592176224KB: 1869469 of 5256571 candidates;</font><br class="">
<font size="3" class="">Predicted Data Pool Utilization in KB and %:</font><br class="">
<font size="3" class="">Pool_Name KB_Occupied KB_Total Percent_Occupied</font><br class="">
<font size="3" class="">gpfs23capacity 122483878944 124983549952 97.999999993%</font><br class="">
<font size="3" class="">gpfs23data 104742360032 343753326592 30.470209865%</font><br class="">
<font size="3" class="">system 0 0 0.000000000% (no user data)</font><br class="">
<br class="">
<font size="3" class="">Notice that it says it’s only going to migrate less than 2 million of the 5.25 million candidate files!! And sure enough, that’s all it did:</font><br class="">
<br class="">
<font size="3" class="">[I] A total of 1869469 files have been migrated, deleted or processed by an EXTERNAL EXEC/script;</font><br class="">
<font size="3" class=""> 0 'skipped' files and/or errors.</font><br class="">
<br class="">
<font size="3" class="">And, not surprisingly, the gpfs23capacity pool on gpfs23 is nowhere near 98% full:</font><br class="">
<br class="">
<font size="3" class="">Disks in storage pool: gpfs23capacity (Maximum disk size allowed is 519 TB)</font><br class="">
<font size="3" class="">eon35Ansd 58.2T 35 No Yes 29.54T ( 51%) 63.93G ( 0%)
</font><br class="">
<font size="3" class="">eon35Dnsd 58.2T 35 No Yes 29.54T ( 51%) 64.39G ( 0%)
</font><br class="">
<font size="3" class=""> ------------- -------------------- -------------------</font><br class="">
<font size="3" class="">(pool total) 116.4T 59.08T ( 51%) 128.3G ( 0%)</font><br class="">
<br class="">
<font size="3" class="">I don’t understand why it only migrated a small subset of what it could / should have?</font><br class="">
<br class="">
<font size="3" class="">We are doing a migration from one filesystem (gpfs21) to gpfs23 and I really need to stuff my gpfs23capacity pool as full of data as I can to keep the migration going. Any ideas anyone? Thanks in advance…</font><br class="">
<br class="">
<font size="3" class="">—</font><br class="">
<font size="3" class="">Kevin Buterbaugh - Senior System Administrator</font><br class="">
<font size="3" class="">Vanderbilt University - Advanced Computing Center for Research and Education</font><br class="">
<a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" class=""><font size="3" color="blue" class=""><u class="">Kevin.Buterbaugh@vanderbilt.edu</u></font></a><font size="3" class="">- (615)875-9633</font><br class="">
<br class="">
<br class="">
<tt class=""><font size="2" class="">_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a><br class="">
</font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class=""><tt class=""><font size="2" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt class=""><font size="2" class=""><br class="">
</font></tt><br class="">
<br class="">
<br class="">
_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a><br class="">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>