<font size=2 face="sans-serif">Kevin,</font><br><br><font size=2 face="sans-serif">1. Running with both fairly simple rules
so that you migrate "in both directions" is fine. It was
designed to do that!</font><br><br><font size=2 face="sans-serif">2. Glad you understand the logic of
"rules hit" vs "files chosen". </font><br><br><font size=2 face="sans-serif">3. To begin to understand "what
the hxxx is going on" (as our fearless leader liked to say before
he was in charge ;-) ) I suggest:</font><br><br><font size=2 face="sans-serif">(a) Run mmapplypolicy on directory of
just a few files `mmapplypolicy /gpfs23/test-directory -I test ...`
and check that the </font><br><font size=2 face="sans-serif">[I] ... Current data pool utilization
</font><br><font size=2 face="sans-serif">message is consistent with the output
of `mmdf gpfs23`. </font><br><br><font size=2 face="sans-serif">They should be, but if they're not,
that's a weird problem right there since they're supposed to be looking
at the same metadata!</font><br><br><font size=2 face="sans-serif">You can do this anytime, should complete
almost instantly...</font><br><br><font size=2 face="sans-serif">(b) When time and resources permit,
re-run mmapplypolicy on the full FS with your desired migration policy.<br>Again, do the "Current", "Chosen" and "Predicted"
messages make sense, and "add up"?<br>Do the file counts seem reasonable, considering that you recently did migrations/deletions
that should have changed the counts compared to previous runs<br>of mmapplypolicy? If you just want to look and not actually change
anything, use `-I test` which will skip the migration steps. If you
want to see the list of files chosen</font><br><br><font size=2 face="sans-serif">(c) If you continue to see significant
discrepancies between mmapplypolicy and mmdf, let us know.</font><br><font size=2 face="sans-serif"><br>(d) Also at some point you may consider running mmrestripefs with options
to make sure every file has its data blocks where they are supposed to
be and is replicated<br>as you have specified.</font><br><br><font size=2 face="sans-serif">Let's see where those steps take us...<br></font><br><font size=2 face="sans-serif">-- marc of Spectrum Scale (né GPFS)</font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">"Buterbaugh, Kevin
L" <Kevin.Buterbaugh@Vanderbilt.Edu></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug main discussion
list <gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">04/17/2017 11:25 AM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
mmapplypolicy didn't migrate everything it should
have - why not?</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><font size=3>Hi Marc, </font><br><br><font size=3>I do understand what you’re saying about mmapplypolicy
deciding it only needed to move ~1.8 million files to fill the capacity
pool to ~98% full. However, it is now more than 24 hours since the
mmapplypolicy finished “successfully” and:</font><br><br><font size=3>Disks in storage pool: gpfs23capacity (Maximum disk size
allowed is 519 TB)</font><br><font size=3>eon35Ansd
58.2T 35 No Yes
29.66T ( 51%) 64.16G ( 0%)
</font><br><font size=3>eon35Dnsd
58.2T 35 No Yes
29.66T ( 51%) 64.61G ( 0%)
</font><br><font size=3>
-------------
-------------------- -------------------</font><br><font size=3>(pool total) 116.4T
59.33T ( 51%) 128.8G
( 0%)</font><br><br><font size=3>And yes, I did run the mmapplypolicy with “-I yes” …
here’s the partially redacted command line:</font><br><br><font size=3>/usr/lpp/mmfs/bin/mmapplypolicy gpfs23 -A 75 -a 4 -g <some
folder on another gpfs filesystem> -I yes -L 1 -P ~/gpfs/gpfs23_migration.policy
-N some,list,of,NSD,server,nodes</font><br><br><font size=3>And here’s that policy file:</font><br><br><font size=3>define(access_age,(DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME)))</font><br><font size=3>define(GB_ALLOCATED,(KB_ALLOCATED/1048576.0))</font><br><br><font size=3>RULE 'OldStuff'</font><br><font size=3> MIGRATE FROM POOL 'gpfs23data'</font><br><font size=3> TO POOL 'gpfs23capacity'</font><br><font size=3> LIMIT(98)</font><br><font size=3> WHERE ((access_age > 14) AND (KB_ALLOCATED >
3584))</font><br><br><font size=3>RULE 'INeedThatAfterAll'</font><br><font size=3> MIGRATE FROM POOL 'gpfs23capacity'</font><br><font size=3> TO POOL 'gpfs23data'</font><br><font size=3> LIMIT(75)</font><br><font size=3> WHERE (access_age < 14)</font><br><br><font size=3>The one thing that has changed is that formerly I only
ran the migration in one direction at a time … i.e. I used to have those
two rules in two separate files and would run an mmapplypolicy using the
OldStuff rule the 1st weekend of the month and run the other rule the other
weekends of the month. This is the 1st weekend that I attempted to
run an mmapplypolicy that did both at the same time. Did I mess something
up with that?</font><br><br><font size=3>I have not run it again yet because we also run migrations
on the other filesystem that we are still in the process of migrating off
of. So gpfs23 goes 1st and as soon as it’s done the other filesystem
migration kicks off. I don’t like to run two migrations simultaneously
if at all possible. The 2nd migration ran until this morning, when
it was unfortunately terminated by a network switch crash that has also
had me tied up all morning until now. :-(</font><br><br><font size=3>And yes, there is something else going on … well, was
going on - the network switch crash killed this too … I have been running
an rsync on one particular ~80TB directory tree from the old filesystem
to gpfs23. I understand that the migration wouldn’t know about those
files and that’s fine … I just don’t understand why mmapplypolicy said
it was going to fill the capacity pool to 98% but didn’t do it … wait,
mmapplypolicy hasn’t gone into politics, has it?!? ;-)</font><br><br><font size=3>Thanks - and again, if I should open a PMR for this please
let me know...</font><br><br><font size=3>Kevin</font><br><br><font size=3>On Apr 16, 2017, at 2:15 PM, Marc A Kaplan <</font><a href=mailto:makaplan@us.ibm.com><font size=3 color=blue><u>makaplan@us.ibm.com</u></font></a><font size=3>>
wrote:</font><br><br><font size=2 face="sans-serif">Let's look at how mmapplypolicy does
the reckoning.<br>Before it starts, it see your pools as:</font><font size=3><br><br>[I] GPFS Current Data Pool Utilization in KB and %<br>Pool_Name
KB_Occupied KB_Total Percent_Occupied<br>gpfs23capacity 55365193728
124983549952 44.297984614%<br>gpfs23data 166747037696
343753326592 48.507759721%<br>system
0
0 0.000000000% (no user data)<br>[I] 75142046 of 209715200 inodes used: 35.830520%.</font><font size=2 face="sans-serif"><br><br>Your rule says you want to migrate data to gpfs23capacity, up to 98% full:</font><font size=3><br><br>RULE 'OldStuff'<br> MIGRATE FROM POOL 'gpfs23data'<br> TO POOL 'gpfs23capacity'<br> LIMIT(98) WHERE ...<br><br>We scan your files and find and reckon...<br>[I] Summary of Rule Applicability and File Choices:<br> Rule# Hit_Cnt KB_Hit
Chosen KB_Chosen
KB_Ill Rule<br> 0 5255960 237675081344
1868858 67355430720
0 RULE 'OldStuff' MIGRATE
FROM POOL 'gpfs23data' TO POOL 'gpfs23capacity' LIMIT(98.000000) WHERE(.)<br><br>So yes, 5.25Million files match the rule, but the utility chooses 1.868Million
files that add up to 67,355GB and figures that if it migrates those to
gpfs23capacity,<br>(and also figuring the other migrations by your second rule)then
gpfs23 will end up 97.9999% full.<br>We show you that with our "predictions" message.<br><br>Predicted Data Pool Utilization in KB and %:<br>Pool_Name
KB_Occupied KB_Total Percent_Occupied<br>gpfs23capacity 122483878944
124983549952 97.999999993%<br>gpfs23data 104742360032
343753326592 30.470209865%<br><br>So that's why it chooses to migrate "only" 67GB....<br><br>See? Makes sense to me.<br><br>Questions:<br>Did you run with -I yes or -I defer ?<br><br>Were some of the files illreplicated or illplaced?<br><br>Did you give the cluster-wide space reckoning protocols time to see the
changes? mmdf is usually "behind" by some non-neglible
amount of time.<br><br>What else is going on?</font><font size=2 face="sans-serif"><br>If you're moving or deleting or creating data by other means
while mmapplypolicy is running -- it doesn't "know" about that!
</font><font size=3><br></font><font size=2 face="sans-serif"><br>Run it again!</font><font size=3><br><br><ATT00001.gif><br><br><br></font><font size=1 color=#5f5f5f face="sans-serif"><br>From: </font><font size=1 face="sans-serif">"Buterbaugh,
Kevin L" <</font><a href=mailto:Kevin.Buterbaugh@Vanderbilt.Edu><font size=1 color=blue face="sans-serif"><u>Kevin.Buterbaugh@Vanderbilt.Edu</u></font></a><font size=1 face="sans-serif">></font><font size=1 color=#5f5f5f face="sans-serif"><br>To: </font><font size=1 face="sans-serif">gpfsug
main discussion list <</font><a href="mailto:gpfsug-discuss@spectrumscale.org"><font size=1 color=blue face="sans-serif"><u>gpfsug-discuss@spectrumscale.org</u></font></a><font size=1 face="sans-serif">></font><font size=1 color=#5f5f5f face="sans-serif"><br>Date: </font><font size=1 face="sans-serif">04/16/2017
09:47 AM</font><font size=1 color=#5f5f5f face="sans-serif"><br>Subject: </font><font size=1 face="sans-serif">[gpfsug-discuss]
mmapplypolicy didn't migrate everything it should
have - why not?</font><font size=1 color=#5f5f5f face="sans-serif"><br>Sent by: </font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><font size=1 color=blue face="sans-serif"><u>gpfsug-discuss-bounces@spectrumscale.org</u></font></a><font size=3><br></font><hr noshade><font size=3><br><br><br>Hi All, <br><br>First off, I can open a PMR for this if I need to. Second, I am far
from an mmapplypolicy guru. With that out of the way … I have an
mmapplypolicy job that didn’t migrate anywhere close to what it could
/ should have. From the log file I have it create, here is the part
where it shows the policies I told it to invoke:<br><br>[I] Qos 'maintenance' configured as inf<br>[I] GPFS Current Data Pool Utilization in KB and %<br>Pool_Name
KB_Occupied KB_Total Percent_Occupied<br>gpfs23capacity 55365193728
124983549952 44.297984614%<br>gpfs23data 166747037696
343753326592 48.507759721%<br>system
0
0 0.000000000% (no user data)<br>[I] 75142046 of 209715200 inodes used: 35.830520%.<br>[I] Loaded policy rules from /root/gpfs/gpfs23_migration.policy.<br>Evaluating policy rules with CURRENT_TIMESTAMP = 2017-04-15@01:13:02 UTC<br>Parsed 2 policy rules.<br><br>RULE 'OldStuff'<br> MIGRATE FROM POOL 'gpfs23data'<br> TO POOL 'gpfs23capacity'<br> LIMIT(98)<br> WHERE (((DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME)) > 14) AND
(KB_ALLOCATED > 3584))<br><br>RULE 'INeedThatAfterAll'<br> MIGRATE FROM POOL 'gpfs23capacity'<br> TO POOL 'gpfs23data'<br> LIMIT(75)<br> WHERE ((DAYS(CURRENT_TIMESTAMP) - DAYS(ACCESS_TIME)) < 14)<br><br>And then the log shows it scanning all the directories and then says, "OK,
here’s what I’m going to do":<br><br>[I] Summary of Rule Applicability and File Choices:<br> Rule# Hit_Cnt KB_Hit
Chosen KB_Chosen
KB_Ill Rule<br> 0 5255960 237675081344
1868858 67355430720
0 RULE 'OldStuff' MIGRATE
FROM POOL 'gpfs23data' TO POOL 'gpfs23capacity' LIMIT(98.000000) WHERE(.)<br> 1 611
236745504 611
236745504 0
RULE 'INeedThatAfterAll' MIGRATE FROM POOL 'gpfs23capacity' TO POOL 'gpfs23data'
LIMIT(75.000000) WHERE(.)<br><br>[I] Filesystem objects with no applicable rules: 414911602.<br><br>[I] GPFS Policy Decisions and File Choice Totals:<br> Chose to migrate 67592176224KB: 1869469 of 5256571 candidates;<br>Predicted Data Pool Utilization in KB and %:<br>Pool_Name
KB_Occupied KB_Total Percent_Occupied<br>gpfs23capacity 122483878944
124983549952 97.999999993%<br>gpfs23data 104742360032
343753326592 30.470209865%<br>system
0
0 0.000000000% (no user data)<br><br>Notice that it says it’s only going to migrate less than 2 million of
the 5.25 million candidate files!! And sure enough, that’s all it
did:<br><br>[I] A total of 1869469 files have been migrated, deleted or processed by
an EXTERNAL EXEC/script;<br> 0 'skipped' files and/or errors.<br><br>And, not surprisingly, the gpfs23capacity pool on gpfs23 is nowhere near
98% full:<br><br>Disks in storage pool: gpfs23capacity (Maximum disk size allowed is 519
TB)<br>eon35Ansd 58.2T
35 No Yes
29.54T ( 51%) 63.93G ( 0%) <br>eon35Dnsd 58.2T
35 No Yes
29.54T ( 51%) 64.39G ( 0%) <br> -------------
-------------------- -------------------<br>(pool total) 116.4T
59.08T ( 51%) 128.3G ( 0%)<br><br>I don’t understand why it only migrated a small subset of what it could
/ should have?<br><br>We are doing a migration from one filesystem (gpfs21) to gpfs23 and I really
need to stuff my gpfs23capacity pool as full of data as I can to keep the
migration going. Any ideas anyone? Thanks in advance…<br><br>—<br>Kevin Buterbaugh - Senior System Administrator<br>Vanderbilt University - Advanced Computing Center for Research and Education</font><font size=3 color=blue><u><br></u></font><a href=mailto:Kevin.Buterbaugh@vanderbilt.edu><font size=3 color=blue><u>Kevin.Buterbaugh@vanderbilt.edu</u></font></a><font size=3>-
(615)875-9633<br><br></font><tt><font size=2><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font></tt><a href=http://spectrumscale.org/><tt><font size=2 color=blue><u>spectrumscale.org</u></font></tt></a><font size=3 color=blue><u><br></u></font><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></tt></a><font size=3><br><br><br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href=http://spectrumscale.org/><font size=3 color=blue><u>spectrumscale.org</u></font></a><font size=3 color=blue><u><br></u></font><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><font size=3 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></a><br><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br></font></tt><br><br><BR>