<font size=2 face="sans-serif">Kevin, Wow. Never underestimate the
power of ...</font><br><br><font size=2 face="sans-serif">Anyhow try this as a fix. </font><br><br><font size=2 face="sans-serif">Add the clause SIZE(KB_ALLOCATED/NLINK)
to your MIGRATE rules.</font><br><br><font size=2 face="sans-serif">This spreads the total actual size over
each hardlink...<br></font><br><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">"Buterbaugh, Kevin
L" <Kevin.Buterbaugh@Vanderbilt.Edu></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug main discussion
list <gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">04/18/2017 12:33 PM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
mmapplypolicy didn't migrate everything it should
have - why not?</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><font size=3>Hi Marc, </font><br><br><font size=3>Two things:</font><br><br><font size=3>1. I have a PMR open now.</font><br><br><font size=3>2. You *may* have identified the problem … I’m
still checking … but files with hard links may be our problem. I
wrote a simple Perl script to interate over the log file I had mmapplypolicy
create. Here’s the code (don’t laugh, I’m a SysAdmin, not a programmer,
and I whipped this out in < 5 minutes … and yes, I realize the fact
that I used Perl instead of Python shows my age as well <grin>):</font><br><br><font size=3>#!/usr/bin/perl</font><br><font size=3>#</font><br><font size=3>use strict;</font><br><font size=3>use warnings;</font><br><font size=3>my $InputFile = "/tmp/mmapplypolicy.gpfs23.log";</font><br><font size=3>my $TotalFiles = 0;</font><br><font size=3>my $TotalLinks = 0;</font><br><font size=3>my $TotalSize = 0;</font><br><font size=3>open INPUT, $InputFile or die "Couldn\'t open $InputFile
for read: $!\n";</font><br><font size=3>while (<INPUT>) {</font><br><font size=3> next unless /MIGRATED/;</font><br><font size=3> $TotalFiles++;</font><br><font size=3> my $FileName = (split / /)[3];</font><br><font size=3> if ( -f $FileName ) { # some files may have
been deleted since mmapplypolicy ran</font><br><font size=3> my ($NumLinks, $FileSize) = (stat($FileName))[3,7];</font><br><font size=3> $TotalLinks += $NumLinks;</font><br><font size=3> $TotalSize += $FileSize;</font><br><font size=3> }</font><br><font size=3>}</font><br><font size=3>close INPUT;</font><br><font size=3>print "Number of files / links = $TotalFiles / $TotalLinks,
Total size = $TotalSize\n";</font><br><font size=3>exit 0;</font><br><br><font size=3>And here’s what it kicked out:</font><br><br><font size=3>Number of files / links = 1620263 / 80818483, Total size
= 53966202814094</font><br><br><font size=3>1.6 million files but 80 million hard links!!!</font><br><br><font size=3>I’m doing some checking right now, but it appears that
it is one particular group - and therefore one particular fileset - that
is responsible for this … they’ve got thousands of files with 50 or more
hard links each … and they’re not inconsequential in size.</font><br><br><font size=3>IIRC (and keep in mind I’m far from a GPFS policy guru),
there is a way to say something to the effect of “and the path does not
contain /gpfs23/fileset/path” … may need a little help getting that right.</font><br><br><font size=3>I’ll post this information to the ticket as well but
wanted to update the list. This wouldn’t be the first time we were
an “edge case” for something in GPFS… ;-)</font><br><br><font size=3>Thanks...</font><br><br><font size=3>Kevin</font><br><br><br><font size=3>On Apr 18, 2017, at 10:11 AM, Marc A Kaplan <</font><a href=mailto:makaplan@us.ibm.com><font size=3 color=blue><u>makaplan@us.ibm.com</u></font></a><font size=3>>
wrote:</font><br><br><font size=2 face="sans-serif">ANYONE else reading this saga? Who
uses mmapplypolicy to migrate files within multi-TB file systems? Problems?
Or all working as expected?</font><font size=3><br></font><font size=2 face="sans-serif"><br>------</font><font size=3><br></font><font size=2 face="sans-serif"><br>Well, again mmapplypolicy "thinks" it has "chosen"
1.6 million files whose total size is 61 Terabytes and migrating those
will bring the occupancy of gpfs23capacity pool to 98% and then we're done.</font><font size=3><br></font><font size=2 face="sans-serif"><br>So now I'm wondering where this is going wrong. Is there some bug
in the reckoning inside of mmapplypolicy or somewhere else in GPFS?</font><font size=3><br></font><font size=2 face="sans-serif"><br>Sure you can put in an PMR, and probably should. I'm guessing whoever
picks up the PMR will end up calling or emailing me ... but maybe she can
do some of the clerical work for us... </font><font size=3><br></font><font size=2 face="sans-serif"><br>While we're waiting for that... Here's what I suggest next.</font><font size=3><br></font><font size=2 face="sans-serif"><br>Add a clause ...</font><font size=3><br></font><font size=2 face="sans-serif"><br>SHOW(varchar(KB_ALLOCATED) || ' n=' || varchar(NLINK))</font><font size=3><br></font><font size=2 face="sans-serif"><br>before the WHERE clause to each of your rules.</font><font size=3><br></font><font size=2 face="sans-serif"><br>Re-run the command with options '-I test -L 2' and collect
the output. </font><font size=3><br></font><font size=2 face="sans-serif"><br>We're not actually going to move any data, but we're going to look at the
files and file sizes that are "chosen"...</font><font size=3><br></font><font size=2 face="sans-serif"><br>You should see 1.6 million lines that look kind of like this:</font><font size=3><br></font><font size=2 face="sans-serif"><br>/yy/dat/bigC RULE 'msx' MIGRATE FROM POOL 'system' TO POOL
'xtra' WEIGHT(inf) SHOW( 1024 n=1)</font><font size=3><br></font><font size=2 face="sans-serif"><br>Run a script over the output to add up all the SHOW() values in the lines
that contain TO POOL 'gpfs23capacity' and verify that they do indeed<br>add up to 61TB... (The show is in KB so the SHOW numbers should add
up to 61 billion).</font><font size=3><br></font><font size=2 face="sans-serif"><br>That sanity checks the policy arithmetic. Let's assume that's okay.
</font><font size=3><br></font><font size=2 face="sans-serif"><br>Then the next question is whether the individual numbers are correct...
Zach Giles made a suggestion... which I'll interpret as <br>find some of the biggest of those files and check that they really are
that big....</font><font size=3><br></font><font size=2 face="sans-serif"><br>At this point, I really don't know, but I'm guessing there's some discrepances
in the reported KB_ALLOCATED numbers for many of the files...<br>and/or they are "illplaced" - the data blocks aren't all
in the pool FROM POOL ...</font><font size=3><br></font><font size=2 face="sans-serif"><br>HMMMM.... I just thought about this some more and added the NLINK
statistic. It would be unusual for this to be a big problem, but
files that are hard linked are<br>not recognized by mmapplypolicy as sharing storage... <br>This has not come to my attention as a significant problem -- does the
file system in question have significant GBs of hard linked files?</font><font size=3><br></font><font size=2 face="sans-serif"><br>The truth is that you're the first customer/user/admin in a long time to
question/examine how mmapplypolicy does its space reckoning ... <br>Optimistically that means it works fine for most customers... </font><font size=3><br></font><font size=2 face="sans-serif"><br>So sorry, something unusual about your installation or usage...</font><font size=3><br></font><font size=2 face="sans-serif"><br></font><font size=3><br><br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href=http://spectrumscale.org/><font size=3 color=blue><u>spectrumscale.org</u></font></a><font size=3 color=blue><u><br></u></font><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><font size=3 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></a><br><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br></font></tt><br><br><BR>