<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Hi Marc,
<div class=""><br class="">
</div>
<div class="">Two things:</div>
<div class=""><br class="">
</div>
<div class="">1. I have a PMR open now.</div>
<div class=""><br class="">
</div>
<div class="">2. You *may* have identified the problem … I’m still checking … but files with hard links may be our problem. I wrote a simple Perl script to interate over the log file I had mmapplypolicy create. Here’s the code (don’t laugh, I’m a SysAdmin,
not a programmer, and I whipped this out in < 5 minutes … and yes, I realize the fact that I used Perl instead of Python shows my age as well <grin>):</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">#!/usr/bin/perl</div>
<div class="">#</div>
<div class="">use strict;</div>
<div class="">use warnings;</div>
<div class="">my $InputFile = "/tmp/mmapplypolicy.gpfs23.log";</div>
<div class="">my $TotalFiles = 0;</div>
<div class="">my $TotalLinks = 0;</div>
<div class="">my $TotalSize = 0;</div>
<div class="">open INPUT, $InputFile or die "Couldn\'t open $InputFile for read: $!\n";</div>
<div class="">while (<INPUT>) {</div>
<div class=""> next unless /MIGRATED/;</div>
<div class=""> $TotalFiles++;</div>
<div class=""> my $FileName = (split / /)[3];</div>
<div class=""> if ( -f $FileName ) { # some files may have been deleted since mmapplypolicy ran</div>
<div class=""> my ($NumLinks, $FileSize) = (stat($FileName))[3,7];</div>
<div class=""> $TotalLinks += $NumLinks;</div>
<div class=""> $TotalSize += $FileSize;</div>
<div class=""> }</div>
<div class="">}</div>
<div class="">close INPUT;</div>
<div class="">print "Number of files / links = $TotalFiles / $TotalLinks, Total size = $TotalSize\n";</div>
<div class="">exit 0;</div>
</div>
<div class=""><br class="">
</div>
<div class="">And here’s what it kicked out:</div>
<div class=""><br class="">
</div>
<div class="">Number of files / links = 1620263 / 80818483, Total size = 53966202814094</div>
<div class=""><br class="">
</div>
<div class="">1.6 million files but 80 million hard links!!!</div>
<div class=""><br class="">
</div>
<div class="">I’m doing some checking right now, but it appears that it is one particular group - and therefore one particular fileset - that is responsible for this … they’ve got thousands of files with 50 or more hard links each … and they’re not inconsequential
in size.</div>
<div class=""><br class="">
</div>
<div class="">IIRC (and keep in mind I’m far from a GPFS policy guru), there is a way to say something to the effect of “and the path does not contain /gpfs23/fileset/path” … may need a little help getting that right.</div>
<div class=""><br class="">
</div>
<div class="">I’ll post this information to the ticket as well but wanted to update the list. This wouldn’t be the first time we were an “edge case” for something in GPFS… ;-)</div>
<div class=""><br class="">
</div>
<div class="">Thanks...</div>
<div class=""><br class="">
</div>
<div class="">Kevin</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Apr 18, 2017, at 10:11 AM, Marc A Kaplan <<a href="mailto:makaplan@us.ibm.com" class="">makaplan@us.ibm.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class=""><font size="2" face="sans-serif" class="">ANYONE else reading this saga? Who uses mmapplypolicy to migrate files within multi-TB file systems? Problems? Or all working as expected?</font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">------</font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">Well, again mmapplypolicy "thinks" it has "chosen" 1.6 million files whose total size is 61 Terabytes and migrating those will bring the occupancy of gpfs23capacity pool to 98% and then we're done.</font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">So now I'm wondering where this is going wrong. Is there some bug in the reckoning inside of mmapplypolicy or somewhere else in GPFS?</font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">Sure you can put in an PMR, and probably should. I'm guessing whoever picks up the PMR will end up calling or emailing me ... but maybe she can do some of the clerical work for us... </font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">While we're waiting for that... Here's what I suggest next.</font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">Add a clause ...</font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">SHOW(varchar(KB_ALLOCATED) || ' n=' || varchar(NLINK))</font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">before the WHERE clause to each of your rules.</font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">Re-run the command with options '-I test -L 2' and collect the output. </font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">We're not actually going to move any data, but we're going to look at the files and file sizes that are "chosen"...</font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">You should see 1.6 million lines that look kind of like this:</font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">/yy/dat/bigC RULE 'msx' MIGRATE FROM POOL 'system' TO POOL 'xtra' WEIGHT(inf) SHOW( 1024 n=1)</font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">Run a script over the output to add up all the SHOW() values in the lines that contain TO POOL 'gpfs23capacity' and verify that they do indeed</font><br class="">
<font size="2" face="sans-serif" class="">add up to 61TB... (The show is in KB so the SHOW numbers should add up to 61 billion).<br class="">
</font><br class="">
<font size="2" face="sans-serif" class="">That sanity checks the policy arithmetic. Let's assume that's okay.
</font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">Then the next question is whether the individual numbers are correct... Zach Giles made a suggestion... which I'll interpret as
<br class="">
find some of the biggest of those files and check that they really are that big....</font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">At this point, I really don't know, but I'm guessing there's some discrepances in the reported KB_ALLOCATED numbers for many of the files...</font><br class="">
<font size="2" face="sans-serif" class="">and/or they are "illplaced" - the data blocks aren't all in the pool FROM POOL ...</font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">HMMMM.... I just thought about this some more and added the NLINK statistic. It would be unusual for this to be a big problem, but files that are hard linked are<br class="">
not recognized by mmapplypolicy as sharing storage... <br class="">
This has not come to my attention as a significant problem -- does the file system in question have significant GBs of hard linked files?</font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">The truth is that you're the first customer/user/admin in a long time to question/examine how mmapplypolicy does its space reckoning ...
<br class="">
Optimistically that means it works fine for most customers... </font><br class="">
<br class="">
<font size="2" face="sans-serif" class="">So sorry, something unusual about your installation or usage...</font><br class="">
<br class="">
<font size="2" face="sans-serif" class=""><br class="">
</font><br class="">
<br class="">
_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a><br class="">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</body>
</html>