[gpfsug-discuss] du --apparent-size and quota

[Ulrich Sibiller]

On 6/2/21 4:12 PM, IBM Spectrum Scale wrote:
> The data and metadata replications are 2 on both source and destination filesystems, so from:
> 
> $ mmrepquota -j srcfilesys | grep fileset
> srcfileset FILESET         800        800        800          0     none |      863       0        0
>         0     none
> 
> $ mmrepquota -j dstfilesys | grep fileset
> fileset root       FILESET         457        400        400          0     none |      853       0
>        0        0     none
> 
> the quota data should be changed from 800G to 457G (or 400G to 228.5G), after "rsync -AHS".

Why?

Did you notice that on the dstfilesys we have

    ignoreReplicationOnStatfs yes
    IgnoreReplicaSpaceOnStat yes
    ignoreReplicationForQuota yes

while the srcfilesys has

    ignoreReplicaSpaceOnStat 0
    ignoreReplicationForQuota 0
    ignoreReplicationOnStatfs 0
?

Changing the quota limit to 457 on the dstfilesys will surely help for the user but I still would 
like to understand why that happens? Losing > 10% of space when migrating to a newer filesystem is 
not something you'd expect. dstfilesys is ~6PB, so this means we lose more than 600TB, which is a 
serious issue I'd like to understand in detail (and maybe take countermeasures).

> Do you have sparse files on the first filesystem? Since the second filesystem
> has a larger blocksize than the first one, the copied file may not be sparse on the
> second filesystem. I think gpfs only supports holes that line up will a full filesystem
> block.

Maybe that's an issue, but I
a) use rsync -S so I guess the sparse files will be handled in the most compatible way
b) have no idea how to check this reliably


> mmrepquota reports without the --block-size parameter the size in units of 1KiB, so (if no ill-advised copy-paste editing confuses us) we are not talking about 400GiB but 400KiB.
> With just 863 files (from the inode part of the repquota output) and therefore 0.5KiB/file on average that could be explained by the sub-block size(although many files should vanish in the inodes).
> If it's 400GiB in 863 files with 500MiB/File the subblock overhead would not matter at all!

Upps, you are right in assuming a copy-and-paste accident, I had called mmrepquota with --block-size 
G. So the values we are talking about are really GiB, not KiB.

Uli


-- 
Science + Computing AG
Vorstandsvorsitzender/Chairman of the board of management:
Dr. Martin Matzke
Vorstand/Board of Management:
Matthias Schempp, Sabine Hohenstein
Vorsitzender des Aufsichtsrats/
Chairman of the Supervisory Board:
Philippe Miltin
Aufsichtsrat/Supervisory Board:
Martin Wibbe, Ursula Morgenstern
Sitz/Registered Office: Tuebingen
Registergericht/Registration Court: Stuttgart
Registernummer/Commercial Register No.: HRB 382196