[gpfsug-discuss] du --apparent-size and quota
Ulrich Sibiller
u.sibiller at science-computing.de
Tue Jun 1 15:26:10 BST 2021
Hi,
I experience some strangeness that I fail to understand completely. I have a fileset that got copied
(rsynced) from one cluster to another. The reported size (mmrepquota) of the source filesystem is
800G (and due to data and metadata replication being set to 2 this effectively means 400G). After
syncing the data to the destination the size there is ~457GB.
$ mmrepquota -j srcfilesys | grep fileset
srcfileset FILESET 800 800 800 0 none | 863 0 0
0 none
$ mmrepquota -j dstfilesys | grep fileset
fileset root FILESET 457 400 400 0 none | 853 0
0 0 none
(note: on the destination filesystem we have set
ignoreReplicationOnStatfs yes
IgnoreReplicaSpaceOnStat yes
ignoreReplicationForQuota yes
so there's no need to to divisions
)
(note2: the destination hat 10 files less. These where small leftover .nfs* files:
$ du --block-size 1 /srcfilesys/fileset/.nfs*
1024 /srvfilesys/fileset//.nfs0000000000f8a57800000505
1024 /srvfilesys/fileset//.nfs0000000002808f4d000000cb
1024 /srvfilesys/fileset//.nfs0000000002af44db00005509
1024 /srvfilesys/fileset//.nfs00000000034eb9270000072a
1024 /srvfilesys/fileset//.nfs0000000003a9b48300002974
1024 /srvfilesys/fileset//.nfs0000000003d10f990000028a
$ du --apparent-size --block-size 1 /srcfilesys/fileset/.nfs*
524 /srvfilesys/fileset//.nfs0000000000f8a57800000505
524 /srvfilesys/fileset//.nfs0000000002808f4d000000cb
524 /srvfilesys/fileset//.nfs0000000002af44db00005509
524 /srvfilesys/fileset//.nfs00000000034eb9270000072a
524 /srvfilesys/fileset//.nfs0000000003a9b48300002974
524 /srvfilesys/fileset//.nfs0000000003d10f990000028a
)
While trying to understand what's going on here I found this on the source file system (which is
valid for all files, with different number of course):
$ du --block-size 1 /srcfilesys/fileset/filename
65536 /srcfilesys/fileset/filename
$ du --apparent-size --block-size 1 /srcfilesys/fileset/filename
3994 /srcfilesys/fileset/filename
$ stat /srcfilesys/fileset/filename
File: ‘/srcfilesys/fileset/filename’
Size: 3994 Blocks: 128 IO Block: 1048576 regular file
Device: 2ah/42d Inode: 23266095 Links: 1
Access: (0660/-rw-rw----) Uid: (73018/ cpunnoo) Gid: (50070/ dc-rti)
Context: system_u:object_r:unlabeled_t:s0
Access: 2021-05-12 20:10:13.814459305 +0200
Modify: 2020-07-16 11:08:41.631006000 +0200
Change: 2020-07-16 11:08:41.630896177 +0200
Birth: -
If I sum up the disk usage of the first du I end up with 799.986G in total which matches the
mmrepquota output. If I add up the disk usage of the second du I end up at 456.569G _which matches
the mmrepquota output on the destination system_.
So on the source filesystem the quota seems to add up the apparent size while on the destination
filesystem the quota value is the sum of the du without --apparent-size.
Running the dus on the destination filesystem reports other numbers:
$ du --block-size 1 /dstfilesys/fileset/filename
8192 /dstfilesys/fileset/filename
$ du --apparent-size --block-size 1 /dstfilesys/fileset/filename
3994 /dstfilesys/fileset/filename
$ stat /dstfilesys/fileset/filename
File: /dstfilesys/fileset/filename
Size: 3994 Blocks: 16 IO Block: 4194304 regular file
Device: 3dh/61d Inode: 2166358719 Links: 1
Access: (0660/-rw-rw----) Uid: (73018/ cpunnoo) Gid: (50070/ dc-rti)
Access: 2021-05-29 07:52:56.069440382 +0200
Modify: 2020-07-16 11:08:41.631006000 +0200
Change: 2021-05-12 20:10:13.970443145 +0200
Birth: -
Summing them up shows almost identical numbers for both dus: 467528 467527 which I really do not get
at all...
So is there an explanation of how mmrepquota and du and du --apparent-size are related?
Uli
PS: Some more details:
The source filesystem is RHEL7 with gpfs.base 5.0.5-5:
flag value description
------------------- ------------------------ -----------------------------------
-f 32768 Minimum fragment (subblock) size in bytes
-i 4096 Inode size in bytes
-I 32768 Indirect block size in bytes
-m 2 Default number of metadata replicas
-M 2 Maximum number of metadata replicas
-r 2 Default number of data replicas
-R 2 Maximum number of data replicas
-j scatter Block allocation type
-D nfs4 File locking semantics in effect
-k all ACL semantics in effect
-n 32 Estimated number of nodes that will mount file system
-B 1048576 Block size
-Q user;group;fileset Quotas accounting enabled
user;group;fileset Quotas enforced
none Default quotas enabled
--perfileset-quota No Per-fileset quota enforcement
--filesetdf Yes Fileset df enabled?
-V 16.00 (4.2.2.0) Current file system version
14.10 (4.1.0.4) Original file system version
--create-time Tue Feb 3 11:46:10 2015 File system creation time
-z No Is DMAPI enabled?
-L 4194304 Logfile size
-E Yes Exact mtime mount option
-S No Suppress atime mount option
-K whenpossible Strict replica allocation option
--fastea Yes Fast external attributes enabled?
--encryption No Encryption enabled?
--inode-limit 179217920 Maximum number of inodes in all inode spaces
--log-replicas 0 Number of log replicas
--is4KAligned Yes is4KAligned?
--rapid-repair No rapidRepair enabled?
--write-cache-threshold 0 HAWC Threshold (max 65536)
--subblocks-per-full-block 32 Number of subblocks per full block
-P system Disk storage pools in file system
--file-audit-log No File Audit Logging enabled?
--maintenance-mode No Maintenance Mode enabled?
-d
NSDDCS1N01;NSDDCS1N02;NSDDCS1N03;NSDDCS1N04;NSDDCS1N05;NSDDCS1N06;NSDDCS1N07;NSDDCS1N08;NSDDCS1N09;NSDDCS1N10;NSDDCS1N11;NSDDCS1N12;NSDNECE54001N01;NSDNECE54001N02;
-d
NSDNECE54001N03;NSDNECE54001N04;NSDNECE54001N05;NSDNECE54001N06;NSDNECE54001N07;NSDNECE54001N08;NSDNECE54001N09;NSDNECE54001N10;NSDNECE54001N11;NSDNECE54001N12;DESC1
Disks in file system
-A yes Automatic mount option
-o none Additional mount options
-T /srcfilesys Default mount point
--mount-priority 0 Mount priority
The destination cluster is RHEL8 with gpfs.base-5.1.0-3.x86_64:
flag value description
------------------- ------------------------ -----------------------------------
-f 8192 Minimum fragment (subblock) size in bytes
-i 4096 Inode size in bytes
-I 32768 Indirect block size in bytes
-m 2 Default number of metadata replicas
-M 2 Maximum number of metadata replicas
-r 2 Default number of data replicas
-R 2 Maximum number of data replicas
-j scatter Block allocation type
-D nfs4 File locking semantics in effect
-k all ACL semantics in effect
-n 32 Estimated number of nodes that will mount file system
-B 4194304 Block size
-Q user;group;fileset Quotas accounting enabled
none Quotas enforced
none Default quotas enabled
--perfileset-quota yes Per-fileset quota enforcement
--filesetdf yes Fileset df enabled?
-V 23.00 (5.0.5.0) File system version
--create-time Tue May 11 16:51:27 2021 File system creation time
-z no Is DMAPI enabled?
-L 33554432 Logfile size
-E yes Exact mtime mount option
-S relatime Suppress atime mount option
-K whenpossible Strict replica allocation option
--fastea yes Fast external attributes enabled?
--encryption no Encryption enabled?
--inode-limit 105259008 Maximum number of inodes in all inode spaces
--log-replicas 0 Number of log replicas
--is4KAligned yes is4KAligned?
--rapid-repair yes rapidRepair enabled?
--write-cache-threshold 0 HAWC Threshold (max 65536)
--subblocks-per-full-block 512 Number of subblocks per full block
-P system Disk storage pools in file system
--file-audit-log no File Audit Logging enabled?
--maintenance-mode no Maintenance Mode enabled?
-d RG001VS021;RG002VS021;RG001VS022;RG002VS022 Disks in file system
-A yes Automatic mount option
-o none Additional mount options
-T /dstfilesys Default mount point
--mount-priority 0 Mount priority
--
Science + Computing AG
Vorstandsvorsitzender/Chairman of the board of management:
Dr. Martin Matzke
Vorstand/Board of Management:
Matthias Schempp, Sabine Hohenstein
Vorsitzender des Aufsichtsrats/
Chairman of the Supervisory Board:
Philippe Miltin
Aufsichtsrat/Supervisory Board:
Martin Wibbe, Ursula Morgenstern
Sitz/Registered Office: Tuebingen
Registergericht/Registration Court: Stuttgart
Registernummer/Commercial Register No.: HRB 382196
More information about the gpfsug-discuss
mailing list