[gpfsug-discuss] Compression question
IBM Spectrum Scale
scale at us.ibm.com
Thu Nov 28 17:22:43 GMT 2019
Hai Zhong,
Can you please help the customer with their compression related query.
Regards, The Spectrum Scale (GPFS) team
------------------------------------------------------------------------------------------------------------------
If you feel that your question can benefit other users of Spectrum Scale
(GPFS), then please post it to the public IBM developerWroks Forum at
https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479.
If your query concerns a potential software error in Spectrum Scale (GPFS)
and you have an IBM software maintenance contract please contact
1-800-237-5511 in the United States or your local IBM Service Center in
other countries.
The forum is informally monitored as time permits and should not be used
for priority messages to the Spectrum Scale (GPFS) team.
From: "Daniel Kidger" <daniel.kidger at uk.ibm.com>
To: gpfsug-discuss at spectrumscale.org
Cc: gpfsug-discuss at spectrumscale.org
Date: 28-11-2019 20:58
Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
Sent by: gpfsug-discuss-bounces at spectrumscale.org
Olaf's explanation makes excellent sense.
So is this definitely the case? ..
The now snapshotted inode is not updated when a live file is
compressed, as it point to the current inode which itself has compressed
blocks (and the compressed flag set).
And likewise for deeper (older) snapshots.
sDaniel
_________________________________________________________
Daniel Kidger
IBM Technical Sales Specialist
Spectrum Scale, Spectrum NAS and IBM Cloud Object Store
+44-(0)7818 522 266
daniel.kidger at uk.ibm.com
----- Original message -----
From: "Olaf Weiser" <olaf.weiser at de.ibm.com>
Sent by: gpfsug-discuss-bounces at spectrumscale.org
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Cc:
Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
Date: Thu, Nov 28, 2019 15:01
Hi Alex,
not 100% sure about my answer.. but so far as I see it.. it is working,
because of the so called "dito resolution " .. In the snaphost's inode ..
die pointer to the DA's point the the next (more recent) inode
information ..
so accessing a file in a snapshot- "redirects" the request to the origin
inode - and there ..the information about compression is given and points
to the origin DA
(of course.. only as long nobody changed the file since the snapshot was
taken)
From: "Alexander Wolf" <A.Wolf-Reber at de.ibm.com>
To: gpfsug-discuss at spectrumscale.org
Date: 11/28/2019 07:03 AM
Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
Sent by: gpfsug-discuss-bounces at spectrumscale.org
I see the same behavior of mmlsattr on my system (with some post 5.0.4
development build). Funny enough if I look at the file content in the
snapshot it gets properly decompressed.
Mit freundlichen Grüßen / Kind regards
IBM Spectrum Scale Dr. Alexander Wolf-Reber
Spectrum Scale Release Lead Architect
Department M069 / Spectrum Scale Software Development
+49-160-90540880
a.wolf-reber at de.ibm.com
IBM Data Privacy Statement
IBM Deutschland Research & Development GmbH / Vorsitzende des
Aufsichtsrats: Matthias Hartmann / Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart,
HRB 243294
----- Original message -----
From: "Luis Bolinches" <luis.bolinches at fi.ibm.com>
Sent by: gpfsug-discuss-bounces at spectrumscale.org
To: "gpfsug main discussion list" <gpfsug-discuss at spectrumscale.org>
Cc:
Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
Date: Thu, Nov 28, 2019 14:00
Which version are you running?
I was involved on a big for compressed file sets and snapshots that were
related to what you see.
--
Cheers
On 28. Nov 2019, at 14.57, Cregan, Bob <b.cregan at imperial.ac.uk> wrote:
Hi
Sounds logical - except the snap metadata does not have the
compression flag set. So if the inode now points to a set of compressed
blocks how does the client know to decompress it?
After compression of an existing file we get in the snap
-bash-4.2$ mmlsattr
-L .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf
file
name: .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf
metadata replication: 2 max 2
data replication: 1 max 3
immutable: no
appendOnly: no
flags:
storage pool name: sata1
fileset name: userdirs
snapshot name: @GMT-2019.11.27-19.30.14
creation time: Tue Mar 5 16:16:40 2019
Misc attributes: ARCHIVE
Encrypted: no
and the original file is definitely compressed.
-bash-4.2$ mmlsattr -L UserGuide_13.06.pdf
file name: UserGuide_13.06.pdf
metadata replication: 2 max 2
data replication: 1 max 3
immutable: no
appendOnly: no
flags:
storage pool name: sata1
fileset name: userdirs
snapshot name:
creation time: Tue Mar 5 16:16:40 2019
Misc attributes: ARCHIVE COMPRESSION (library z)
Encrypted: no
Bob
Bob Cregan
HPC Systems Analyst
Information & Communication Technologies
Imperial College London,
South Kensington Campus London, SW7 2AZ
T: 07712388129
E: b.cregan at imperial.ac.uk
W: www.imperial.ac.uk/ict/rcs
<Outlook-1505984389.png>
@imperialRCS @imperialRSE
<Outlook-1505983882.png>
From: gpfsug-discuss-bounces at spectrumscale.org
<gpfsug-discuss-bounces at spectrumscale.org> on behalf of Daniel Kidger
<daniel.kidger at uk.ibm.com>
Sent: 28 November 2019 12:30
To: gpfsug-discuss at spectrumscale.org <gpfsug-discuss at spectrumscale.org>
Cc: gpfsug-discuss at spectrumscale.org <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Compression question
|------------------------------------------------------------------------------|
|Caution - This email from daniel.kidger at uk.ibm.com originated outside Imperial|
| |
|------------------------------------------------------------------------------|
Alexander,
Can you then confirm then that the inodes in the snapshot will now point
to fewer but compressed blocks ?
Daniel
_________________________________________________________
Daniel Kidger
IBM Technical Sales Specialist
Spectrum Scale, Spectrum NAS and IBM Cloud Object Store
+44-(0)7818 522 266
daniel.kidger at uk.ibm.com
----- Original message -----
From: "Alexander Wolf" <A.Wolf-Reber at de.ibm.com>
Sent by: gpfsug-discuss-bounces at spectrumscale.org
To: gpfsug-discuss at spectrumscale.org
Cc:
Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
Date: Thu, Nov 28, 2019 12:21
I just tested this. Compressing a file did free up space in the file
system. Looks like our compression code does not trigger COW on the
snapshot. You can test this yourself by looking into mmlssnapshot -d
(please not on a large production fs, this command is expensive).
Mit freundlichen Grüßen / Kind regards
<Image.15749424191280.png>
<Image.15749424191281.png> Dr. Alexander Wolf-Reber
Spectrum Scale Release Lead Architect
Department M069 / Spectrum Scale Software Development
+49-160-90540880
a.wolf-reber at de.ibm.com
<Image.15749424191282.png>
IBM Data Privacy Statement
IBM Deutschland Research & Development GmbH / Vorsitzende des
Aufsichtsrats: Matthias Hartmann / Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart,
HRB 243294
----- Original message -----
From: "Luis Bolinches" <luis.bolinches at fi.ibm.com>
Sent by: gpfsug-discuss-bounces at spectrumscale.org
To: gpfsug-discuss at spectrumscale.org
Cc: gpfsug-discuss at spectrumscale.org
Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
Date: Thu, Nov 28, 2019 11:47
Hi
Same principle COW. The data blocks do not get modified.
--
Ystävällisin terveisin / Kind regards / Saludos cordiales / Salutations /
Salutacions
Luis Bolinches
Consultant IT Specialist
Mobile Phone: +358503112585
https://www.youracclaim.com/user/luis-bolinches
"If you always give you will always have" -- Anonymous
----- Original message -----
From: "Cregan, Bob" <b.cregan at imperial.ac.uk>
Sent by: gpfsug-discuss-bounces at spectrumscale.org
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Cc:
Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
Date: Thu, Nov 28, 2019 12:06 PM
Just to clarify this is SS compression, so
mmchattr --compression yes <filename>
or an ILM equivalent
So not a regular modification.
Bob
Bob Cregan
HPC Systems Analyst
Information & Communication Technologies
Imperial College London,
South Kensington Campus London, SW7 2AZ
T: 07712388129
E: b.cregan at imperial.ac.uk
W: www.imperial.ac.uk/ict/rcs
<Image.f7e9a6a7-c4ab-4a2c-a979-4513774aa054.png>
@imperialRCS @imperialRSE
<Image.dbd2fd87-b9f2-4b15-903f-c74c7b7298d3.png>
From: Cregan, Bob
Sent: 28 November 2019 09:43
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Compression question
Hi All,
Can someone answer the following question on compression in a
snapshot context? We have tried various experiments and they are
inconclusive - too tedious to go into the detail.
What happens to the snapshot when a file is compressed in SS? The logic as
I see it is
####### In a non compressed situation ###############
1) create a file,
2) create a snapshot.
3) modify a file in a normal way - the blocks on disk are changed and the
old blocks are then written to the snap.
######In a compressed situation ############
1) create a file,
2) create a snapshot.
3) Compress the file. Now IBM says the blocks are rewritten when the file
is compressed. So do the old uncompressed blocks go into the snap? If so
we now have 2 copies of the file and unless the compression > 50% we have
used more space until the snap is deleted.
You get the space back in the end, but if you are in a tight situation
then potentially compression might not work for you in the short term.
Thanks
Bob
Bob Cregan
HPC Systems Analyst
Information & Communication Technologies
Imperial College London,
South Kensington Campus London, SW7 2AZ
T: 07712388129
E: b.cregan at imperial.ac.uk
W: www.imperial.ac.uk/ict/rcs
<Image.ba38b5d6-12d6-476a-bcd8-7d647bfab5b3.png>
@imperialRCS @imperialRSE
<Image.e34493c2-d35b-4fff-a6df-ce87e2ae5ed4.png>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Ellei edellä ole toisin mainittu: / Unless stated otherwise above:
Oy IBM Finland Ab
PL 265, 00101 Helsinki, Finland
Business ID, Y-tunnus: 0195876-3
Registered in Finland
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Ellei edellä ole toisin mainittu: / Unless stated otherwise above:
Oy IBM Finland Ab
PL 265, 00101 Helsinki, Finland
Business ID, Y-tunnus: 0195876-3
Registered in Finland
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=6F5tNsgC3a8tnTmVCVxGFJgNmWLUYm_MUO-zYDgeQ5o&s=F1aJcNbQxqosWD5WimhGS19MpeszOyIp9CxuUib-c2Q&e=
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20191128/2bc8c444/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20191128/2bc8c444/attachment-0006.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 14065652.gif
Type: image/gif
Size: 1134 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20191128/2bc8c444/attachment-0007.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 14488338.gif
Type: image/gif
Size: 6645 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20191128/2bc8c444/attachment-0008.gif>
More information about the gpfsug-discuss
mailing list