[gpfsug-discuss] Compression question

Cregan, Bob b.cregan at imperial.ac.uk
Thu Nov 28 13:05:43 GMT 2019


Hi
     5.0.2 with a hotfix for a lz4 compression bug

Bob



Bob Cregan
HPC Systems Analyst

Information & Communication Technologies

Imperial College London,
South Kensington Campus London, SW7 2AZ
T: 07712388129
E: b.cregan at imperial.ac.uk

W: www.imperial.ac.uk/ict/rcs<http://www.imperial.ac.uk/ict/rcs>

[1505984389175_twitter.png] @imperialRCS @imperialRSE

[1505983882959_Imperial-RCS.png]


________________________________
From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Luis Bolinches <luis.bolinches at fi.ibm.com>
Sent: 28 November 2019 13:00
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Compression question


Caution - This email from luis.bolinches at fi.ibm.com originated outside Imperial



Which version are you running?

I was involved on a big for compressed file sets and snapshots that were related to what you see.

--
Cheers

On 28. Nov 2019, at 14.57, Cregan, Bob <b.cregan at imperial.ac.uk> wrote:


Hi
      Sounds logical - except the snap metadata does not have the compression flag set. So if the inode now points to a set of compressed blocks how does the client know to decompress it?

After compression of an existing file we get in the snap


-bash-4.2$ mmlsattr -L .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf

file name:            .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf

metadata replication: 2 max 2

data replication:     1 max 3

immutable:            no

appendOnly:           no

flags:

storage pool name:    sata1

fileset name:         userdirs

snapshot name:        @GMT-2019.11.27-19.30.14

creation time:        Tue Mar  5 16:16:40 2019

Misc attributes:      ARCHIVE

Encrypted:            no

 and the original file is definitely compressed.


-bash-4.2$ mmlsattr -L UserGuide_13.06.pdf

file name:            UserGuide_13.06.pdf

metadata replication: 2 max 2

data replication:     1 max 3

immutable:            no

appendOnly:           no

flags:

storage pool name:    sata1

fileset name:         userdirs

snapshot name:

creation time:        Tue Mar  5 16:16:40 2019

Misc attributes:      ARCHIVE COMPRESSION (library z)

Encrypted:            no

Bob




Bob Cregan
HPC Systems Analyst

Information & Communication Technologies

Imperial College London,
South Kensington Campus London, SW7 2AZ
T: 07712388129
E: b.cregan at imperial.ac.uk

W: www.imperial.ac.uk/ict/rcs<http://www.imperial.ac.uk/ict/rcs>

<Outlook-1505984389.png>
 @imperialRCS @imperialRSE

<Outlook-1505983882.png>



________________________________
From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Daniel Kidger <daniel.kidger at uk.ibm.com>
Sent: 28 November 2019 12:30
To: gpfsug-discuss at spectrumscale.org <gpfsug-discuss at spectrumscale.org>
Cc: gpfsug-discuss at spectrumscale.org <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Compression question


Caution - This email from daniel.kidger at uk.ibm.com originated outside Imperial



Alexander,

Can you then confirm then that the inodes in the snapshot will now point to fewer but compressed blocks ?
Daniel

_________________________________________________________
Daniel Kidger
IBM Technical Sales Specialist
Spectrum Scale, Spectrum NAS and IBM Cloud Object Store

+44-(0)7818 522 266
daniel.kidger at uk.ibm.com
[https://images.youracclaim.com/images/c49300ae-d13e-4071-90f5-15f59d199c9e/IBM%2BVolunteers%2BGold%2Bv6.png]<https://www.youracclaim.com/badges/687cf790-fe65-4a92-b129-d23ae41862ac/public_url>       [https://images.youracclaim.com/images/f2539224-f951-46b4-b376-b88f21c2be98/IBM-Selling-Certification---Level-1.png] <https://www.youracclaim.com/badges/8153c6a7-3e02-40be-87ee-24e27ae9459c/public_url>       [https://images.youracclaim.com/images/ea52b12f-97ac-4e72-8d24-b0ced8054e7d/Storage%2BTechnical%2BV1.png] <https://www.youracclaim.com/badges/78197e2c-4277-4ec9-808b-ad6abe1e1b16/public_url>



----- Original message -----
From: "Alexander Wolf" <A.Wolf-Reber at de.ibm.com>
Sent by: gpfsug-discuss-bounces at spectrumscale.org
To: gpfsug-discuss at spectrumscale.org
Cc:
Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
Date: Thu, Nov 28, 2019 12:21

I just tested this. Compressing a file did free up space in the file system. Looks like our compression code does not trigger COW on the snapshot. You can test this yourself by looking into mmlssnapshot -d (please not on a large production fs, this command is expensive).

Mit freundlichen Grüßen / Kind regards

<Image.15749424191280.png>

<Image.15749424191281.png>

  *

  *

  *   Dr. Alexander Wolf-Reber
Spectrum Scale Release Lead Architect
Department M069 / Spectrum Scale Software Development

+49-160-90540880
a.wolf-reber at de.ibm.com

<Image.15749424191282.png>
IBM Data Privacy Statement<https://www.ibm.com/privacy/us/en/>

IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294



----- Original message -----
From: "Luis Bolinches" <luis.bolinches at fi.ibm.com>
Sent by: gpfsug-discuss-bounces at spectrumscale.org
To: gpfsug-discuss at spectrumscale.org
Cc: gpfsug-discuss at spectrumscale.org
Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
Date: Thu, Nov 28, 2019 11:47

Hi

Same principle COW. The data blocks do not get modified.
--
Ystävällisin terveisin / Kind regards / Saludos cordiales / Salutations / Salutacions
Luis Bolinches
Consultant IT Specialist
Mobile Phone: +358503112585
https://www.youracclaim.com/user/luis-bolinches

"If you always give you will always have" --  Anonymous



----- Original message -----
From: "Cregan, Bob" <b.cregan at imperial.ac.uk>
Sent by: gpfsug-discuss-bounces at spectrumscale.org
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Cc:
Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
Date: Thu, Nov 28, 2019 12:06 PM

Just to clarify this is SS compression, so

mmchattr --compression yes <filename>

or an ILM equivalent

So not a regular modification.

Bob






Bob Cregan
HPC Systems Analyst

Information & Communication Technologies

Imperial College London,
South Kensington Campus London, SW7 2AZ
T: 07712388129
E: b.cregan at imperial.ac.uk

W: www.imperial.ac.uk/ict/rcs<http://www.imperial.ac.uk/ict/rcs>

<Image.f7e9a6a7-c4ab-4a2c-a979-4513774aa054.png>
 @imperialRCS @imperialRSE

<Image.dbd2fd87-b9f2-4b15-903f-c74c7b7298d3.png>










________________________________
From: Cregan, Bob
Sent: 28 November 2019 09:43
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Compression question

Hi All,
           Can someone answer the following question on compression in a snapshot context? We have tried various experiments and they are inconclusive - too tedious to go into the detail.

What happens to the snapshot when a file is compressed in SS? The logic as I see it is

####### In a non compressed situation ###############

1) create a file,
2) create a snapshot.
3) modify a file in a normal way - the blocks on disk are changed and the old blocks are then written to the snap.


######In a compressed situation ############



1) create a file,
2) create a snapshot.
3) Compress the file. Now IBM says the blocks are rewritten when the file is compressed. So do the old uncompressed blocks go into the snap? If so we now have 2 copies of the file and unless the compression > 50%  we have used more space until the snap is deleted.

You get the space back in the end, but if you are in a tight situation then potentially compression might not work for you in the short term.
Thanks

Bob





Bob Cregan
HPC Systems Analyst

Information & Communication Technologies

Imperial College London,
South Kensington Campus London, SW7 2AZ
T: 07712388129
E: b.cregan at imperial.ac.uk

W: www.imperial.ac.uk/ict/rcs<http://www.imperial.ac.uk/ict/rcs>

<Image.ba38b5d6-12d6-476a-bcd8-7d647bfab5b3.png>
 @imperialRCS @imperialRSE

<Image.e34493c2-d35b-4fff-a6df-ce87e2ae5ed4.png>









_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


Ellei edellä ole toisin mainittu: / Unless stated otherwise above:
Oy IBM Finland Ab
PL 265, 00101 Helsinki, Finland
Business ID, Y-tunnus: 0195876-3
Registered in Finland

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU


Ellei edellä ole toisin mainittu: / Unless stated otherwise above:
Oy IBM Finland Ab
PL 265, 00101 Helsinki, Finland
Business ID, Y-tunnus: 0195876-3
Registered in Finland

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20191128/15c219a3/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Outlook-1505984389.png
Type: image/png
Size: 1641 bytes
Desc: Outlook-1505984389.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20191128/15c219a3/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Outlook-1505983882.png
Type: image/png
Size: 17519 bytes
Desc: Outlook-1505983882.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20191128/15c219a3/attachment-0005.png>


More information about the gpfsug-discuss mailing list