[gpfsug-discuss] Compression question

Luis Bolinches luis.bolinches at fi.ibm.com
Thu Nov 28 14:07:27 GMT 2019


That is the fix. Before it was an error. 


So blocks that are pointed from active get compressed. Ones that are no longer linked are not modified. 

--
Cheers

> On 28. Nov 2019, at 16.03, Alexander Wolf <A.Wolf-Reber at de.ibm.com> wrote:
> 
> 
> I see the same behavior of mmlsattr on my system (with some post 5.0.4 development build). Funny enough if I look at the file content in the snapshot it gets properly decompressed.
>  
> Mit freundlichen Grüßen / Kind regards
> <Image.15749424191286.png>
> 
> <Image.15749424191287.png>
>  
>      
> Dr. Alexander Wolf-Reber
> Spectrum Scale Release Lead Architect
> Department M069 / Spectrum Scale Software Development
> 
> +49-160-90540880
> a.wolf-reber at de.ibm.com
> <Image.15749424191288.png>
> IBM Data Privacy Statement
> IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Geschäftsführung: Dirk Wittkopp
> Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
> 
>  
>  
> ----- Original message -----
> From: "Luis Bolinches" <luis.bolinches at fi.ibm.com>
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> To: "gpfsug main discussion list" <gpfsug-discuss at spectrumscale.org>
> Cc:
> Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
> Date: Thu, Nov 28, 2019 14:00
>  
> Which version are you running?
>  
> I was involved on a big for compressed file sets and snapshots that were related to what you see. 
>  
> --
> Cheers
>  
>> 
>>> On 28. Nov 2019, at 14.57, Cregan, Bob <b.cregan at imperial.ac.uk> wrote:
>>>  
>> 
>> Hi 
>>       Sounds logical - except the snap metadata does not have the compression flag set. So if the inode now points to a set of compressed blocks how does the client know to decompress it?
>>  
>> After compression of an existing file we get in the snap
>>  
>> -bash-4.2$ mmlsattr -L .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf
>> file name:            .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf
>> metadata replication: 2 max 2
>> data replication:     1 max 3
>> immutable:            no
>> appendOnly:           no
>> flags:                
>> storage pool name:    sata1
>> fileset name:         userdirs
>> snapshot name:        @GMT-2019.11.27-19.30.14
>> creation time:        Tue Mar  5 16:16:40 2019
>> Misc attributes:      ARCHIVE
>> Encrypted:            no
>>  
>>  and the original file is definitely compressed.
>>  
>> -bash-4.2$ mmlsattr -L UserGuide_13.06.pdf 
>> file name:            UserGuide_13.06.pdf
>> metadata replication: 2 max 2
>> data replication:     1 max 3
>> immutable:            no
>> appendOnly:           no
>> flags:                
>> storage pool name:    sata1
>> fileset name:         userdirs
>> snapshot name:        
>> creation time:        Tue Mar  5 16:16:40 2019
>> Misc attributes:      ARCHIVE COMPRESSION (library z)
>> Encrypted:            no
>> Bob
>>  
>>  
>>  
>>  
>> Bob Cregan
>> HPC Systems Analyst
>> Information & Communication Technologies
>> Imperial College London, 
>> South Kensington Campus London, SW7 2AZ
>> T: 07712388129
>> E: b.cregan at imperial.ac.uk
>> W: www.imperial.ac.uk/ict/rcs
>>  
>> <Outlook-1505984389.png>
>>  @imperialRCS @imperialRSE
>>  
>> 
>>  
>> <Outlook-1505983882.png>
>>  
>>  
>> 
>>  
>>  
>>  
>>  
>>  
>> From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Daniel Kidger <daniel.kidger at uk.ibm.com>
>> Sent: 28 November 2019 12:30
>> To: gpfsug-discuss at spectrumscale.org <gpfsug-discuss at spectrumscale.org>
>> Cc: gpfsug-discuss at spectrumscale.org <gpfsug-discuss at spectrumscale.org>
>> Subject: Re: [gpfsug-discuss] Compression question
>>  
>> Caution - This email from daniel.kidger at uk.ibm.com originated outside Imperial
>>  
>> 
>> Alexander,
>>  
>> Can you then confirm then that the inodes in the snapshot will now point to fewer but compressed blocks ?
>> Daniel
>>  
>> _________________________________________________________
>> Daniel Kidger
>> IBM Technical Sales Specialist
>> Spectrum Scale, Spectrum NAS and IBM Cloud Object Store
>> 
>> +44-(0)7818 522 266 
>> daniel.kidger at uk.ibm.com
>> 		
>>  
>>  
>>  
>> ----- Original message -----
>> From: "Alexander Wolf" <A.Wolf-Reber at de.ibm.com>
>> Sent by: gpfsug-discuss-bounces at spectrumscale.org
>> To: gpfsug-discuss at spectrumscale.org
>> Cc:
>> Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
>> Date: Thu, Nov 28, 2019 12:21
>>  
>> I just tested this. Compressing a file did free up space in the file system. Looks like our compression code does not trigger COW on the snapshot. You can test this yourself by looking into mmlssnapshot -d (please not on a large production fs, this command is expensive).
>>  
>> Mit freundlichen Grüßen / Kind regards
>>  
>> 
>> <Image.15749424191280.png>
>>  
>> 
>> <Image.15749424191281.png>
>>  
>>      
>> Dr. Alexander Wolf-Reber
>> Spectrum Scale Release Lead Architect
>> Department M069 / Spectrum Scale Software Development
>> 
>> +49-160-90540880
>> a.wolf-reber at de.ibm.com
>> <Image.15749424191282.png>
>> IBM Data Privacy Statement
>> IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Geschäftsführung: Dirk Wittkopp
>> Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
>> 
>>  
>>  
>> ----- Original message -----
>> From: "Luis Bolinches" <luis.bolinches at fi.ibm.com>
>> Sent by: gpfsug-discuss-bounces at spectrumscale.org
>> To: gpfsug-discuss at spectrumscale.org
>> Cc: gpfsug-discuss at spectrumscale.org
>> Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
>> Date: Thu, Nov 28, 2019 11:47
>>  
>> Hi
>>  
>> Same principle COW. The data blocks do not get modified.
>> --
>> Ystävällisin terveisin / Kind regards / Saludos cordiales / Salutations / Salutacions
>> Luis Bolinches
>> Consultant IT Specialist
>> Mobile Phone: +358503112585
>> https://www.youracclaim.com/user/luis-bolinches
>> 
>> "If you always give you will always have" --  Anonymous
>>  
>>  
>>  
>> ----- Original message -----
>> From: "Cregan, Bob" <b.cregan at imperial.ac.uk>
>> Sent by: gpfsug-discuss-bounces at spectrumscale.org
>> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
>> Cc:
>> Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question
>> Date: Thu, Nov 28, 2019 12:06 PM
>>  
>> Just to clarify this is SS compression, so 
>>  
>> mmchattr --compression yes <filename>
>>  
>> or an ILM equivalent
>>  
>> So not a regular modification.
>>  
>> Bob
>>  
>>  
>>  
>> Bob Cregan
>> HPC Systems Analyst
>> Information & Communication Technologies
>> Imperial College London, 
>> South Kensington Campus London, SW7 2AZ
>> T: 07712388129
>> E: b.cregan at imperial.ac.uk
>> W: www.imperial.ac.uk/ict/rcs
>>  
>> <Image.f7e9a6a7-c4ab-4a2c-a979-4513774aa054.png>
>>  @imperialRCS @imperialRSE
>>  
>> 
>>  
>> <Image.dbd2fd87-b9f2-4b15-903f-c74c7b7298d3.png>
>>  
>> 
>>  
>>  
>>  
>>  
>>  
>> From: Cregan, Bob
>> Sent: 28 November 2019 09:43
>> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
>> Subject: Compression question
>>  
>> Hi All,
>>            Can someone answer the following question on compression in a snapshot context? We have tried various experiments and they are inconclusive - too tedious to go into the detail.
>>  
>> What happens to the snapshot when a file is compressed in SS? The logic as I see it is 
>>  
>> ####### In a non compressed situation ###############
>>  
>> 1) create a file,
>> 2) create a snapshot.
>> 3) modify a file in a normal way - the blocks on disk are changed and the old blocks are then written to the snap. 
>>  
>> ######In a compressed situation ############
>>  
>> 1) create a file,
>> 2) create a snapshot.
>> 3) Compress the file. Now IBM says the blocks are rewritten when the file is compressed. So do the old uncompressed blocks go into the snap? If so we now have 2 copies of the file and unless the compression > 50%  we have used more space until the snap is deleted.
>>  
>> You get the space back in the end, but if you are in a tight situation then potentially compression might not work for you in the short term.
>> Thanks
>>  
>> Bob
>>  
>>  
>> Bob Cregan
>> HPC Systems Analyst
>> Information & Communication Technologies
>> Imperial College London, 
>> South Kensington Campus London, SW7 2AZ
>> T: 07712388129
>> E: b.cregan at imperial.ac.uk
>> W: www.imperial.ac.uk/ict/rcs
>>  
>> <Image.ba38b5d6-12d6-476a-bcd8-7d647bfab5b3.png>
>>  @imperialRCS @imperialRSE
>>  
>> 
>>  
>> <Image.e34493c2-d35b-4fff-a6df-ce87e2ae5ed4.png>
>>  
>> 
>>  
>>  
>>  
>>  
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss 
>>  
>> 
>> Ellei edellä ole toisin mainittu: / Unless stated otherwise above:
>> Oy IBM Finland Ab
>> PL 265, 00101 Helsinki, Finland
>> Business ID, Y-tunnus: 0195876-3
>> Registered in Finland
>>  
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss 
>>  
>>  
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss 
>>  
>> Unless stated otherwise above:
>> IBM United Kingdom Limited - Registered in England and Wales with number 741598.
>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>>  
> 
> Ellei edellä ole toisin mainittu: / Unless stated otherwise above:
> Oy IBM Finland Ab
> PL 265, 00101 Helsinki, Finland
> Business ID, Y-tunnus: 0195876-3
> Registered in Finland
>  
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss 
>  
> 

Ellei edellä ole toisin mainittu: / Unless stated otherwise above:
Oy IBM Finland Ab
PL 265, 00101 Helsinki, Finland
Business ID, Y-tunnus: 0195876-3 
Registered in Finland

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20191128/88ebb5c7/attachment-0002.htm>


More information about the gpfsug-discuss mailing list