[gpfsug-discuss] file layout API + file fragmentation

Aaron Knister aaron.s.knister at nasa.gov
Mon Nov 6 05:10:30 GMT 2017


Thanks, Frank!

That's truly fascinating and has some interesting implications that I
hadn't thought of before. I just ran a test on an ~8G fs with a block
size of 1M:

for i in `seq 1 100000`; do
	dd if=/dev/zero of=foofile${i} bs=520K count=1
done

The fs is "full" according to df/mmdf but there's 3.6G left in subblocks
but yeah, I can't allocate any new files that wouldn't fit into the
inode and I can't seem to allocate any new subblocks to existing files
(e.g. append).

What's interesting is if I do the same exercise but with a file size of
30K or even 260K I don't seem to run into the same issue. I'm not sure I
understand that yet.

I was curious about what this meant in the case of appending to a file
where the last offset in the file was allocated to a fragment. By
looking at "tsdbfs listda" and appending to a file I could see that the
last DA would change (presumably to point to the DA of the start of a
contiguous subblock) once the amount of data appended caused the file
size to exceed the space available in the trailing subblocks.

-Aaron


On 11/5/17 7:57 PM, Frank Schmuck wrote:
> In GPFS blocks within a file are never fragmented.  For example, if you
> have a file of size 7.3 MB and your file system block size is 1MB, then
> this file will be made up of 7 full blocks and one fragment of size 320k
> (10 subblocks).  Each of the 7 full blocks will be contiguous on a singe
> diks (LUN) behind a single NSD server.  The fragment that makes up the
> last part of the file will also be contiguous on a single disk, just
> shorter than a full block.
>  
> Frank Schmuck
> IBM Almaden Research Center
>  
>  
> 
>     ----- Original message -----
>     From: Aaron Knister <aaron.s.knister at nasa.gov>
>     Sent by: gpfsug-discuss-bounces at spectrumscale.org
>     To: <gpfsug-discuss at spectrumscale.org>
>     Cc:
>     Subject: Re: [gpfsug-discuss] file layout API + file fragmentation
>     Date: Sun, Nov 5, 2017 3:39 PM
>      
>     Thanks Marc, that helps. I can't easily use tsdbfs for what I'm working
>     on since it needs to be run as unprivileged users.
> 
>     Perhaps I'm not asking the right question. I'm wondering how the file
>     layout api behaves if a given "block"-aligned offset in a file is made
>     up of sub-blocks/fragments that are not all on the same NSD. The
>     assumption based on how I've seen the API used so far is that all
>     sub-blocks within a block at a given offset within a file are all on the
>     same NSD.
> 
>     -Aaron
> 
>     On 11/5/17 6:01 PM, Marc A Kaplan wrote:
>     > I googled GPFS_FCNTL_GET_DATABLKDISKIDX
>     >
>     > and found this discussion:
>     >
>     >
>      https://www.ibm.com/developerworks/community/forums/html/topic?id=db48b190-4f2f-4e24-a035-25d3e2b06b2d&ps=50
>     >
>     > In general, GPFS files ARE deliberately "fragmented" but we don't say
>     > that - we say they are "striped" over many disks -- and that is
>     > generally a good thing for parallel performance.
>     >
>     > Also, in GPFS, if the last would-be block of a file is less than a
>     > block, then it is stored in a "fragment" of a block.  
>     > So you see we use "fragment" to mean something different than
>     other file
>     > systems you may know.
>     >
>     > --marc
>     >
>     >
>     >
>     > From:        Aaron Knister <aaron.s.knister at nasa.gov>
>     > To:        gpfsug main discussion list
>     <gpfsug-discuss at spectrumscale.org>
>     > Date:        11/04/2017 12:22 PM
>     > Subject:        [gpfsug-discuss] file layout API + file
>     fragmentation
>     > Sent by:        gpfsug-discuss-bounces at spectrumscale.org
>     >
>     ------------------------------------------------------------------------
>     >
>     >
>     >
>     > I've got a question about the file layout API and how it reacts in the
>     > case of fragmented files.
>     >
>     > I'm using the GPFS_FCNTL_GET_DATABLKDISKIDX structure and have
>     some code
>     > based on tsGetDataBlk.C. I'm basing the block size based off of what's
>     > returned by filemapOut.blockSize but that only seems to return a
>     value >
>     > 0 when filemapIn.startOffset is 0.
>     >
>     > In a case where a file were to be made up of a significant number of
>     > non-contiguous fragments (which... would be awful in of itself) how
>     > would this be reported by the file layout API? Does the interface
>     > technically just report the disk location information of the first
>     block
>     > of the $blockSize range and assume that it's contiguous?
>     >
>     > Thanks!
>     >
>     > -Aaron
>     >
>     > --
>     > Aaron Knister
>     > NASA Center for Climate Simulation (Code 606.2)
>     > Goddard Space Flight Center
>     > (301) 286-2776
>     > _______________________________________________
>     > gpfsug-discuss mailing list
>     > gpfsug-discuss at spectrumscale.org
>     >
>     https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=wnR7m6d4urZ_8dM4mkHQjMbFD9xJEeesmJyzt1osCnM&s=-dgGO6O5i1EqWj-8MmzjxJ1Iz2I5gT1aRmtyP44Cvdg&e=
>     >
>     >
>     >
>     >
>     >
>     >
>     > _______________________________________________
>     > gpfsug-discuss mailing list
>     > gpfsug-discuss at spectrumscale.org
>     >
>     https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIF-g&c=jf_iaSHvJObTbx-siA1ZOg&r=ai3ddVzf50ktH78ovGv6NU4O2LZUOWLpiUiggb8lEgA&m=pUdB4fbWLD03ZTAhk9OlpRdIasz628Oa_yG8z8NOjsk&s=kisarJ7IVnyYBx05ZZiGzdwaXnPqNR8UJoywU1OJNRU&e=
>     >
> 
>     --
>     Aaron Knister
>     NASA Center for Climate Simulation (Code 606.2)
>     Goddard Space Flight Center
>     (301) 286-2776
>     _______________________________________________
>     gpfsug-discuss mailing list
>     gpfsug-discuss at spectrumscale.org
>     https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIF-g&c=jf_iaSHvJObTbx-siA1ZOg&r=ai3ddVzf50ktH78ovGv6NU4O2LZUOWLpiUiggb8lEgA&m=pUdB4fbWLD03ZTAhk9OlpRdIasz628Oa_yG8z8NOjsk&s=kisarJ7IVnyYBx05ZZiGzdwaXnPqNR8UJoywU1OJNRU&e=
>      
> 
>  
> 
> 
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> 

-- 
Aaron Knister
NASA Center for Climate Simulation (Code 606.2)
Goddard Space Flight Center
(301) 286-2776



More information about the gpfsug-discuss mailing list