[gpfsug-discuss] file layout API + file fragmentation
Aaron Knister
aaron.s.knister at nasa.gov
Mon Nov 6 05:10:30 GMT 2017
Thanks, Frank!
That's truly fascinating and has some interesting implications that I
hadn't thought of before. I just ran a test on an ~8G fs with a block
size of 1M:
for i in `seq 1 100000`; do
dd if=/dev/zero of=foofile${i} bs=520K count=1
done
The fs is "full" according to df/mmdf but there's 3.6G left in subblocks
but yeah, I can't allocate any new files that wouldn't fit into the
inode and I can't seem to allocate any new subblocks to existing files
(e.g. append).
What's interesting is if I do the same exercise but with a file size of
30K or even 260K I don't seem to run into the same issue. I'm not sure I
understand that yet.
I was curious about what this meant in the case of appending to a file
where the last offset in the file was allocated to a fragment. By
looking at "tsdbfs listda" and appending to a file I could see that the
last DA would change (presumably to point to the DA of the start of a
contiguous subblock) once the amount of data appended caused the file
size to exceed the space available in the trailing subblocks.
-Aaron
On 11/5/17 7:57 PM, Frank Schmuck wrote:
> In GPFS blocks within a file are never fragmented. For example, if you
> have a file of size 7.3 MB and your file system block size is 1MB, then
> this file will be made up of 7 full blocks and one fragment of size 320k
> (10 subblocks). Each of the 7 full blocks will be contiguous on a singe
> diks (LUN) behind a single NSD server. The fragment that makes up the
> last part of the file will also be contiguous on a single disk, just
> shorter than a full block.
>
> Frank Schmuck
> IBM Almaden Research Center
>
>
>
> ----- Original message -----
> From: Aaron Knister <aaron.s.knister at nasa.gov>
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> To: <gpfsug-discuss at spectrumscale.org>
> Cc:
> Subject: Re: [gpfsug-discuss] file layout API + file fragmentation
> Date: Sun, Nov 5, 2017 3:39 PM
>
> Thanks Marc, that helps. I can't easily use tsdbfs for what I'm working
> on since it needs to be run as unprivileged users.
>
> Perhaps I'm not asking the right question. I'm wondering how the file
> layout api behaves if a given "block"-aligned offset in a file is made
> up of sub-blocks/fragments that are not all on the same NSD. The
> assumption based on how I've seen the API used so far is that all
> sub-blocks within a block at a given offset within a file are all on the
> same NSD.
>
> -Aaron
>
> On 11/5/17 6:01 PM, Marc A Kaplan wrote:
> > I googled GPFS_FCNTL_GET_DATABLKDISKIDX
> >
> > and found this discussion:
> >
> >
> Â https://www.ibm.com/developerworks/community/forums/html/topic?id=db48b190-4f2f-4e24-a035-25d3e2b06b2d&ps=50
> >
> > In general, GPFS files ARE deliberately "fragmented" but we don't say
> > that - we say they are "striped" over many disks -- and that is
> > generally a good thing for parallel performance.
> >
> > Also, in GPFS, if the last would-be block of a file is less than a
> > block, then it is stored in a "fragment" of a block. Â
> > So you see we use "fragment" to mean something different than
> other file
> > systems you may know.
> >
> > --marc
> >
> >
> >
> > From: Â Â Â Â Aaron Knister <aaron.s.knister at nasa.gov>
> > To: Â Â Â Â gpfsug main discussion list
> <gpfsug-discuss at spectrumscale.org>
> > Date: Â Â Â Â 11/04/2017 12:22 PM
> > Subject: Â Â Â Â [gpfsug-discuss] file layout API + file
> fragmentation
> > Sent by: Â Â Â Â gpfsug-discuss-bounces at spectrumscale.org
> >
> ------------------------------------------------------------------------
> >
> >
> >
> > I've got a question about the file layout API and how it reacts in the
> > case of fragmented files.
> >
> > I'm using the GPFS_FCNTL_GET_DATABLKDISKIDX structure and have
> some code
> > based on tsGetDataBlk.C. I'm basing the block size based off of what's
> > returned by filemapOut.blockSize but that only seems to return a
> value >
> > 0 when filemapIn.startOffset is 0.
> >
> > In a case where a file were to be made up of a significant number of
> > non-contiguous fragments (which... would be awful in of itself) how
> > would this be reported by the file layout API? Does the interface
> > technically just report the disk location information of the first
> block
> > of the $blockSize range and assume that it's contiguous?
> >
> > Thanks!
> >
> > -Aaron
> >
> > --
> > Aaron Knister
> > NASA Center for Climate Simulation (Code 606.2)
> > Goddard Space Flight Center
> > (301) 286-2776
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=wnR7m6d4urZ_8dM4mkHQjMbFD9xJEeesmJyzt1osCnM&s=-dgGO6O5i1EqWj-8MmzjxJ1Iz2I5gT1aRmtyP44Cvdg&e=
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIF-g&c=jf_iaSHvJObTbx-siA1ZOg&r=ai3ddVzf50ktH78ovGv6NU4O2LZUOWLpiUiggb8lEgA&m=pUdB4fbWLD03ZTAhk9OlpRdIasz628Oa_yG8z8NOjsk&s=kisarJ7IVnyYBx05ZZiGzdwaXnPqNR8UJoywU1OJNRU&e=
> >
>
> --
> Aaron Knister
> NASA Center for Climate Simulation (Code 606.2)
> Goddard Space Flight Center
> (301) 286-2776
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIF-g&c=jf_iaSHvJObTbx-siA1ZOg&r=ai3ddVzf50ktH78ovGv6NU4O2LZUOWLpiUiggb8lEgA&m=pUdB4fbWLD03ZTAhk9OlpRdIasz628Oa_yG8z8NOjsk&s=kisarJ7IVnyYBx05ZZiGzdwaXnPqNR8UJoywU1OJNRU&e=
>
>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
--
Aaron Knister
NASA Center for Climate Simulation (Code 606.2)
Goddard Space Flight Center
(301) 286-2776
More information about the gpfsug-discuss
mailing list