[gpfsug-discuss] file layout API + file fragmentation

Daniel Kidger daniel.kidger at uk.ibm.com
Mon Nov 6 10:00:39 GMT 2017


Frank,

For clarity in the understanding the underlying mechanism in GPFS, could you describe what happens in the case say of a particular file that is appended to every 24 hours?  
ie. as that file gets to 7MB, it then writes to a new sub-block (1/32 of the next 1MB block). I guess that sub block could be 10th in a a block that already has 9 used. Later on, the file grows to need an 11th subblock and so on. So at what point does this growing file at 8MB occupy all 32 sunblocks of  8 full blocks?

Daniel


 

 
 	
Dr Daniel Kidger 
IBM Technical Sales Specialist
Software Defined Solution Sales

+ 44-(0)7818 522 266 
daniel.kidger at uk.ibm.com

> On 6 Nov 2017, at 00:57, Frank Schmuck <fschmuck at us.ibm.com> wrote:
> 
> In GPFS blocks within a file are never fragmented.  For example, if you have a file of size 7.3 MB and your file system block size is 1MB, then this file will be made up of 7 full blocks and one fragment of size 320k (10 subblocks).  Each of the 7 full blocks will be contiguous on a singe diks (LUN) behind a single NSD server.  The fragment that makes up the last part of the file will also be contiguous on a single disk, just shorter than a full block.
>  
> Frank Schmuck
> IBM Almaden Research Center
>  
>  
> ----- Original message -----
> From: Aaron Knister <aaron.s.knister at nasa.gov>
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> To: <gpfsug-discuss at spectrumscale.org>
> Cc:
> Subject: Re: [gpfsug-discuss] file layout API + file fragmentation
> Date: Sun, Nov 5, 2017 3:39 PM
>  
> Thanks Marc, that helps. I can't easily use tsdbfs for what I'm working
> on since it needs to be run as unprivileged users.
> 
> Perhaps I'm not asking the right question. I'm wondering how the file
> layout api behaves if a given "block"-aligned offset in a file is made
> up of sub-blocks/fragments that are not all on the same NSD. The
> assumption based on how I've seen the API used so far is that all
> sub-blocks within a block at a given offset within a file are all on the
> same NSD.
> 
> -Aaron
> 
> On 11/5/17 6:01 PM, Marc A Kaplan wrote:
> > I googled GPFS_FCNTL_GET_DATABLKDISKIDX
> >
> > and found this discussion:
> >
> > Â https://www.ibm.com/developerworks/community/forums/html/topic?id=db48b190-4f2f-4e24-a035-25d3e2b06b2d&ps=50
> >
> > In general, GPFS files ARE deliberately "fragmented" but we don't say
> > that - we say they are "striped" over many disks -- and that is
> > generally a good thing for parallel performance.
> >
> > Also, in GPFS, if the last would-be block of a file is less than a
> > block, then it is stored in a "fragment" of a block. Â 
> > So you see we use "fragment" to mean something different than other file
> > systems you may know.
> >
> > --marc
> >
> >
> >
> > From: Â  Â  Â  Â Aaron Knister <aaron.s.knister at nasa.gov>
> > To: Â  Â  Â  Â gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> > Date: Â  Â  Â  Â 11/04/2017 12:22 PM
> > Subject: Â  Â  Â  Â [gpfsug-discuss] file layout API + file fragmentation
> > Sent by: Â  Â  Â  Â gpfsug-discuss-bounces at spectrumscale.org
> > ------------------------------------------------------------------------
> >
> >
> >
> > I've got a question about the file layout API and how it reacts in the
> > case of fragmented files.
> >
> > I'm using the GPFS_FCNTL_GET_DATABLKDISKIDX structure and have some code
> > based on tsGetDataBlk.C. I'm basing the block size based off of what's
> > returned by filemapOut.blockSize but that only seems to return a value >
> > 0 when filemapIn.startOffset is 0.
> >
> > In a case where a file were to be made up of a significant number of
> > non-contiguous fragments (which... would be awful in of itself) how
> > would this be reported by the file layout API? Does the interface
> > technically just report the disk location information of the first block
> > of the $blockSize range and assume that it's contiguous?
> >
> > Thanks!
> >
> > -Aaron
> >
> > --
> > Aaron Knister
> > NASA Center for Climate Simulation (Code 606.2)
> > Goddard Space Flight Center
> > (301) 286-2776
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=wnR7m6d4urZ_8dM4mkHQjMbFD9xJEeesmJyzt1osCnM&s=-dgGO6O5i1EqWj-8MmzjxJ1Iz2I5gT1aRmtyP44Cvdg&e=
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIF-g&c=jf_iaSHvJObTbx-siA1ZOg&r=ai3ddVzf50ktH78ovGv6NU4O2LZUOWLpiUiggb8lEgA&m=pUdB4fbWLD03ZTAhk9OlpRdIasz628Oa_yG8z8NOjsk&s=kisarJ7IVnyYBx05ZZiGzdwaXnPqNR8UJoywU1OJNRU&e=
> >
> 
> --
> Aaron Knister
> NASA Center for Climate Simulation (Code 606.2)
> Goddard Space Flight Center
> (301) 286-2776
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIF-g&c=jf_iaSHvJObTbx-siA1ZOg&r=ai3ddVzf50ktH78ovGv6NU4O2LZUOWLpiUiggb8lEgA&m=pUdB4fbWLD03ZTAhk9OlpRdIasz628Oa_yG8z8NOjsk&s=kisarJ7IVnyYBx05ZZiGzdwaXnPqNR8UJoywU1OJNRU&e=
>  
>  
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HlQDuUjgJx4p54QzcXd0_zTwf4Cr2t3NINalNhLTA2E&m=WH1GLDCza1Rvd9bzdVYoz2Pdzs7l90XNnhUb40FYCqQ&s=LOkUY79m5Ow2FeKqfCqc31cfXZVmqYlvBuQRPirGOFU&e=
> 
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20171106/725590bd/attachment-0002.htm>


More information about the gpfsug-discuss mailing list