[gpfsug-discuss] mmfsd write behavior
Aaron Knister
aaron.s.knister at nasa.gov
Mon Oct 9 21:46:59 BST 2017
Hi Sven,
Just wondering if you've had any additional thoughts/conversations about
this.
-Aaron
On 9/8/17 5:21 PM, Sven Oehme wrote:
> Hi,
>
> the code assumption is that the underlying device has no volatile write
> cache, i was absolute sure we have that somewhere in the FAQ, but i
> couldn't find it, so i will talk to somebody to correct this.
> if i understand
> https://www.kernel.org/doc/Documentation/block/writeback_cache_control.txt correct
> one could enforce this by setting REQ_FUA, but thats not explicitly set
> today, at least i can't see it. i will discuss this with one of our devs
> who owns this code and come back.
>
> sven
>
>
> On Thu, Sep 7, 2017 at 8:05 PM Aaron Knister <aaron.s.knister at nasa.gov
> <mailto:aaron.s.knister at nasa.gov>> wrote:
>
> Thanks Sven. I didn't think GPFS itself was caching anything on that
> layer, but it's my understanding that O_DIRECT isn't sufficient to force
> I/O to be flushed (e.g. the device itself might have a volatile caching
> layer). Take someone using ZFS zvol's as NSDs. I can write() all day log
> to that zvol (even with O_DIRECT) but there is absolutely no guarantee
> those writes have been committed to stable storage and aren't just
> sitting in RAM until an fsync() occurs (or some other bio function that
> causes a flush). I also don't believe writing to a SATA drive with
> O_DIRECT will force cache flushes of the drive's writeback cache..
> although I just tested that one and it seems to actually trigger a scsi
> cache sync. Interesting.
>
> -Aaron
>
> On 9/7/17 10:55 PM, Sven Oehme wrote:
> > I am not sure what exactly you are looking for but all
> blockdevices are
> > opened with O_DIRECT , we never cache anything on this layer .
> >
> >
> > On Thu, Sep 7, 2017, 7:11 PM Aaron Knister
> <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov>
> > <mailto:aaron.s.knister at nasa.gov
> <mailto:aaron.s.knister at nasa.gov>>> wrote:
> >
> > Hi Everyone,
> >
> > This is something that's come up in the past and has recently
> resurfaced
> > with a project I've been working on, and that is-- it seems
> to me as
> > though mmfsd never attempts to flush the cache of the block
> devices its
> > writing to (looking at blktrace output seems to confirm
> this). Is this
> > actually the case? I've looked at the gpl headers for linux
> and I don't
> > see any sign of blkdev_fsync, blkdev_issue_flush, WRITE_FLUSH, or
> > REQ_FLUSH. I'm sure there's other ways to trigger this
> behavior that
> > GPFS may very well be using that I've missed. That's why I'm
> asking :)
> >
> > I figure with FPO being pushed as an HDFS replacement using
> commodity
> > drives this feature has *got* to be in the code somewhere.
> >
> > -Aaron
> >
> > --
> > Aaron Knister
> > NASA Center for Climate Simulation (Code 606.2)
> > Goddard Space Flight Center
> > (301) 286-2776 <tel:(301)%20286-2776>
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org
> <http://spectrumscale.org> <http://spectrumscale.org>
> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> >
> >
> >
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> >
>
> --
> Aaron Knister
> NASA Center for Climate Simulation (Code 606.2)
> Goddard Space Flight Center
> (301) 286-2776 <tel:(301)%20286-2776>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
--
Aaron Knister
NASA Center for Climate Simulation (Code 606.2)
Goddard Space Flight Center
(301) 286-2776
More information about the gpfsug-discuss
mailing list