[gpfsug-discuss] mmfsd write behavior

Sven Oehme oehmes at gmail.com
Fri Sep 8 22:21:00 BST 2017


Hi,

the code assumption is that the underlying device has no volatile write
cache, i was absolute sure we have that somewhere in the FAQ, but i
couldn't find it, so i will talk to somebody to correct this.
if i understand
https://www.kernel.org/doc/Documentation/block/writeback_cache_control.txt
correct
one could enforce this by setting REQ_FUA, but thats not explicitly set
today, at least i can't see it. i will discuss this with one of our devs
who owns this code and come back.

sven


On Thu, Sep 7, 2017 at 8:05 PM Aaron Knister <aaron.s.knister at nasa.gov>
wrote:

> Thanks Sven. I didn't think GPFS itself was caching anything on that
> layer, but it's my understanding that O_DIRECT isn't sufficient to force
> I/O to be flushed (e.g. the device itself might have a volatile caching
> layer). Take someone using ZFS zvol's as NSDs. I can write() all day log
> to that zvol (even with O_DIRECT) but there is absolutely no guarantee
> those writes have been committed to stable storage and aren't just
> sitting in RAM until an fsync() occurs (or some other bio function that
> causes a flush). I also don't believe writing to a SATA drive with
> O_DIRECT will force cache flushes of the drive's writeback cache..
> although I just tested that one and it seems to actually trigger a scsi
> cache sync. Interesting.
>
> -Aaron
>
> On 9/7/17 10:55 PM, Sven Oehme wrote:
> > I am not sure what exactly you are looking for but all blockdevices are
> > opened with O_DIRECT , we never cache anything on this layer .
> >
> >
> > On Thu, Sep 7, 2017, 7:11 PM Aaron Knister <aaron.s.knister at nasa.gov
> > <mailto:aaron.s.knister at nasa.gov>> wrote:
> >
> >     Hi Everyone,
> >
> >     This is something that's come up in the past and has recently
> resurfaced
> >     with a project I've been working on, and that is-- it seems to me as
> >     though mmfsd never attempts to flush the cache of the block devices
> its
> >     writing to (looking at blktrace output seems to confirm this). Is
> this
> >     actually the case? I've looked at the gpl headers for linux and I
> don't
> >     see any sign of blkdev_fsync, blkdev_issue_flush, WRITE_FLUSH, or
> >     REQ_FLUSH. I'm sure there's other ways to trigger this behavior that
> >     GPFS may very well be using that I've missed. That's why I'm asking
> :)
> >
> >     I figure with FPO being pushed as an HDFS replacement using commodity
> >     drives this feature has *got* to be in the code somewhere.
> >
> >     -Aaron
> >
> >     --
> >     Aaron Knister
> >     NASA Center for Climate Simulation (Code 606.2)
> >     Goddard Space Flight Center
> >     (301) 286-2776
> >     _______________________________________________
> >     gpfsug-discuss mailing list
> >     gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
> >     http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> >
> >
> >
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org
> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> >
>
> --
> Aaron Knister
> NASA Center for Climate Simulation (Code 606.2)
> Goddard Space Flight Center
> (301) 286-2776
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170908/b2985540/attachment-0002.htm>


More information about the gpfsug-discuss mailing list