<div dir="ltr">Hi,<div><br></div><div>yeah sorry i intended to reply back before my vacation and forgot about it the the vacation flushed it all away :-D</div><div>so right now the assumption in Scale/GPFS is that the underlying storage doesn't have any form of enabled volatile write cache. the problem seems to be that even if we set REQ_FUA some stacks or devices may not have implemented that at all or correctly, so even we would set it there is no guarantee that it will do what you think it does. the benefit of adding the flag at least would allow us to blame everything on the underlying stack/device , but i am not sure that will make somebody happy if bad things happen, therefore the requirement of a non-volatile device will still be required at all times underneath Scale.</div><div>so if you think we should do this, please open a PMR with the details of your test so it can go its regular support path. you can mention me in the PMR as a reference as we already looked at the places the request would have to be added. </div><div dir="ltr"><div><br></div><div>Sven</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Oct 9, 2017 at 1:47 PM Aaron Knister <<a href="mailto:aaron.s.knister@nasa.gov" target="_blank">aaron.s.knister@nasa.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Sven,<br>
<br>
Just wondering if you've had any additional thoughts/conversations about<br>
this.<br>
<br>
-Aaron<br>
<br>
On 9/8/17 5:21 PM, Sven Oehme wrote:<br>
> Hi,<br>
><br>
> the code assumption is that the underlying device has no volatile write<br>
> cache, i was absolute sure we have that somewhere in the FAQ, but i<br>
> couldn't find it, so i will talk to somebody to correct this.<br>
> if i understand<br>
> <a href="https://www.kernel.org/doc/Documentation/block/writeback_cache_control.txt" rel="noreferrer" target="_blank">https://www.kernel.org/doc/Documentation/block/writeback_cache_control.txt</a> correct<br>
> one could enforce this by setting REQ_FUA, but thats not explicitly set<br>
> today, at least i can't see it. i will discuss this with one of our devs<br>
> who owns this code and come back.<br>
><br>
> sven<br>
><br>
><br>
> On Thu, Sep 7, 2017 at 8:05 PM Aaron Knister <<a href="mailto:aaron.s.knister@nasa.gov" target="_blank">aaron.s.knister@nasa.gov</a><br>
> <mailto:<a href="mailto:aaron.s.knister@nasa.gov" target="_blank">aaron.s.knister@nasa.gov</a>>> wrote:<br>
><br>
> Thanks Sven. I didn't think GPFS itself was caching anything on that<br>
> layer, but it's my understanding that O_DIRECT isn't sufficient to force<br>
> I/O to be flushed (e.g. the device itself might have a volatile caching<br>
> layer). Take someone using ZFS zvol's as NSDs. I can write() all day log<br>
> to that zvol (even with O_DIRECT) but there is absolutely no guarantee<br>
> those writes have been committed to stable storage and aren't just<br>
> sitting in RAM until an fsync() occurs (or some other bio function that<br>
> causes a flush). I also don't believe writing to a SATA drive with<br>
> O_DIRECT will force cache flushes of the drive's writeback cache..<br>
> although I just tested that one and it seems to actually trigger a scsi<br>
> cache sync. Interesting.<br>
><br>
> -Aaron<br>
><br>
> On 9/7/17 10:55 PM, Sven Oehme wrote:<br>
> > I am not sure what exactly you are looking for but all<br>
> blockdevices are<br>
> > opened with O_DIRECT , we never cache anything on this layer .<br>
> ><br>
> ><br>
> > On Thu, Sep 7, 2017, 7:11 PM Aaron Knister<br>
> <<a href="mailto:aaron.s.knister@nasa.gov" target="_blank">aaron.s.knister@nasa.gov</a> <mailto:<a href="mailto:aaron.s.knister@nasa.gov" target="_blank">aaron.s.knister@nasa.gov</a>><br>
> > <mailto:<a href="mailto:aaron.s.knister@nasa.gov" target="_blank">aaron.s.knister@nasa.gov</a><br>
> <mailto:<a href="mailto:aaron.s.knister@nasa.gov" target="_blank">aaron.s.knister@nasa.gov</a>>>> wrote:<br>
> ><br>
> > Hi Everyone,<br>
> ><br>
> > This is something that's come up in the past and has recently<br>
> resurfaced<br>
> > with a project I've been working on, and that is-- it seems<br>
> to me as<br>
> > though mmfsd never attempts to flush the cache of the block<br>
> devices its<br>
> > writing to (looking at blktrace output seems to confirm<br>
> this). Is this<br>
> > actually the case? I've looked at the gpl headers for linux<br>
> and I don't<br>
> > see any sign of blkdev_fsync, blkdev_issue_flush, WRITE_FLUSH, or<br>
> > REQ_FLUSH. I'm sure there's other ways to trigger this<br>
> behavior that<br>
> > GPFS may very well be using that I've missed. That's why I'm<br>
> asking :)<br>
> ><br>
> > I figure with FPO being pushed as an HDFS replacement using<br>
> commodity<br>
> > drives this feature has *got* to be in the code somewhere.<br>
> ><br>
> > -Aaron<br>
> ><br>
> > --<br>
> > Aaron Knister<br>
> > NASA Center for Climate Simulation (Code 606.2)<br>
> > Goddard Space Flight Center<br>
> > <a href="tel:(301)%20286-2776" value="+13012862776" target="_blank">(301) 286-2776</a> <tel:(301)%20286-2776><br>
> > _______________________________________________<br>
> > gpfsug-discuss mailing list<br>
> > gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
> <<a href="http://spectrumscale.org" rel="noreferrer" target="_blank">http://spectrumscale.org</a>> <<a href="http://spectrumscale.org" rel="noreferrer" target="_blank">http://spectrumscale.org</a>><br>
> > <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
> ><br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > gpfsug-discuss mailing list<br>
> > gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a> <<a href="http://spectrumscale.org" rel="noreferrer" target="_blank">http://spectrumscale.org</a>><br>
> > <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
> ><br>
><br>
> --<br>
> Aaron Knister<br>
> NASA Center for Climate Simulation (Code 606.2)<br>
> Goddard Space Flight Center<br>
> <a href="tel:(301)%20286-2776" value="+13012862776" target="_blank">(301) 286-2776</a> <tel:(301)%20286-2776><br>
> _______________________________________________<br>
> gpfsug-discuss mailing list<br>
> gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a> <<a href="http://spectrumscale.org" rel="noreferrer" target="_blank">http://spectrumscale.org</a>><br>
> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
><br>
><br>
><br>
> _______________________________________________<br>
> gpfsug-discuss mailing list<br>
> gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
><br>
<br>
--<br>
Aaron Knister<br>
NASA Center for Climate Simulation (Code 606.2)<br>
Goddard Space Flight Center<br>
<a href="tel:(301)%20286-2776" value="+13012862776" target="_blank">(301) 286-2776</a><br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
</blockquote></div></div>