<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
just for completeness, let's use this thread to document an undocumented parameter, which should not be used any more</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<b>disableDIO=yes </b>IS NOT disabling DIO 😉</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The parameter is a bit misleading by its name. DIRECT IO is a hint from an application to bypass cache. However mostly it is expected by application programmers to have the IO safely on disk, when it gets ack'd</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
To make this behavior absolutely sure, there is the O_SYNC flag for writes to make that happen. However, is pretty common , just to expect DIRECT is similar seen as a synonym more details here :<a href="https://man7.org/linux/man-pages/man2/open.2.html" id="OWA97b5b366-a3ce-3736-7902-74ad7c9201d7" class="OWAAutoLink" data-loopstyle="linkonly">https://man7.org/linux/man-pages/man2/open.2.html</a></div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
However, GPFS will handle O_DIRECT similar to follow the expectation of programmers.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
So in GPFS, for directIO (O_CIRECT) (by passing caches) we can benefit and use a so called optimized IO path .. which takes advantage of LINUX AIO...</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
however, there are multiple situation, where you can NOT write directly.. e.g. if the IO is not aligned or you want to append a file, when there so no block (no disk address yet allocated to that file) etc...Then you can't process the IO in this directIO ..
optimized path</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
[[some other aspects:<br>
for data, which gets accessed w/o caching.. is that you disable prefetching 😉 <br>
and<br>
more relevant here: you need other (more efficient tokens) than for buffered writes]]</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Lets say, an application appends a file (similar to create a file and write, ) Usually direct IOs are rather small. As long you write small IOs into an existing block, direct IO is fine - but then - if you fill the last block fully with data, you need to allocate
a new block. You can't write directly DIRECT_IO .. to no-where 😉 So we need to allocate a new block (as any other file system as well) .</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
This means for GPFS, that we need to leave the optimized (direct)IO path and allocate a new block by allocating according buffers first</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
When this is done, we finally sync the data from the "direct"-IO before ack'in the IO to follow the expected symantec of the application programmer, which obviously used the O_DIRECT . </div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
For some workloads , which mostly create/write and append files, this is happen on each block .. so frequently... causing GPFS to change going in and out the optimized IO path, which causes also changing tokens</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
To avoid that, a very long time ago -<b>disableDIO </b>- was introduced as a quick and efficient work around.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
in the meantime, there is an heuristic in our code, that automatically detects such cases and should
<b>NOT use</b> this parameter <b>disableDIO</b> any more</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Since GPFS 5.0.x we introduced <b> dioSmallSeqWriteBatching=yes . PLEASE use this parameter.</b> By default, the optimization kicks in when we see three AIO/DIO writes that are no larger 64k bytes each and no more than than one write length appart.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
If you know, that your application does larger DIO writes..let us know, open a SF ticket, there are further options.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature">
<div style="margin-top: 10px;"></div>
</div>
<div id="appendonsend"></div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
BACK to the origin question 🙂you may consider</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
--preallocate blocks to the file (s)</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
--double check "active snapshots" (copy on write for DIRECT IO is very expensive)</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
--adjust your block size / RAID config to lower write amplification</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
--check network rtt for token traffic !!!</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
--try to avoid HDD based backend, as they # IOPS is very limited</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
last but not least - talk to your application programmers ... 😉 if they really need, what they programmed 😉</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<hr style="display: inline-block; width: 98%;">
<div dir="ltr" id="divRplyFwdMsg"><span style="font-family: Calibri, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><b>Von:</b> gpfsug-discuss <gpfsug-discuss-bounces@gpfsug.org> im Auftrag von Uwe Falke <uwe.falke@kit.edu><br>
<b>Gesendet:</b> Dienstag, 12. März 2024 12:21<br>
<b>An:</b> gpfsug-discuss@gpfsug.org <gpfsug-discuss@gpfsug.org><br>
<b>Betreff:</b> [EXTERNAL] Re: [gpfsug-discuss] Slow performance on writes when using direct io</span>
<div> </div>
</div>
<p>Just thinking: an application should do direct IO for a good reason, and only then. "Forcing DIO" is probably not the right thing to do - rather check why an app does DIO and either change the app's behaviour if reasonable are maybe use a special pool for
it using mirrored SSDs or so.</p>
<p>BTW, the ESS have some nice mechanism to do small IOs (also direct ones I suppose) quickly by buffering them on flash/NVRAM (where the data is considered persistently stored, hence the IO requests are completed quickly).</p>
<p>Uwe</p>
<p><br>
</p>
<div>On 12.03.24 11:59, Peter Hruška wrote:</div>
<div style="margin-right: 40px; margin-left: 40px;">Hello,</div>
<div style="margin-right: 40px; margin-left: 40px;"><br>
</div>
<div style="margin-right: 40px; margin-left: 40px;">The direct writes are problematic on both writes and rewrites. Rewrites alone are another issue we have noticed.</div>
<div style="margin-right: 40px; margin-left: 40px;">Since indirect (direct=0) workloads are fine, it seems that the easiest solution could be to force indirect IO operations for all workloads. However we didn't find such possibility.</div>
<div style="margin-right: 40px; margin-left: 40px;"><br>
</div>
<pre style="margin: 0px 40px;"><div>--
</div></pre>
<div style="margin-right: 40px; margin-left: 40px; font-family: verdana; font-size: 9pt; color: rgb(89, 89, 89);">
S přáním pěkného dne / Best regards</div>
<div style="margin-right: 40px; margin-left: 40px;"><br>
</div>
<div style="margin-right: 40px; margin-left: 40px; font-family: verdana; font-size: 11pt; color: rgb(220, 38, 30);">
<b>Mgr. Peter Hruška</b></div>
<div style="margin-right: 40px; margin-left: 40px; font-family: verdana; font-size: 9pt;">
<span style="color: rgb(89, 89, 89);">IT specialista</span><br>
<br>
</div>
<div style="margin-right: 40px; margin-left: 40px; font-family: Verdana, sans-serif; font-size: 9pt;">
<span style="color: rgb(89, 89, 89);"><b>M Computers s.r.o.</b><br>
</span><span style="color: rgb(220, 38, 30);">Úlehlova 3100/10, 628 00 Brno-Líšeň (</span><span style="color: rgb(5, 99, 193);"><a href="https://mapy.cz/s/gafufehufe" id="OWAf831250e-f775-772b-a6b4-69471ee93b14" class="OWAAutoLink" title="Click to open https://mapy.cz/s/gafufehufe" style="color: rgb(5, 99, 193);" data-auth="NotApplicable" data-loopstyle="linkonly">mapa</a></span><span style="color: rgb(220, 38, 30);">)</span><br>
<span style="color: rgb(89, 89, 89);">T:+420 515 538 136</span><br>
<span style="color: rgb(89, 89, 89);">E: </span><span style="color: rgb(5, 99, 193);"><a href="mailto:peter.hruska@mcomputers.cz" id="OWAd7b85a7f-8a4a-f70f-0125-715acb0db60c" class="OWAAutoLink" title="Click to mail peter.hruska@mcomputers.cz" style="color: rgb(5, 99, 193);" data-loopstyle="linkonly">peter.hruska@mcomputers.cz</a></span><span style="color: rgb(89, 89, 89);"><br>
</span><br>
<span style="color: rgb(5, 99, 193);"><a href="http://www.mcomputers.cz/" id="OWA4945efda-8411-f9a7-3991-36a8dde1f6f0" class="OWAAutoLink" title="Click to open http://www.mcomputers.cz/" style="color: rgb(5, 99, 193);" data-auth="NotApplicable" data-loopstyle="linkonly">www.mcomputers.cz</a></span></div>
<div style="margin-right: 40px; margin-left: 40px; font-family: Verdana, sans-serif; font-size: 9pt;">
<a href="http://www.lenovoshop.cz/" id="OWA87b78c74-9066-449c-422c-21e0e93ceb76" class="OWAAutoLink" title="Click to open http://www.lenovoshop.cz/" data-auth="NotApplicable" data-loopstyle="linkonly">www.lenovoshop.cz</a></div>
<div style="margin-right: 40px; margin-left: 40px;"><img style="width: 457px; height: 106px;" width="457" height="106" data-outlook-trace="F:1|T:1" src="cid:04ece728-7e44-4ab2-b1c6-1928bb3c29ee"></div>
<div style="text-align: left; text-indent: 0px; margin-right: 40px; margin-left: 40px; font-family: verdana; font-size: 9pt; color: rgb(89, 89, 89);">
<br>
</div>
<div style="text-align: left; text-indent: 0px; margin-right: 40px; margin-left: 40px; font-family: Cantarell; font-size: 14.6667px; color: rgb(0, 0, 0);">
<br>
</div>
<div style="margin-right: 40px; margin-left: 40px;"><br>
</div>
<div style="margin-right: 40px; margin-left: 40px;">On Tue, 2024-03-12 at 09:59 +0100, Zdenek Salvet wrote:</div>
<blockquote style="margin-top: 0px; margin-bottom: 0px; padding-left: 1ex; border-left: 2px solid rgb(114, 159, 207);">
<div>EXTERNÍ ODESÍLATEL</div>
<div><br>
</div>
<div><br>
</div>
<div>On Mon, Mar 11, 2024 at 01:21:32PM +0000, Peter Hruška wrote:</div>
<blockquote style="margin: 0px 0px 0px 0.8ex; padding-left: 1ex; border-left: 2px solid rgb(114, 159, 207);">
<div>We encountered a problem with performance of writes on GPFS when the application uses direct io access. To simulate the issue it is enough to run fio with option direct=1. The performance drop is quite dramatic - 250 MiB/s vs. 2955 MiB/s. We've tried to
instruct GPFS to ignore direct IO by using "disableDIO=yes". The directive didn't have any effect. Is there any possibility how to achieve that GPFS would ignore direct IO requests and use caching for everything?</div>
</blockquote>
<div><br>
</div>
<div>Hello,</div>
<div>did you use pre-allocated file(s) (was it re-write) ?</div>
<div>libaio traffic is not really asynchronous with respect to necessary metadata</div>
<div>operations (allocating new space and writing allocation structures to disk)</div>
<div>in most Linux filesystems and I guess this case is not heavily optimized</div>
<div>in GPFS either (dioSmallSeqWriteBatching feature may help a little but</div>
<div>it targets different scenario I think).</div>
<div><br>
</div>
<div>Best regards,</div>
<div>Zdenek Salvet <a href="mailto:salvet@ics.muni.cz" id="OWA8f207b58-8979-ad2b-ddd4-589ba7f010d3" class="x_moz-txt-link-freetext OWAAutoLink" data-loopstyle="linkonly">
salvet@ics.muni.cz</a></div>
<div>Institute of Computer Science of Masaryk University, Brno, Czech Republic</div>
<div>and CESNET, z.s.p.o., Prague, Czech Republic</div>
<div>Phone: ++420-549 49 6534 Fax: ++420-541 212 747</div>
<div>----------------------------------------------------------------------------</div>
<div> Teamwork is essential -- it allows you to blame someone else.</div>
<div><br>
</div>
<div><br>
</div>
<div>_______________________________________________</div>
<div>gpfsug-discuss mailing list</div>
<div>gpfsug-discuss at gpfsug.org</div>
<div><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org" id="OWAe1e10ba4-d630-269d-db37-2ae67b79d82c" class="x_moz-txt-link-freetext OWAAutoLink" data-auth="NotApplicable" data-loopstyle="linkonly">http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org</a></div>
</blockquote>
<div style="margin-right: 40px; margin-left: 40px;"><br>
</div>
<fieldset class="x_moz-mime-attachment-header"></fieldset>
<pre style="margin: 0px 40px;"><div>_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org" id="OWA3128930f-12bf-2931-d2f7-b5e459f84fa1" class="x_moz-txt-link-freetext OWAAutoLink" data-auth="NotApplicable" data-loopstyle="linkonly">http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org</a>
</div></pre>
<pre style="margin: 0px;"><div>--
Karlsruhe Institute of Technology (KIT)
Scientific Computing Centre (SCC)
Scientific Data Management (SDM)
Uwe Falke
Hermann-von-Helmholtz-Platz 1, Building 442, Room 187
D-76344 Eggenstein-Leopoldshafen
Tel: +49 721 608 28024
Email: <a href="mailto:uwe.falke@kit.edu" id="OWA4e2fe131-bb35-e5a7-2ace-6f34f0fafc76" class="x_moz-txt-link-abbreviated OWAAutoLink" data-loopstyle="linkonly">uwe.falke@kit.edu</a>
<a href="http://www.scc.kit.edu" id="OWAfc410978-99a4-149d-7b1d-d7e997c5c94d" class="x_moz-txt-link-abbreviated OWAAutoLink" data-auth="NotApplicable" data-loopstyle="linkonly">www.scc.kit.edu</a>
Registered office:
Kaiserstraße 12, 76131 Karlsruhe, Germany
KIT – The Research University in the Helmholtz Association
</div></pre>
</body>
</html>