<html><body><p><font size="2">Dear Mailing List readers,</font><br><br><font size="2">I've come to a preliminary conclusion that explains the behavior in an appropriate manner, so I'm trying to summarize my current thinking with this audience.</font><br><br><b><font size="2">Problem statement: </font></b><ul><font size="2">Big performance derivation between native GPFS (fast) and loopback NFS mount on the same node (way slower) for single client, single thread, small files workload.</font></ul><br><br><b><font size="2">Current explanation:</font></b><ul><font size="2">tar seems to use close() on files, not fclose(). That is an application choice and common behavior. The ideas is to allow OS write caching to speed up process run time.</font><br><br><font size="2">When running locally on ext3 / xfs / GPFS / .. that allows async destaging of data down to disk, somewhat compromising data for better performance. </font><br><font size="2">As we're talking about write caching on the same node that the application runs on - a crash is missfortune but in the same failure domain.</font><br><font size="2">E.g. if you run a compile job that includes extraction of a tar and the node crashes you'll have to restart the entire job, anyhow.</font><br><br><font size="2">The NFSv2 spec defined that NFS io's are to be 'sync', probably because the compile job on the nfs client would survive if the NFS Server crashes, so the failure domain would be different</font><br><font size="2"><br>NFSv3 in rfc1813 below acknowledged the performance impact and introduced the 'async' flag for NFS, which would handle IO's similar to local IOs, allowing to destage in the background.</font><br><br><font size="2">Keep in mind - applications, independent if running locally or via NFS can always decided to use the fclose() option, which will ensure that data is destaged to persistent storage right away.<br>But its an applications choice if that's really mandatory or whether performance has higher priority.</font><br><br><font size="2">The linux 'sync' (man sync) tool allows to sync 'dirty' memory cache down to disk - very filesystem independent.</font><br><br></ul><font size="2">-> single client, single thread, small files workload on GPFS can be destaged async, allowing to hide latency and parallelizing disk IOs.<br>-> NFS client IO's are sync, so the second IO can only be started after the first one hit non volatile memory -> much higher latency</font><ul><br><br><font size="2">The Spectrum Scale NFS implementation (based on ganesha) does not support the async mount option, which is a bit of a pitty. There might also be implementation differences compared to kernel-nfs, I did not investigate into that direction.<br></font><br><font size="2">However, the principles of the difference are explained for my by the above behavior. <br></font><br><font size="2">One workaround that I saw working well for multiple customers was to replace the NFS client by a Spectrum Scale nsd client.</font><br><font size="2">That has two advantages, but is certainly not suitable in all cases:</font><ul><font size="2">- Improved speed by efficent NSD protocol and NSD client side write caching</font><br><font size="2">- Write Caching in the same failure domain as the application (on NSD client) which seems to be more reasonable compared to NFS Server side write caching.</font></ul><br></ul><b><font size="2">References:</font></b><br><font size="2"><br>NFS sync vs async</font><br><font size="2"> </font><a href="https://tools.ietf.org/html/rfc1813"><font size="2">https://tools.ietf.org/html/rfc1813</font></a><br><font size="2"> </font><i><font size="2">The write throughput bottleneck caused by the synchronous definition of write in the NFS version 2 protocol has been addressed by adding support so that the NFS server can do unsafe writes.</font></i><br><font size="2"> Unsafe writes are writes which have not been committed to stable storage before the operation returns. This specification defines a method for committing these unsafe writes to stable storage in a reliable way.</font><br><br><br><b><font size="2">sync() vs fsync()</font></b><br><font size="2"> </font><a href="https://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.aix.performance/using_sync_fsync_calls.htm"><font size="2">https://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.aix.performance/using_sync_fsync_calls.htm</font></a><br><font size="2"> - An application program makes an fsync() call for a specified file. This causes all of the pages that contain modified data for that file to be written to disk. The writing is complete when the fsync() call returns to the program.</font><br><br><font size="2"> - An application program makes a sync() call. This causes all of the file pages in memory that contain modified data to be scheduled for writing to disk. The writing is not necessarily complete when the sync() call returns to the program.</font><br><br><font size="2"> - A user can enter the sync command, which in turn issues a sync() call. Again, some of the writes may not be complete when the user is prompted for input (or the next command in a shell script is processed).</font><br><br><br><b><font size="2">close() vs fclose()</font></b><br><font size="2"> A successful close does not guarantee that the data has been successfully saved to disk, as the kernel defers writes. It is not common for a file system to flush the buffers when the stream is closed. If you need to be sure that the data is</font><br><font size="2"> physically stored use fsync(2). (It will depend on the disk hardware at this point.)<br></font><br><br><font size="2">Mit freundlichen Grüßen / Kind regards</font><br><br><b>Alexander Saupp</b><br><br><font color="#0055AA">IBM Systems, Storage Platform, EMEA Storage Competence Center</font><table border="0" cellspacing="0" cellpadding="0"><tr valign="top"><td class="small_font" width="800" colspan="4" valign="middle"><hr width="100%" size="2" align="left" style="color:#696969; "></td></tr>
<tr valign="top"><td class="small_font" width="800" colspan="4" valign="middle"><img width="1" height="1" src="cid:1__=4EBB09BADFAF7CB38f9e8a93df938690918c4EB@" border="0" alt=""></td></tr>
<tr valign="top"><td class="inner" width="88" valign="middle"><font size="1" color="#0055AA">Phone:</font></td><td class="inner" width="322" valign="middle"><font size="1" color="#0055AA">+49 7034-643-1512</font></td><td class="inner" width="250" valign="middle"><font size="1" color="#0055AA"> IBM Deutschland GmbH</font></td><td width="140" rowspan="4" valign="middle"><div align="right"><img src="cid:2__=4EBB09BADFAF7CB38f9e8a93df938690918c4EB@" width="87" height="30" align="bottom"></div></td></tr>
<tr valign="top"><td class="inner" width="88" valign="middle"><font size="1" color="#0055AA">Mobile:</font></td><td class="inner" width="322" valign="middle"><font size="1" color="#0055AA">+49-172 7251072</font></td><td class="inner" width="250" valign="middle"><font size="1" color="#0055AA"> Am Weiher 24</font></td></tr>
<tr valign="top"><td class="inner" width="88" valign="middle"><font size="1" color="#0055AA">Email:</font></td><td class="inner" width="322" valign="middle"><font size="1" color="#0055AA">alexander.saupp@de.ibm.com</font></td><td class="inner" width="250" valign="middle"><font size="1" color="#0055AA"> 65451 Kelsterbach</font></td></tr>
<tr valign="top"><td class="inner" width="88" valign="middle"><img width="1" height="1" src="cid:1__=4EBB09BADFAF7CB38f9e8a93df938690918c4EB@" border="0" alt=""></td><td class="inner" width="322" valign="middle"><img width="1" height="1" src="cid:1__=4EBB09BADFAF7CB38f9e8a93df938690918c4EB@" border="0" alt=""></td><td class="inner" width="250" valign="middle"><font size="1" color="#0055AA"> Germany</font></td></tr>
<tr valign="top"><td class="small_font" width="800" colspan="4" valign="middle"><hr width="100%" size="2" align="left" style="color:#696969; "></td></tr>
<tr valign="top"><td class="foot" width="800" colspan="4" valign="middle"><font size="1" color="#999999">IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter<br>Geschäftsführung: Matthias Hartmann (Vorsitzender), Norbert Janzen, Stefan Lutz, Nicole Reimer, Dr. Klaus Seifert, Wolfgang Wendt<br>Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 </font></td></tr></table><br><BR>
</body></html>