<font size=2 face="sans-serif">Jallo Jan, </font><br><font size=2 face="sans-serif">you can expect to get slightly improved
numbers from the lower response times of the HAWC ... but the loss of performance
comes from the fact, that </font><br><font size=2 face="sans-serif">GPFS or (async kNFS) writes with multiple
parallel threads - in opposite to e.g. tar via GaneshaNFS comes with
single threads fsync on each file.. </font><br><br><font size=2 face="sans-serif">you'll never outperform e.g. 128 (maybe
slower), but, parallel threads (running write-behind) <--->
with one single but fast threads, ....</font><br><br><font size=2 face="sans-serif">so as Alex suggest.. if possible.. take
gpfs client of kNFS for those types of workloads..</font><br><br><br><br><div><br><br><br><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Jan-Frode Myklebust
<janfrode@tanso.net></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug main discussion
list <gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">10/17/2018 02:24 PM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
Preliminary conclusion: single client, single thread, small files - native
Scale vs NFS</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><font size=3>Do you know if the slow throughput is caused by the network/nfs-protocol
layer, or does it help to use faster storage (ssd)? If on storage, have
you considered if HAWC can help?<br><br>I’m thinking about adding an SSD pool as a first tier to hold the active
dataset for a similar setup, but that’s mainly to solve the small file
read workload (i.e. random I/O ).<br><br><br>-jf</font><br><font size=3>ons. 17. okt. 2018 kl. 07:47 skrev Alexander Saupp <</font><a href="mailto:Alexander.Saupp@de.ibm.com"><font size=3 color=blue><u>Alexander.Saupp@de.ibm.com</u></font></a><font size=3>>:</font><br><font size=2>Dear Mailing List readers,</font><font size=3><br></font><font size=2><br>I've come to a preliminary conclusion that explains the behavior in an
appropriate manner, so I'm trying to summarize my current thinking with
this audience.</font><font size=3><br></font><font size=2><b><br>Problem statement: </b></font><br><font size=2>Big performance derivation between native GPFS (fast)
and loopback NFS mount on the same node (way slower) for single client,
single thread, small files workload.</font><br><font size=3><br></font><font size=2><b><br>Current explanation:</b></font><br><font size=2>tar seems to use close() on files, not fclose(). That
is an application choice and common behavior. The ideas is to allow OS
write caching to speed up process run time.</font><font size=3><br></font><font size=2><br>When running locally on ext3 / xfs / GPFS / .. that allows async destaging
of data down to disk, somewhat compromising data for better performance.
<br>As we're talking about write caching on the same node that the application
runs on - a crash is missfortune but in the same failure domain.<br>E.g. if you run a compile job that includes extraction of a tar and the
node crashes you'll have to restart the entire job, anyhow.</font><font size=3><br></font><font size=2><br>The NFSv2 spec defined that NFS io's are to be 'sync', probably because
the compile job on the nfs client would survive if the NFS Server crashes,
so the failure domain would be different<br><br>NFSv3 in rfc1813 below acknowledged the performance impact and introduced
the 'async' flag for NFS, which would handle IO's similar to local IOs,
allowing to destage in the background.</font><font size=3><br></font><font size=2><br>Keep in mind - applications, independent if running locally or via NFS
can always decided to use the fclose() option, which will ensure that data
is destaged to persistent storage right away.<br>But its an applications choice if that's really mandatory or whether performance
has higher priority.</font><font size=3><br></font><font size=2><br>The linux 'sync' (man sync) tool allows to sync 'dirty' memory cache down
to disk - very filesystem independent.</font><font size=3><br></font><br><font size=2>-> single client, single thread, small files workload
on GPFS can be destaged async, allowing to hide latency and parallelizing
disk IOs.<br>-> NFS client IO's are sync, so the second IO can only be started after
the first one hit non volatile memory -> much higher latency</font><br><font size=3><br></font><font size=2><br>The Spectrum Scale NFS implementation (based on ganesha) does not support
the async mount option, which is a bit of a pitty. There might also be
implementation differences compared to kernel-nfs, I did not investigate
into that direction.</font><font size=3><br></font><font size=2><br>However, the principles of the difference are explained for my by the above
behavior. </font><font size=3><br></font><font size=2><br>One workaround that I saw working well for multiple customers was to replace
the NFS client by a Spectrum Scale nsd client.<br>That has two advantages, but is certainly not suitable in all cases:</font><br><font size=2>- Improved speed by efficent NSD protocol and NSD client
side write caching<br>- Write Caching in the same failure domain as the application (on NSD client)
which seems to be more reasonable compared to NFS Server side write caching.</font><br><br><font size=2><b>References:</b><br><br>NFS sync vs async</font><font size=3 color=blue><u><br></u></font><a href="https://tools.ietf.org/html/rfc1813" target="_blank"><font size=2 color=blue><u>https://tools.ietf.org/html/rfc1813</u></font></a><font size=2><i><br>The write throughput bottleneck caused by the synchronous definition of
write in the NFS version 2 protocol has been addressed by adding support
so that the NFS server can do unsafe writes.</i><br>Unsafe writes are writes which have not been committed to stable storage
before the operation returns. This specification defines a method for committing
these unsafe writes to stable storage in a reliable way.</font><font size=3><br><br></font><font size=2><b><br>sync() vs fsync()</b></font><font size=3 color=blue><u><br></u></font><a href="https://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.aix.performance/using_sync_fsync_calls.htm" target="_blank"><font size=2 color=blue><u>https://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.aix.performance/using_sync_fsync_calls.htm</u></font></a><font size=2><br>- An application program makes an fsync() call for a specified file. This
causes all of the pages that contain modified data for that file to be
written to disk. The writing is complete when the fsync() call returns
to the program.</font><font size=3><br></font><font size=2><br>- An application program makes a sync() call. This causes all of the file
pages in memory that contain modified data to be scheduled for writing
to disk. The writing is not necessarily complete when the sync() call returns
to the program.</font><font size=3><br></font><font size=2><br>- A user can enter the sync command, which in turn issues a sync() call.
Again, some of the writes may not be complete when the user is prompted
for input (or the next command in a shell script is processed).</font><font size=3><br><br></font><font size=2><b><br>close() vs fclose()</b><br>A successful close does not guarantee that the data has been successfully
saved to disk, as the kernel defers writes. It is not common for a file
system to flush the buffers when the stream is closed. If you need to be
sure that the data is<br>physically stored use fsync(2). (It will depend on the disk hardware at
this point.)</font><font size=3><br><br></font><font size=2><br>Mit freundlichen Grüßen / Kind regards</font><font size=3><br><b><br>Alexander Saupp</b><br></font><font size=3 color=#0060a0><br>IBM Systems, Storage Platform, EMEA Storage Competence Center</font><table width=800 style="border-collapse:collapse;"><tr height=8><td width=800 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><hr><tr height=8><td width=800 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><tr height=8><td width=88 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0>Phone:</font><td width=322 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0>+49
7034-643-1512</font><td width=250 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0>IBM
Deutschland GmbH</font><td width=140 rowspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><div align=right></div><tr height=8><td width=88 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0>Mobile:</font><td width=322 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0>+49-172
7251072</font><td width=250 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0>Am
Weiher 24</font><tr height=8><td width=88 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0>Email:</font><td width=322 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><a href="mailto:alexander.saupp@de.ibm.com" target="_blank"><font size=1 color=blue><u>alexander.saupp@de.ibm.com</u></font></a><td width=250 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0>65451
Kelsterbach</font><tr height=8><td width=88 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><td width=322 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><td width=250 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0>Germany</font><tr height=8><td width=800 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><hr><tr height=8><td width=800 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#a2a2a2>IBM
Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter<br>Geschäftsführung: Matthias Hartmann (Vorsitzender), Norbert Janzen, Stefan
Lutz, Nicole Reimer, Dr. Klaus Seifert, Wolfgang Wendt<br>Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart,
HRB 14562 / WEEE-Reg.-Nr. DE 99369940 </font></table><br><font size=3><br></font><br><font size=3>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="http://spectrumscale.org" target="_blank"><font size=3 color=blue><u>spectrumscale.org</u></font></a><font size=3 color=blue><u><br></u></font><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><font size=3 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></a><font size=3 color=blue><u>[attachment
"ecblank.gif" deleted by Olaf Weiser/Germany/IBM] [attachment
"19995626.gif" deleted by Olaf Weiser/Germany/IBM] [attachment
"ecblank.gif" deleted by Olaf Weiser/Germany/IBM] </u></font><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br></font></tt><br><br></div><BR>