<font size=2 face="sans-serif">Hi Alex, </font><br><br><font size=2 face="sans-serif">i am not quite sure, that you are hitting
the recently fixed performance issue with ganesha with your scenario, as
this was fixing a sendq issue.</font><br><font size=2 face="sans-serif">it would be quite interesting to see
</font><br><font size=2 face="sans-serif">- which ganesha nfs version you are
using. i think the fix went into the ibm028 level. </font><br><font size=2 face="sans-serif">- netstat output during your tests,
are there any findings on sendq values for the nfsd port ?</font><br><font size=2 face="sans-serif"> ( like : netstat -an | grep 2049
| awk '{ print "SendQ: " $3 " Source: " $4 "
Destination: " $5 }' | grep -v "SendQ: 0" )</font><br><br><font size=1 face="Arial"><br>Mit freundlichen Grüßen / Kind regards</font><p><font size=2 face="Arial"><b>Achim Rehor</b></font><p><table width=600 style="border-collapse:collapse;"><tr height=8><td width=600 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><div align=center><hr noshade></div><tr height=8><td width=600 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 face="Arial"> </font><tr height=8><td width=496 colspan=3 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 face="Arial">Software
Technical Support Specialist AIX/ Emea HPC Support</font><td width=103 rowspan=10 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><div align=right><img align=bottom src=cid:_1_DD9ED2ECDD9ECD6C00350744C1258328 style="border:0px solid;"></div><tr height=8><td width=496 colspan=3 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 face="Arial">IBM
Certified Advanced Technical Expert - Power Systems with AIX</font><tr height=8><td width=496 colspan=3 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 face="Arial">TSCC
Software Service, Dept. 7922</font><tr height=8><td width=496 colspan=3 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 face="Arial">Global
Technology Services </font><tr height=8><td width=496 colspan=3 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><div align=center><hr noshade></div><tr height=8><td width=44 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial">Phone:</font><td width=197 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial">+49-7034-274-7862</font><td width=255 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial"> IBM
Deutschland</font><tr height=8><td width=44 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial">E-Mail:</font><td width=197 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial">Achim.Rehor@de.ibm.com</font><td width=255 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial"> Am
Weiher 24</font><tr height=8><td width=44 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial"> </font><td width=197 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial"> </font><td width=255 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial"> 65451
Kelsterbach</font><tr height=8><td width=44 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial"> </font><td width=197 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial"> </font><td width=255 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial"> Germany</font><tr height=8><td width=44 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial"> </font><td width=197 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial"> </font><td width=255 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0 face="Arial"> </font><tr height=8><td width=600 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><div align=center><hr noshade></div><tr height=8><td width=600 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 face="Arial"> </font><tr height=8><td width=600 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#a2a2a2 face="Arial">IBM
Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter <br>Geschäftsführung: Martin Hartmann (Vorsitzender), Norbert Janzen, Stefan
Lutz, Nicole Reimer, Dr. Klaus Seifert, Wolfgang Wendt <br>Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart,
HRB 14562 WEEE-Reg.-Nr. DE 99369940 </font></table><br><p><font size=3> </font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">"Alexander Saupp"
<Alexander.Saupp@de.ibm.com></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug-discuss@spectrumscale.org</font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">15/10/2018 20:20</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">[gpfsug-discuss]
Tuning: single client, single thread, small
files - native Scale vs NFS</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><font size=2>Dear Spectrum Scale mailing list,</font><font size=3><br></font><font size=2><br>I'm part of IBM Lab Services - currently i'm having multiple customers
asking me for optimization of a similar workloads.</font><font size=3><br></font><font size=2><br>The task is to tune a Spectrum Scale system (comprising ESS and CES protocol
nodes) for the following workload: <br>A single Linux NFS client mounts an NFS export, extracts a flat tar archive
with lots of ~5KB files. <br>I'm measuring the speed at which those 5KB files are written (`time tar
xf archive.tar`). </font><font size=3><br></font><font size=2><br>I do understand that Spectrum Scale is not designed for such workload (single
client, single thread, small files, single directory), and that such benchmark
in not appropriate to benmark the system. <br>Yet I find myself explaining the performance for such scenario (git clone..)
quite frequently, as customers insist that optimization of that scenario
would impact individual users as it shows task duration.<br>I want to make sure that I have optimized the system as much as possible
for the given workload, and that I have not overlooked something obvious.</font><font size=3><br><br></font><font size=2><br>When writing to GPFS directly I'm able to write ~1800 files / second in
a test setup. <br>This is roughly the same on the protocol nodes (NSD client), as well as
on the ESS IO nodes (NSD server). <br>When writing to the NFS export on the protocol node itself (to avoid any
network effects) I'm only able to write ~230 files / second.<br>Writing to the NFS export from another node (now including network latency)
gives me ~220 files / second.</font><font size=3><br><br></font><font size=2><br>There seems to be a huge performance degradation by adding NFS-Ganesha
to the software stack alone. I wonder what can be done to minimize the
impact.</font><font size=3><br><br></font><font size=2><br>- Ganesha doesn't seem to support 'async' or 'no_wdelay' options... anything
equivalent available?<br>- Is there and expected advantage of using the network-latency tuned profile,
as opposed to the ESS default throughput-performance profile?<br>- Are there other relevant Kernel params?<br>- Is there an expected advantage of raising the number of threads (NSD
server (nsd*WorkerThreads) / NSD client (workerThreads) / Ganesha (NB_WORKER))
for the given workload (single client, single thread, small files)?<br>- Are there other relevant GPFS params?<br>- Impact of Sync replication, disk latency, etc is understood. <br>- I'm aware that 'the real thing' would be to work with larger files in
a multithreaded manner from multiple nodes - and that this scenario will
scale quite well.<br>I just want to ensure that I'm not missing something obvious over reiterating
that massage to customers.</font><font size=3><br></font><font size=2><br>Any help was greatly appreciated - thanks much in advance!<br>Alexander Saupp<br>IBM Germany</font><font size=3><br><br></font><font size=2><br>Mit freundlichen Grüßen / Kind regards</font><font size=3><br><b><br>Alexander Saupp</b><br></font><font size=3 color=#0060a0><br>IBM Systems, Storage Platform, EMEA Storage Competence Center</font><table width=800 style="border-collapse:collapse;"><tr height=8><td width=800 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><hr><tr height=8><td width=800 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><img align=bottom src=cid:_1_DDA0E804DDA0E41800350744C1258328 width=1 height=1 style="border:0px solid;"><tr height=8><td width=88 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0>Phone:</font><td width=322 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0>+49
7034-643-1512</font><td width=250 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0>IBM
Deutschland GmbH</font><td width=140 rowspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><div align=right><img align=bottom src=cid:_1_DDA1B8CCDDA1B4E000350744C1258328 style="border:0px solid;"></div><tr height=8><td width=88 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0>Mobile:</font><td width=322 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0>+49-172
7251072</font><td width=250 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0>Am
Weiher 24</font><tr height=8><td width=88 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0>Email:</font><td width=322 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0>alexander.saupp@de.ibm.com</font><td width=250 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0>65451
Kelsterbach</font><tr height=8><td width=88 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><img align=bottom src=cid:_1_DDA1A7B8DDA1A3CC00350744C1258328 width=1 height=1 style="border:0px solid;"><td width=322 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><img align=bottom src=cid:_1_DDA1ADCCDDA1A9E000350744C1258328 width=1 height=1 style="border:0px solid;"><td width=250 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#0060a0>Germany</font><tr height=8><td width=800 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><hr><tr height=8><td width=800 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#a2a2a2>IBM
Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter<br>Geschäftsführung: Matthias Hartmann (Vorsitzender), Norbert Janzen, Stefan
Lutz, Nicole Reimer, Dr. Klaus Seifert, Wolfgang Wendt<br>Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart,
HRB 14562 / WEEE-Reg.-Nr. DE 99369940 </font></table><br><font size=3><br></font><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br></font></tt><br><br><BR>