<font size=2 face="sans-serif">All NFS IO requires syncing.</font><br><font size=2 face="sans-serif">The client does send explicit fsync
(commit). If the NFS server does not sync, a server fail will cause data
loss!</font><br><font size=2 face="sans-serif">(for small files <1M it really does
not matter if it is sync on write or sync on close/explicit commit)</font><br><br><br><font size=2 face="sans-serif">while that may be ok for a "git
pull" or similar, in general it violates the NFS spec.</font><br><br><font size=2 face="sans-serif">The client can decide to cache, and
usually NFSv4 does less caching (for better consistency)</font><br><br><font size=2 face="sans-serif">So the observed factor 100 is realistic.
</font><br><font size=2 face="sans-serif">Latencies will make matters worse, so
the FS should be tuned for very small random IO (small blocksize - small
subblock-size will not help)</font><br><br><font size=2 face="sans-serif">If you were to put the Linux kernel
NFS server into the picture, it will behave very much the same - although
Ganesha could be a bit more efficient (by some percent - certainly less
then 200%). </font><br><br><br><font size=2 face="sans-serif">But hey - this is a GPFS cluster not
some NAS box.</font><br><font size=2 face="sans-serif">Run "git pull" on tthe GPFS
client. Enjoy the 1800 files/sec (or more). Modify the files on your XY
client mounting over NFS. Use a wrapper script to automatically have
your AD or LDAP user id SSH into the cluster to perform it.</font><br><br><font size=2 face="sans-serif">Michael</font><br><table width=1024 style="border-collapse:collapse;"><tr valign=top height=8><td width=1024 colspan=4 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=2 face="Tahoma">Mit
freundlichen Grüßen / with best regards <br> </font><tr valign=top height=8><td width=289 rowspan=2 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><a href="http://w3.ibm.com/bluepages/simpleSearch.wss?searchBy=Internet+address&location=All+locations&searchFor=DIEDERIC@de.ibm.com"><font size=2 color=#004080 face="Tahoma"><b>Michael
Diederich</b></font></a><font size=2 color=#5f5f5f face="Tahoma"><br>IBM Systems Group <br>Spectrum Scale</font><br><font size=2 color=#5f5f5f face="Tahoma">Software Development</font><td width=376 colspan=2 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=2 color=#4f4f4f face="Tahoma"><b>Contact
Information</b></font><td width=358 rowspan=2 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#5f5f5f face="Tahoma">IBM
Deutschland Research & Development GmbH<br>Vorsitzender des Aufsichtsrats: Martina Koederitz<br>Geschäftsführung: Dirk Wittkopp <br>Sitz der Gesellschaft: Böblingen <br>Registergericht: Amtsgericht Stuttgart, HRB 243294 <br></font><tr valign=top height=8><td width=79 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=2 color=#5f5f5f face="Tahoma">mail:<br>fon:<br>address:</font><td width=297 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><a href=mailto:DIEDERIC@de.ibm.com><font size=2 color=#004080 face="Tahoma">michael.diederich@de.ibm.com</font></a><font size=2 color=#5f5f5f face="Tahoma"><br>+49-7034-274-4062<br>Am Weiher 24<br>D-65451 Kelsterbach</font><tr valign=top height=8><td width=1024 colspan=4 bgcolor=#efefef style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><div align=center></div></table><br><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">valdis.kletnieks@vt.edu</font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug main discussion
list <gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Cc:
</font><font size=1 face="sans-serif">Silvana De Gyves <silvanad@mx1.ibm.com>,
Jay Vaddi <jayvaddi@us.ibm.com>, Michael Diederich <diederich@de.ibm.com></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">10/16/2018 02:42 AM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
Tuning: single client, single thread, small files - native Scale vs NFS</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">Valdis Kletnieks
<valdis@vt.edu></font><br><hr noshade><br><br><br><tt><font size=2>On Mon, 15 Oct 2018 18:34:50 -0400, "Kumaran
Rajaram" said:<br><br>> 1. >>When writing to GPFS directly I'm able to write ~1800 files
/ second in a test setup. <br>> >>This is roughly the same on the protocol nodes (NSD client),
as well as <br>> on the ESS IO nodes (NSD server). <br>><br>> 2. >> When writing to the NFS export on the protocol node itself
(to avoid <br>> any network effects) I'm only able to write ~230 files / second.<br><br>> IMHO #2, writing to the NFS export on the protocol node should be
same as #1.<br>> Protocol node is also a NSD client and when you write from a protocol
node, it<br>> will use the NSD protocol to write to the ESS IO nodes. In #1, you
cite seeing<br>> ~1800 files from protocol node and in #2 you cite seeing ~230 file/sec
which<br>> seem to contradict each other. <br><br>I think he means this:<br><br>1) ssh nsd_server<br>2) cd /gpfs/filesystem/testarea<br>3) (whomp out 1800 files/sec)<br>4) mount -t nfs localhost:/gpfs/filesystem/testarea /mnt/test<br>5) cd /mnt/test<br>6) Watch the same test struggle to hit 230.<br><br>Indicating the issue is going from NFS to GPFS<br><br>(For what it's worth, we've had issues with Ganesha as well...)<br>[attachment "att4z9wh.dat" deleted by Michael Diederich/Germany/IBM]
</font></tt><br><br><BR>