<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
tt
{mso-style-priority:99;
font-family:"Courier New";}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial",sans-serif;color:black;background:white'>while most said here is correct, it can’t explain the performance of 200 files /sec and I couldn’t resist jumping in here :-D<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial",sans-serif;color:black;background:white'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial",sans-serif;color:black;background:white'>lets assume for a second each operation is synchronous and its done by just 1 thread. 200 files / sec means 5 ms on average per file write. Lets be generous and say the network layer is 100 usec per roud-trip network hop (including code processing on protocol node or client) and for visualization lets assume the setup looks like this : <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial",sans-serif;color:black;background:white'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial",sans-serif;color:black;background:white'>ESS Node ---ethernet--- Protocol Node –ethernet--- client Node . <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial",sans-serif;color:black;background:white'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial",sans-serif;color:black;background:white'>lets say the ESS write cache can absorb small io at a fixed cost of 300 usec if the heads are ethernet connected and not using IB (then it would be more in the 250 usec range). That’s 300 +100(net1) +100(net2) usec or 500 usec in total. So you are a factor 10 off from your number. So lets just assume a create + write is more than just 1 roundtrip worth or synchronization, lets say it needs to do 2 full roundtrips synchronously one for the create and one for the stable write that’s 1 ms, still 5x off of your 5 ms. <o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>So either there is a bug in the NFS Server, the NFS client or the storage is not behaving properly. To verify this, the best would be to run the following test :<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Create a file on the ESS node itself in the shared filesystem like : <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>/usr/lpp/mmfs/samples/perf/gpfsperf create seq -nongpfs -r 4k -n 1m -th 1 -dio /sharedfs/test<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Now run the following command on one of the ESS nodes, then the protocol node and last the nfs client : <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>/usr/lpp/mmfs/samples/perf/gpfsperf write seq -nongpfs -r 4k -n 1m -th 1 -dio /sharedfs/test<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>This will create 256 stable 4k write i/os to the storage system, I picked the number just to get a statistical relevant number of i/os you can change 1m to 2m or 4m, just don’t make it too high or you might get variations due to de-staging or other side effects happening on the storage system, which you don’t care at this point you want to see the round trip time on each layer. <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>The gpfsperf command will spit out a line like :<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Data rate was XYZ Kbytes/sec, Op Rate was XYZ Ops/sec, <b><u>Avg Latency was 0.266 milliseconds</u></b>, thread utilization 1.000, bytesTransferred 1048576<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>The only number here that matters is the average latency number , write it down.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>What I would expect to get back is something like : <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>On ESS Node – 300 usec average i/o <o:p></o:p></p><p class=MsoNormal>On PN – 400 usec average i/o <o:p></o:p></p><p class=MsoNormal>On Client – 500 usec average i/o <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>If you get anything higher than the numbers above something fundamental is bad (in fact on fast system you may see from client no more than 200-300 usec response time) and it will be in the layer in between or below of where you test. <o:p></o:p></p><p class=MsoNormal>If all the numbers are somewhere in line with my numbers above, it clearly points to a problem in NFS itself and the way it communicates with GPFS. Marc, myself and others have debugged numerous issues in this space in the past last one was fixed beginning of this year and ended up in some Scale 5.0.1.X release. To debug this is very hard and most of the time only possible with GPFS source code access which I no longer have. <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>You would start with something like strace -Ttt -f -o tar-debug.out tar -xvf ….. and check what exact system calls are made to nfs client and how long each takes. You would then run a similar strace on the NFS server to see how many individual system calls will be made to GPFS and how long each takes. This will allow you to narrow down where the issue really is. But I suggest to start with the simpler test above as this might already point to a much simpler problem. <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Btw. I will be also be speaking at the UG Meeting at SC18 in Dallas, in case somebody wants to catch up …<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Sven<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:12.0pt;color:black'>From: </span></b><span style='font-size:12.0pt;color:black'><gpfsug-discuss-bounces@spectrumscale.org> on behalf of Jan-Frode Myklebust <janfrode@tanso.net><br><b>Reply-To: </b>gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br><b>Date: </b>Wednesday, October 17, 2018 at 6:50 AM<br><b>To: </b>gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br><b>Subject: </b>Re: [gpfsug-discuss] Preliminary conclusion: single client, single thread, small files - native Scale vs NFS<o:p></o:p></span></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Also beware there are 2 different linux NFS "async" settings. A client side setting (mount -o async), which still cases sync on file close() -- and a server (knfs) side setting (/etc/exports) that violates NFS protocol and returns requests before data has hit stable storage.<o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal> -jf<o:p></o:p></p></div></div><p class=MsoNormal><o:p> </o:p></p><div><div><p class=MsoNormal>On Wed, Oct 17, 2018 at 9:41 AM Tomer Perry <<a href="mailto:TOMP@il.ibm.com">TOMP@il.ibm.com</a>> wrote:<o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial",sans-serif'>Hi,</span><br><br><span style='font-size:10.0pt;font-family:"Arial",sans-serif'>Without going into to much details, AFAIR, Ontap integrate NVRAM into the NFS write cache ( as it was developed as a NAS product).</span><br><span style='font-size:10.0pt;font-family:"Arial",sans-serif'>Ontap is using the STABLE bit which kind of tell the client "hey, I have no write cache at all, everything is written to stable storage - thus, don't bother with commits ( sync) commands - they are meaningless".</span><br><br><span style='font-size:10.0pt;font-family:"Arial",sans-serif'><br>Regards,<br><br>Tomer Perry<br>Scalable I/O Development (Spectrum Scale)<br>email: <a href="mailto:tomp@il.ibm.com" target="_blank">tomp@il.ibm.com</a><br>1 Azrieli Center, Tel Aviv 67021, Israel<br>Global Tel: +1 720 3422758<br>Israel Tel: +972 3 9188625<br>Mobile: +972 52 2554625<br></span><br><br><br><br><span style='font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F'>From: </span><span style='font-size:7.5pt;font-family:"Arial",sans-serif'>"Keigo Matsubara" <<a href="mailto:MKEIGO@jp.ibm.com" target="_blank">MKEIGO@jp.ibm.com</a>></span><br><span style='font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F'>To: </span><span style='font-size:7.5pt;font-family:"Arial",sans-serif'>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.org</a>></span><br><span style='font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F'>Date: </span><span style='font-size:7.5pt;font-family:"Arial",sans-serif'>17/10/2018 16:35</span><br><span style='font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F'>Subject: </span><span style='font-size:7.5pt;font-family:"Arial",sans-serif'>Re: [gpfsug-discuss] Preliminary conclusion: single client, single thread, small files - native Scale vs NFS</span><br><span style='font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F'>Sent by: </span><span style='font-size:7.5pt;font-family:"Arial",sans-serif'><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a></span><o:p></o:p></p><div class=MsoNormal align=center style='text-align:center'><hr size=0 width="100%" noshade style='color:#A0A0A0' align=center></div><p class=MsoNormal><br><br><br><span style='font-size:10.0pt'>I also wonder how many products actually exploit NFS async mode to improve I/O performance by sacrificing the file system consistency risk:</span><span style='font-size:12.0pt'><br></span><span style='font-size:10.0pt;font-family:"Courier New"'><br><tt><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a> wrote on 2018/10/17 22:26:52:</tt><br><tt>> Using this option usually improves performance, but at</tt><br><tt>> the cost that an unclean server restart (i.e. a crash) can cause </tt><br><tt>> data to be lost or corrupted."</tt></span><span style='font-size:12.0pt'><br></span><span style='font-size:10.0pt'><br>For instance, NetApp, at the very least FAS 3220 running Data OnTap 8.1.2p4 7-mode which I tested with, would forcibly *promote* async mode to sync mode.<br>Promoting means even if NFS client requests async mount mode, the NFS server ignores and allows only sync mount mode.</span><span style='font-size:12.0pt'><br></span><span style='font-size:10.0pt'><br>Best Regards,<br>---<br>Keigo Matsubara, Storage Solutions Client Technical Specialist, IBM Japan<br>TEL: +81-50-3150-0595, T/L: 6205-0595</span><span style='font-size:12.0pt'><br></span><tt><span style='font-size:10.0pt'>_______________________________________________</span></tt><span style='font-size:10.0pt;font-family:"Courier New"'><br><tt>gpfsug-discuss mailing list</tt><br><tt>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a></tt><br></span><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><tt><span style='font-size:10.0pt'>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></tt></a><span style='font-size:10.0pt;font-family:"Courier New"'><br></span><br><br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><o:p></o:p></p></blockquote></div><p class=MsoNormal>_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss <o:p></o:p></p></div></body></html>