<html><body><p><font size="2">Hi Kennmeth,</font><br><br><font size="2">we also had similar performance numbers in our tests. Native was far quicker than through GPFS. When we learned though that the client tested the performance on the FS at a big blocksize (512k) with small files - we were able to speed it up significantly using a smaller FS blocksize (obviously we had to recreate the FS).</font><br><br><font size="2">So really depends on how you do your tests.</font><p><b><font color="#8F8F8F" face="Arial">Cheers,<br></font></b><br><b><font color="#888888" face="Arial">Marcus Koenig</font></b><font size="2" face="Arial"><br>Lab Services Storage & Power Specialist</font><br><i><font size="2" face="Calibri">IBM Australia & New Zealand Advanced Technical Skills</font></i><font size="2" face="Arial"><br>IBM Systems-Hardware</font><table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0"><tr valign="top"><td width="1095" colspan="3" valign="middle"><hr width="100%" size="2" align="left"></td><td width="639"><img width="1" height="1" src="cid:1__=43BB0B9ADF94D42A8f9e8a93df938690918c43B@" border="0" alt=""></td></tr>
<tr valign="top"><td width="119"><img src="cid:2__=43BB0B9ADF94D42A8f9e8a93df938690918c43B@" width="119" height="119"></td><td width="337"><b><font size="1" color="#466BB0" face="Arial">Mobile:</font></b><font size="1" color="#5F5F5F" face="Arial"> +64 21 67 34 27</font><b><font size="1" color="#466BB0" face="Arial"><br>E-mail:</font></b><font size="1" color="#5F5F5F" face="Arial"> </font><a href="mailto:brendanp@nz1.ibm.com" target="_blank"><u><font size="1" color="#5F5F5F" face="Arial">marcusk@nz1.ibm.com</font></u></a><br>
<p><font size="1" color="#5F5F5F" face="Arial">82 Wyndham Street<br>Auckland, AUK 1010<br>New Zealand</font><div align="center"><br><br><br></div><br></td><td width="639"><div align="center"><img src="cid:3__=43BB0B9ADF94D42A8f9e8a93df938690918c43B@" width="132" height="51" align="bottom"><img src="cid:4__=43BB0B9ADF94D42A8f9e8a93df938690918c43B@" width="104" height="52" align="bottom"></div></td><td width="639"><img width="1" height="1" src="cid:1__=43BB0B9ADF94D42A8f9e8a93df938690918c43B@" border="0" alt=""></td></tr>
<tr valign="top"><td width="119"><img width="1" height="1" src="cid:1__=43BB0B9ADF94D42A8f9e8a93df938690918c43B@" border="0" alt=""></td><td width="337"><img width="1" height="1" src="cid:1__=43BB0B9ADF94D42A8f9e8a93df938690918c43B@" border="0" alt=""></td><td width="639"><img width="1" height="1" src="cid:1__=43BB0B9ADF94D42A8f9e8a93df938690918c43B@" border="0" alt=""></td><td width="639"><img width="1" height="1" src="cid:1__=43BB0B9ADF94D42A8f9e8a93df938690918c43B@" border="0" alt=""></td></tr></table><br><img width="16" height="16" src="cid:5__=43BB0B9ADF94D42A8f9e8a93df938690918c43B@" border="0" alt="Inactive hide details for "Uwe Falke" ---04/21/2017 03:07:48 AM---Hi Kennmeth, is prefetching off or on at your storage backe"><font size="2" color="#424282">"Uwe Falke" ---04/21/2017 03:07:48 AM---Hi Kennmeth, is prefetching off or on at your storage backend?</font><br><br><font size="2" color="#5F5F5F">From: </font><font size="2">"Uwe Falke" <UWEFALKE@de.ibm.com></font><br><font size="2" color="#5F5F5F">To: </font><font size="2">gpfsug main discussion list <gpfsug-discuss@spectrumscale.org></font><br><font size="2" color="#5F5F5F">Date: </font><font size="2">04/21/2017 03:07 AM</font><br><font size="2" color="#5F5F5F">Subject: </font><font size="2">Re: [gpfsug-discuss] bizarre performance behavior</font><br><font size="2" color="#5F5F5F">Sent by: </font><font size="2">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br><br><br><tt><font size="2">Hi Kennmeth, <br><br>is prefetching off or on at your storage backend?<br>Raw sequential is very different from GPFS sequential at the storage <br>device !<br>GPFS does its own prefetching, the storage would never know what sectors <br>sequential read at GPFS level maps to at storage level!<br><br> <br>Mit freundlichen Grüßen / Kind regards<br><br> <br>Dr. Uwe Falke<br> <br>IT Specialist<br>High Performance Computing Services / Integrated Technology Services / <br>Data Center Services<br>-------------------------------------------------------------------------------------------------------------------------------------------<br>IBM Deutschland<br>Rathausstr. 7<br>09111 Chemnitz<br>Phone: +49 371 6978 2165<br>Mobile: +49 175 575 2877<br>E-Mail: uwefalke@de.ibm.com<br>-------------------------------------------------------------------------------------------------------------------------------------------<br>IBM Deutschland Business & Technology Services GmbH / Geschäftsführung: <br>Andreas Hasse, Thorsten Moehring<br>Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, <br>HRB 17122 <br><br><br><br><br>From: Kenneth Waegeman <kenneth.waegeman@ugent.be><br>To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Date: 04/20/2017 04:53 PM<br>Subject: Re: [gpfsug-discuss] bizarre performance behavior<br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br><br><br><br>Hi,<br><br>Having an issue that looks the same as this one: <br>We can do sequential writes to the filesystem at 7,8 GB/s total , which is <br>the expected speed for our current storage <br>backend. While we have even better performance with sequential reads on <br>raw storage LUNS, using GPFS we can only reach 1GB/s in total (each nsd <br>server seems limited by 0,5GB/s) independent of the number of clients <br>(1,2,4,..) or ways we tested (fio,dd). We played with blockdev params, <br>MaxMBps, PrefetchThreads, hyperthreading, c1e/cstates, .. as discussed in <br>this thread, but nothing seems to impact this read performance. <br>Any ideas?<br>Thanks!<br><br>Kenneth<br><br>On 17/02/17 19:29, Jan-Frode Myklebust wrote:<br>I just had a similar experience from a sandisk infiniflash system <br>SAS-attached to s single host. Gpfsperf reported 3,2 Gbyte/s for writes. <br>and 250-300 Mbyte/s on sequential reads!! Random reads were on the order <br>of 2 Gbyte/s.<br><br>After a bit head scratching snd fumbling around I found out that reducing <br>maxMBpS from 10000 to 100 fixed the problem! Digging further I found that <br>reducing prefetchThreads from default=72 to 32 also fixed it, while <br>leaving maxMBpS at 10000. Can now also read at 3,2 GByte/s.<br><br>Could something like this be the problem on your box as well?<br><br><br><br>-jf<br>fre. 17. feb. 2017 kl. 18.13 skrev Aaron Knister <aaron.s.knister@nasa.gov<br>>:<br>Well, I'm somewhat scrounging for hardware. This is in our test<br>environment :) And yep, it's got the 2U gpu-tray in it although even<br>without the riser it has 2 PCIe slots onboard (excluding the on-board<br>dual-port mezz card) so I think it would make a fine NSD server even<br>without the riser.<br><br>-Aaron<br><br>On 2/17/17 11:43 AM, Simon Thompson (Research Computing - IT Services)<br>wrote:<br>> Maybe its related to interrupt handlers somehow? You drive the load up <br>on one socket, you push all the interrupt handling to the other socket <br>where the fabric card is attached?<br>><br>> Dunno ... (Though I am intrigued you use idataplex nodes as NSD servers, <br>I assume its some 2U gpu-tray riser one or something !)<br>><br>> Simon<br>> ________________________________________<br>> From: gpfsug-discuss-bounces@spectrumscale.org [<br>gpfsug-discuss-bounces@spectrumscale.org] on behalf of Aaron Knister [<br>aaron.s.knister@nasa.gov]<br>> Sent: 17 February 2017 15:52<br>> To: gpfsug main discussion list<br>> Subject: [gpfsug-discuss] bizarre performance behavior<br>><br>> This is a good one. I've got an NSD server with 4x 16GB fibre<br>> connections coming in and 1x FDR10 and 1x QDR connection going out to<br>> the clients. I was having a really hard time getting anything resembling<br>> sensible performance out of it (4-5Gb/s writes but maybe 1.2Gb/s for<br>> reads). The back-end is a DDN SFA12K and I *know* it can do better than<br>> that.<br>><br>> I don't remember quite how I figured this out but simply by running<br>> "openssl speed -multi 16" on the nsd server to drive up the load I saw<br>> an almost 4x performance jump which is pretty much goes against every<br>> sysadmin fiber in me (i.e. "drive up the cpu load with unrelated crap to<br>> quadruple your i/o performance").<br>><br>> This feels like some type of C-states frequency scaling shenanigans that<br>> I haven't quite ironed down yet. I booted the box with the following<br>> kernel parameters "intel_idle.max_cstate=0 processor.max_cstate=0" which<br>> didn't seem to make much of a difference. I also tried setting the<br>> frequency governer to userspace and setting the minimum frequency to<br>> 2.6ghz (it's a 2.6ghz cpu). None of that really matters-- I still have<br>> to run something to drive up the CPU load and then performance improves.<br>><br>> I'm wondering if this could be an issue with the C1E state? I'm curious<br>> if anyone has seen anything like this. The node is a dx360 M4<br>> (Sandybridge) with 16 2.6GHz cores and 32GB of RAM.<br>><br>> -Aaron<br>><br>> --<br>> Aaron Knister<br>> NASA Center for Climate Simulation (Code 606.2)<br>> Goddard Space Flight Center<br>> (301) 286-2776<br>> _______________________________________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at spectrumscale.org<br>> </font></tt><tt><font size="2"><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></tt><tt><font size="2"><br>> _______________________________________________<br>> gpfsug-discuss mailing list<br>> gpfsug-discuss at spectrumscale.org<br>> </font></tt><tt><font size="2"><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></tt><tt><font size="2"><br>><br><br>--<br>Aaron Knister<br>NASA Center for Climate Simulation (Code 606.2)<br>Goddard Space Flight Center<br>(301) 286-2776<br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><tt><font size="2"><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></tt><tt><font size="2"><br><br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><tt><font size="2"><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></tt><tt><font size="2"><br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><tt><font size="2"><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></tt><tt><font size="2"><br><br><br><br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><tt><font size="2"><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></tt><tt><font size="2"><br><br></font></tt><br><br><BR>
</body></html>