[gpfsug-discuss] bizarre performance behavior
Kenneth Waegeman
kenneth.waegeman at ugent.be
Fri Apr 21 10:52:58 BST 2017
Hi,
Tried these settings, but sadly I'm not seeing any changes.
Thanks,
Kenneth
On 21/04/17 09:25, Olaf Weiser wrote:
> pls check
> workerThreads (assuming you 're > 4.2.2) start with 128 .. increase
> iteratively
> pagepool at least 8 G
> ignorePrefetchLunCount=yes (1)
>
> then you won't see a difference and GPFS is as fast or even faster ..
>
>
>
> From: "Marcus Koenig1" <marcusk at nz1.ibm.com>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Date: 04/21/2017 03:24 AM
> Subject: Re: [gpfsug-discuss] bizarre performance behavior
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> ------------------------------------------------------------------------
>
>
>
> Hi Kennmeth,
>
> we also had similar performance numbers in our tests. Native was far
> quicker than through GPFS. When we learned though that the client
> tested the performance on the FS at a big blocksize (512k) with small
> files - we were able to speed it up significantly using a smaller FS
> blocksize (obviously we had to recreate the FS).
>
> So really depends on how you do your tests.
>
> *Cheers,*
> *
> Marcus Koenig*
> Lab Services Storage & Power Specialist/
> IBM Australia & New Zealand Advanced Technical Skills/
> IBM Systems-Hardware
> ------------------------------------------------------------------------
>
> *Mobile:*+64 21 67 34 27*
> E-mail:*_marcusk at nz1.ibm.com_ <mailto:brendanp at nz1.ibm.com>
>
> 82 Wyndham Street
> Auckland, AUK 1010
> New Zealand
>
>
>
>
>
>
>
>
>
> Inactive hide details for "Uwe Falke" ---04/21/2017 03:07:48 AM---Hi
> Kennmeth, is prefetching off or on at your storage backe"Uwe Falke"
> ---04/21/2017 03:07:48 AM---Hi Kennmeth, is prefetching off or on at
> your storage backend?
>
> From: "Uwe Falke" <UWEFALKE at de.ibm.com>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Date: 04/21/2017 03:07 AM
> Subject: Re: [gpfsug-discuss] bizarre performance behavior
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
>
> ------------------------------------------------------------------------
>
>
>
> Hi Kennmeth,
>
> is prefetching off or on at your storage backend?
> Raw sequential is very different from GPFS sequential at the storage
> device !
> GPFS does its own prefetching, the storage would never know what sectors
> sequential read at GPFS level maps to at storage level!
>
>
> Mit freundlichen Grüßen / Kind regards
>
>
> Dr. Uwe Falke
>
> IT Specialist
> High Performance Computing Services / Integrated Technology Services /
> Data Center Services
> -------------------------------------------------------------------------------------------------------------------------------------------
> IBM Deutschland
> Rathausstr. 7
> 09111 Chemnitz
> Phone: +49 371 6978 2165
> Mobile: +49 175 575 2877
> E-Mail: uwefalke at de.ibm.com
> -------------------------------------------------------------------------------------------------------------------------------------------
> IBM Deutschland Business & Technology Services GmbH / Geschäftsführung:
> Andreas Hasse, Thorsten Moehring
> Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart,
> HRB 17122
>
>
>
>
> From: Kenneth Waegeman <kenneth.waegeman at ugent.be>
> To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Date: 04/20/2017 04:53 PM
> Subject: Re: [gpfsug-discuss] bizarre performance behavior
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
>
>
>
> Hi,
>
> Having an issue that looks the same as this one:
> We can do sequential writes to the filesystem at 7,8 GB/s total ,
> which is
> the expected speed for our current storage
> backend. While we have even better performance with sequential reads on
> raw storage LUNS, using GPFS we can only reach 1GB/s in total (each nsd
> server seems limited by 0,5GB/s) independent of the number of clients
> (1,2,4,..) or ways we tested (fio,dd). We played with blockdev params,
> MaxMBps, PrefetchThreads, hyperthreading, c1e/cstates, .. as discussed in
> this thread, but nothing seems to impact this read performance.
> Any ideas?
> Thanks!
>
> Kenneth
>
> On 17/02/17 19:29, Jan-Frode Myklebust wrote:
> I just had a similar experience from a sandisk infiniflash system
> SAS-attached to s single host. Gpfsperf reported 3,2 Gbyte/s for writes.
> and 250-300 Mbyte/s on sequential reads!! Random reads were on the order
> of 2 Gbyte/s.
>
> After a bit head scratching snd fumbling around I found out that reducing
> maxMBpS from 10000 to 100 fixed the problem! Digging further I found that
> reducing prefetchThreads from default=72 to 32 also fixed it, while
> leaving maxMBpS at 10000. Can now also read at 3,2 GByte/s.
>
> Could something like this be the problem on your box as well?
>
>
>
> -jf
> fre. 17. feb. 2017 kl. 18.13 skrev Aaron Knister <aaron.s.knister at nasa.gov
> >:
> Well, I'm somewhat scrounging for hardware. This is in our test
> environment :) And yep, it's got the 2U gpu-tray in it although even
> without the riser it has 2 PCIe slots onboard (excluding the on-board
> dual-port mezz card) so I think it would make a fine NSD server even
> without the riser.
>
> -Aaron
>
> On 2/17/17 11:43 AM, Simon Thompson (Research Computing - IT Services)
> wrote:
> > Maybe its related to interrupt handlers somehow? You drive the load up
> on one socket, you push all the interrupt handling to the other socket
> where the fabric card is attached?
> >
> > Dunno ... (Though I am intrigued you use idataplex nodes as NSD
> servers,
> I assume its some 2U gpu-tray riser one or something !)
> >
> > Simon
> > ________________________________________
> > From: gpfsug-discuss-bounces at spectrumscale.org [
> gpfsug-discuss-bounces at spectrumscale.org] on behalf of Aaron Knister [
> aaron.s.knister at nasa.gov]
> > Sent: 17 February 2017 15:52
> > To: gpfsug main discussion list
> > Subject: [gpfsug-discuss] bizarre performance behavior
> >
> > This is a good one. I've got an NSD server with 4x 16GB fibre
> > connections coming in and 1x FDR10 and 1x QDR connection going out to
> > the clients. I was having a really hard time getting anything resembling
> > sensible performance out of it (4-5Gb/s writes but maybe 1.2Gb/s for
> > reads). The back-end is a DDN SFA12K and I *know* it can do better than
> > that.
> >
> > I don't remember quite how I figured this out but simply by running
> > "openssl speed -multi 16" on the nsd server to drive up the load I saw
> > an almost 4x performance jump which is pretty much goes against every
> > sysadmin fiber in me (i.e. "drive up the cpu load with unrelated crap to
> > quadruple your i/o performance").
> >
> > This feels like some type of C-states frequency scaling shenanigans that
> > I haven't quite ironed down yet. I booted the box with the following
> > kernel parameters "intel_idle.max_cstate=0 processor.max_cstate=0" which
> > didn't seem to make much of a difference. I also tried setting the
> > frequency governer to userspace and setting the minimum frequency to
> > 2.6ghz (it's a 2.6ghz cpu). None of that really matters-- I still have
> > to run something to drive up the CPU load and then performance improves.
> >
> > I'm wondering if this could be an issue with the C1E state? I'm curious
> > if anyone has seen anything like this. The node is a dx360 M4
> > (Sandybridge) with 16 2.6GHz cores and 32GB of RAM.
> >
> > -Aaron
> >
> > --
> > Aaron Knister
> > NASA Center for Climate Simulation (Code 606.2)
> > Goddard Space Flight Center
> > (301) 286-2776
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org
> > _http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at spectrumscale.org
> > _http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
> >
>
> --
> Aaron Knister
> NASA Center for Climate Simulation (Code 606.2)
> Goddard Space Flight Center
> (301) 286-2776
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org_
> __http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org_
> __http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org_
> __http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org_
> __http://gpfsug.org/mailman/listinfo/gpfsug-discuss_
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170421/cab22a7f/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 45 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170421/cab22a7f/attachment-0016.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 3720 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170421/cab22a7f/attachment-0017.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 2741 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170421/cab22a7f/attachment-0002.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 13421 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170421/cab22a7f/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 45 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170421/cab22a7f/attachment-0018.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 45 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170421/cab22a7f/attachment-0019.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 45 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170421/cab22a7f/attachment-0020.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 45 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170421/cab22a7f/attachment-0021.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 45 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170421/cab22a7f/attachment-0022.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170421/cab22a7f/attachment-0023.gif>
More information about the gpfsug-discuss
mailing list