[gpfsug-discuss] Spectrum Scale Slow to create directories

[Peter Childs]

After a load more debugging, and switching off the quota's the issue looks to be quota related. in that the issue has gone away since I switched quota's off.

I will need to switch them back on, but at least we know the issue is not the network and is likely to be fixed by upgrading.....


Peter Childs
ITS Research Infrastructure
Queen Mary, University of London


________________________________________
From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Peter Childs <p.childs at qmul.ac.uk>
Sent: Tuesday, April 11, 2017 8:35:40 PM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] Spectrum Scale Slow to create directories

Can you remember what version you were running? Don't worry if you can't remember.

It looks like ibm may have withdrawn 4.2.1<tel:4.2.1> from fix central and wish to forget its existences. Never a good sign, 4.2.0<tel:4.2.0>, 4.2.2<tel:4.2.2>, 4.2.3<tel:4.2.3> and even 3.5, so maybe upgrading is worth a try.

I've looked at all the standard trouble shouting guides and got nowhere hence why I asked. But another set of slides always helps.

Thank-you for the help, still head scratching....  Which only makes the issue more random.

Peter Childs
Research Storage
ITS Research and Teaching Support
Queen Mary, University of London


---- Simon Thompson (IT Research Support) wrote ----

We actually saw this for a while on one of our clusters which was new. But
by the time I'd got round to looking deeper, it had gone, maybe we were
using the NSDs more heavily, or possibly we'd upgraded. We are at 4.2.2-2,
so might be worth trying to bump the version and see if it goes away.

We saw it on the NSD servers directly as well, so not some client trying
to talk to it, so maybe there was some buggy code?

Simon

On 11/04/2017, 16:51, "gpfsug-discuss-bounces at spectrumscale.org on behalf
of Bryan Banister" <gpfsug-discuss-bounces at spectrumscale.org on behalf of
bbanister at jumptrading.com> wrote:

>There are so many things to look at and many tools for doing so (iostat,
>htop, nsdperf, mmdiag, mmhealth, mmlsconfig, mmlsfs, etc).  I would
>recommend a review of the presentation that Yuri gave at the most recent
>GPFS User Group:
>https://drive.google.com/drive/folders/0B124dhp9jJC-UjFlVjJTa2ZaVWs
>
>Cheers,
>-Bryan
>
>-----Original Message-----
>From: gpfsug-discuss-bounces at spectrumscale.org
>[mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Peter
>Childs
>Sent: Tuesday, April 11, 2017 3:58 AM
>To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
>Subject: [gpfsug-discuss] Spectrum Scale Slow to create directories
>
>This is a curious issue which I'm trying to get to the bottom of.
>
>We currently have two Spectrum Scale file systems, both are running GPFS
>4.2.1-1 some of the servers have been upgraded to 4.2.1-2.
>
>The older one which was upgraded from GPFS 3.5 works find create a
>directory is always fast and no issue.
>
>The new one, which has nice new SSD for metadata and hence should be
>faster. can take up to 30 seconds to create a directory but usually takes
>less than a second, The longer directory creates usually happen on busy
>nodes that have not used the new storage in a while. (Its new so we've
>not moved much of the data over yet) But it can also happen randomly
>anywhere, including from the NSD servers them selves. (times of 3-4
>seconds from the NSD servers have been seen, on a single directory create)
>
>We've been pointed at the network and suggested we check all network
>settings, and its been suggested to build an admin network, but I'm not
>sure I entirely understand why and how this would help. Its a mixed
>1G/10G network with the NSD servers connected at 40G with an MTU of 9000.
>
>However as I say, the older filesystem is fine, and it does not matter if
>the nodes are connected to the old GPFS cluster or the new one, (although
>the delay is worst on the old gpfs cluster), So I'm really playing spot
>the difference. and the network is not really an obvious difference.
>
>Its been suggested to look at a trace when it occurs but as its difficult
>to recreate collecting one is difficult.
>
>Any ideas would be most helpful.
>
>Thanks
>
>
>
>Peter Childs
>ITS Research Infrastructure
>Queen Mary, University of London
>_______________________________________________
>gpfsug-discuss mailing list
>gpfsug-discuss at spectrumscale.org
>http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>________________________________
>
>Note: This email is for the confidential use of the named addressee(s)
>only and may contain proprietary, confidential or privileged information.
>If you are not the intended recipient, you are hereby notified that any
>review, dissemination or copying of this email is strictly prohibited,
>and to please notify the sender immediately and destroy this email and
>any attachments. Email transmission cannot be guaranteed to be secure or
>error-free. The Company, therefore, does not make any guarantees as to
>the completeness or accuracy of this email or any attachments. This email
>is for informational purposes only and does not constitute a
>recommendation, offer, request or solicitation of any kind to buy, sell,
>subscribe, redeem or perform any type of transaction of a financial
>product.
>_______________________________________________
>gpfsug-discuss mailing list
>gpfsug-discuss at spectrumscale.org
>http://gpfsug.org/mailman/listinfo/gpfsug-discuss

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss