[gpfsug-discuss] Change to default for verbsRdmaMinBytes?
Truong Vu
truongv at us.ibm.com
Fri Sep 1 23:56:23 BST 2017
The discrepancy between the mmlsconfig view and mmdiag has been fixed in
GFPS 4.2.3 version. Note, mmdiag reports the correct default value.
Tru.
From: gpfsug-discuss-request at spectrumscale.org
To: gpfsug-discuss at spectrumscale.org
Date: 09/01/2017 06:43 PM
Subject: gpfsug-discuss Digest, Vol 68, Issue 2
Sent by: gpfsug-discuss-bounces at spectrumscale.org
Send gpfsug-discuss mailing list submissions to
gpfsug-discuss at spectrumscale.org
To subscribe or unsubscribe via the World Wide Web, visit
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HQmkdQWQHoc1Nu6Mg_g8NVugim3OiUUy5n0QgLQcbkM&m=yK4FkYvJ21ubvurR6W1Pi3qvNw9ydj2XP0ghXPc7DUw&s=xZHUN9ZlFjvgBmBB8wnX2cQDQQV42R_q-xHubNA3JBM&e=
or, via email, send a message with subject or body 'help' to
gpfsug-discuss-request at spectrumscale.org
You can reach the person managing the list at
gpfsug-discuss-owner at spectrumscale.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of gpfsug-discuss digest..."
Today's Topics:
1. Re: GPFS GUI Nodes > NSD no data (Sobey, Richard A)
2. Change to default for verbsRdmaMinBytes? (Edward Wahl)
3. Quorum managers (Joshua Akers)
4. Re: Change to default for verbsRdmaMinBytes? (Sven Oehme)
----------------------------------------------------------------------
Message: 1
Date: Fri, 1 Sep 2017 13:36:56 +0000
From: "Sobey, Richard A" <r.sobey at imperial.ac.uk>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] GPFS GUI Nodes > NSD no data
Message-ID:
<HE1PR0602MB3225B01281871606C60C6333DF920 at HE1PR0602MB3225.eurprd06.prod.outlook.com>
Content-Type: text/plain; charset="us-ascii"
Resolved this, guessed at changing GPFSNSDDisk.period to 5.
From: gpfsug-discuss-bounces at spectrumscale.org [
mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey,
Richard A
Sent: 01 September 2017 09:45
To: 'gpfsug-discuss at spectrumscale.org' <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] GPFS GUI Nodes > NSD no data
For some time now if I go into the GUI, select Monitoring > Nodes > NSD
Server Nodes, the only columns with good data are Name, State and NSD
Count. Everything else e.g. Avg Disk Wait Read is listed "N/A".
Is this another config option I need to enable? It's been bugging me for a
while, I don't think I've seen it work since 4.2.1 which was the first time
I saw the GUI.
Cheers
Richard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20170901_2a4162e9_attachment-2D0001.html&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HQmkdQWQHoc1Nu6Mg_g8NVugim3OiUUy5n0QgLQcbkM&m=yK4FkYvJ21ubvurR6W1Pi3qvNw9ydj2XP0ghXPc7DUw&s=jcPGl5zwtQFMbnEmBpNErsD43uwoVeKgKk_8j7ZeCJY&e=
>
------------------------------
Message: 2
Date: Fri, 1 Sep 2017 16:56:25 -0400
From: Edward Wahl <ewahl at osc.edu>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] Change to default for verbsRdmaMinBytes?
Message-ID: <20170901165625.6e4edd4c at osc.edu>
Content-Type: text/plain; charset="US-ASCII"
Howdy. Just noticed this change to min RDMA packet size and I don't seem
to
see it in any patch notes. Maybe I just skipped the one where this
changed?
mmlsconfig verbsRdmaMinBytes
verbsRdmaMinBytes 16384
(in case someone thinks we changed it)
[root at proj-nsd01 ~]# mmlsconfig |grep verbs
verbsRdma enable
verbsRdma disable
verbsRdmasPerConnection 14
verbsRdmasPerNode 1024
verbsPorts mlx5_3/1
verbsPorts mlx4_0
verbsPorts mlx5_0
verbsPorts mlx5_0 mlx5_1
verbsPorts mlx4_1/1
verbsPorts mlx4_1/2
Oddly I also see this in config, though I've seen these kinds of things
before.
mmdiag --config |grep verbsRdmaMinBytes
verbsRdmaMinBytes 8192
We're on a recent efix.
Current GPFS build: "4.2.2.3 efix21 (1028007)".
--
Ed Wahl
Ohio Supercomputer Center
614-292-9302
------------------------------
Message: 3
Date: Fri, 01 Sep 2017 21:06:15 +0000
From: Joshua Akers <akers at vt.edu>
To: "gpfsug-discuss at spectrumscale.org"
<gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] Quorum managers
Message-ID:
<CAHO5rBG+PkntpshV105j54
+O4CtcDXqQCb9AJutq-s_PEN0g3A at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Hi all,
I was wondering how most people set up quorum managers. We historically had
physical admin nodes be the quorum managers, but are switching to a
virtualized admin services infrastructure. We have been choosing a few
compute nodes to act as quorum managers in our client clusters, but have
considered using virtual machines instead. Has anyone else done this?
Regards,
Josh
--
*Joshua D. Akers*
*HPC Team Lead*
NI&S Systems Support (MC0214)
1700 Pratt Drive
Blacksburg, VA 24061
540-231-9506
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20170901_a49947db_attachment-2D0001.html&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HQmkdQWQHoc1Nu6Mg_g8NVugim3OiUUy5n0QgLQcbkM&m=yK4FkYvJ21ubvurR6W1Pi3qvNw9ydj2XP0ghXPc7DUw&s=Gag0raQbp7KZAyINlnmuxlnpjboo9XOWO3dDL2HCsZo&e=
>
------------------------------
Message: 4
Date: Fri, 01 Sep 2017 22:42:55 +0000
From: Sven Oehme <oehmes at gmail.com>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Change to default for verbsRdmaMinBytes?
Message-ID:
<CALssuR3-KmWd4LqKZwQkP7_F6+aX3JM-KqRnc=
+czkxMP3xCMg at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Hi Ed,
yes the defaults for that have changed for customers who had not overridden
the default settings. the reason we did this was that many systems in the
field including all ESS systems that come pre-tuned where manually changed
to 8k from the 16k default due to better performance that was confirmed in
multiple customer engagements and tests with various settings , therefore
we change the default to what it should be in the field so people are not
bothered to set it anymore (simplification) or get benefits by changing the
default to provides better performance.
all this happened when we did the communication code overhaul that did lead
to significant (think factors) of improved RPC performance for RDMA and
VERBS workloads.
there is another round of significant enhancements coming soon , that will
make even more parameters either obsolete or change some of the defaults
for better out of the box performance.
i see that we should probably enhance the communication of this changes,
not that i think this will have any negative effect compared to what your
performance was with the old setting i am actually pretty confident that
you get better performance with the new code, but by setting parameters
back to default on most 'manual tuned' probably makes your system even
faster.
if you have a Scale Client on 4.2.3+ you really shouldn't have anything set
beside maxfilestocache, pagepool, workerthreads and potential prefetch , if
you are a protocol node, this and settings specific to an export (e.g.
SMB, NFS set some special settings) , pretty much everything else these
days should be set to default so the code can pick the correct parameters.,
if its not and you get better performance by manual tweaking something i
like to hear about it.
on the communication side in the next release will eliminate another set of
parameters that are now 'auto set' and we plan to work on NSD next.
i presented various slides about the communication and simplicity changes
in various forums, latest public non NDA slides i presented are here -->
https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2017_Manchester_08-5FResearch-5FTopics.pdf&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HQmkdQWQHoc1Nu6Mg_g8NVugim3OiUUy5n0QgLQcbkM&m=yK4FkYvJ21ubvurR6W1Pi3qvNw9ydj2XP0ghXPc7DUw&s=8c_55Ld_iAC2sr_QU0cyGiOiyU7Z9NjcVknVuRpRIlk&e=
hope this helps .
Sven
On Fri, Sep 1, 2017 at 1:56 PM Edward Wahl <ewahl at osc.edu> wrote:
> Howdy. Just noticed this change to min RDMA packet size and I don't
seem
> to
> see it in any patch notes. Maybe I just skipped the one where this
> changed?
>
> mmlsconfig verbsRdmaMinBytes
> verbsRdmaMinBytes 16384
>
> (in case someone thinks we changed it)
>
> [root at proj-nsd01 ~]# mmlsconfig |grep verbs
> verbsRdma enable
> verbsRdma disable
> verbsRdmasPerConnection 14
> verbsRdmasPerNode 1024
> verbsPorts mlx5_3/1
> verbsPorts mlx4_0
> verbsPorts mlx5_0
> verbsPorts mlx5_0 mlx5_1
> verbsPorts mlx4_1/1
> verbsPorts mlx4_1/2
>
>
> Oddly I also see this in config, though I've seen these kinds of things
> before.
> mmdiag --config |grep verbsRdmaMinBytes
> verbsRdmaMinBytes 8192
>
> We're on a recent efix.
> Current GPFS build: "4.2.2.3 efix21 (1028007)".
>
> --
>
> Ed Wahl
> Ohio Supercomputer Center
> 614-292-9302 <(614)%20292-9302>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
>
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HQmkdQWQHoc1Nu6Mg_g8NVugim3OiUUy5n0QgLQcbkM&m=yK4FkYvJ21ubvurR6W1Pi3qvNw9ydj2XP0ghXPc7DUw&s=xZHUN9ZlFjvgBmBB8wnX2cQDQQV42R_q-xHubNA3JBM&e=
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments_20170901_b75cfc74_attachment.html&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HQmkdQWQHoc1Nu6Mg_g8NVugim3OiUUy5n0QgLQcbkM&m=yK4FkYvJ21ubvurR6W1Pi3qvNw9ydj2XP0ghXPc7DUw&s=LpVpXMgqE_LD-t_J7yfNwURUrdUR29TzWvjVTi18kpA&e=
>
------------------------------
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=HQmkdQWQHoc1Nu6Mg_g8NVugim3OiUUy5n0QgLQcbkM&m=yK4FkYvJ21ubvurR6W1Pi3qvNw9ydj2XP0ghXPc7DUw&s=xZHUN9ZlFjvgBmBB8wnX2cQDQQV42R_q-xHubNA3JBM&e=
End of gpfsug-discuss Digest, Vol 68, Issue 2
*********************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170901/9ac0aa6b/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170901/9ac0aa6b/attachment-0002.gif>
More information about the gpfsug-discuss
mailing list