<font size=2 face="sans-serif">Hi Kenneth, </font><br><font size=2 face="sans-serif">taking the network/TCP part... </font><br><br><font size=2 face="sans-serif">yes , you need the </font><font size=3>socketMaxListenConnections
(1024)</font><font size=2 face="sans-serif">, in case you have so many
nodes... </font><br><font size=2 face="sans-serif">keep in mind, that in addition, your
operating system may need to be adjusted as well </font><br><br><font size=2 face="sans-serif">e.g. depending on your OS ... </font><br><tt><font size=3>[root@ems1 patch]# sysctl net.core.somaxconn
<br>net.core.somaxconn = 1024<br></font></tt><tt><font size=3 color=#ff4040><b>saphana1:/hana/shared # </b></font></tt><tt><font size=3>sysctl
net.core.somaxconn <br>net.core.somaxconn = 128<br></font></tt><div><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Kenneth Waegeman <kenneth.waegeman@ugent.be></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug main discussion
list <gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">09/06/2017 01:55 PM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
Change to default for verbsRdmaMinBytes?</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><font size=3>Hi Sven,</font><p><font size=3>I see two parameters that we have set to non-default values
that are not in your list of options still to configure.</font><p><font size=3>verbsRdmasPerConnection (256) and<br>socketMaxListenConnections (1024)<br><br>I remember we had to set socketMaxListenConnections because our cluster
consist of +550 nodes.<br><br>Are these settings still needed, or is this also tackled in the code?<br><br>Thank you!!<br><br>Cheers,<br>Kenneth<br><br></font><br><font size=3>On 02/09/17 00:42, Sven Oehme wrote:</font><br><font size=3>Hi Ed, </font><br><br><font size=3>yes the defaults for that have changed for customers who
had not overridden the default settings. the reason we did this was that
many systems in the field including all ESS systems that come pre-tuned
where manually changed to 8k from the 16k default due to better performance
that was confirmed in multiple customer engagements and tests with various
settings , therefore we change the default to what it should be in the
field so people are not bothered to set it anymore (simplification) or
get benefits by changing the default to provides better performance. </font><br><font size=3>all this happened when we did the communication code overhaul
that did lead to significant (think factors) of improved RPC performance
for RDMA and VERBS workloads. </font><br><font size=3>there is another round of significant enhancements coming
soon , that will make even more parameters either obsolete or change some
of the defaults for better out of the box performance.</font><br><font size=3>i see that we should probably enhance the communication
of this changes, not that i think this will have any negative effect compared
to what your performance was with the old setting i am actually pretty
confident that you get better performance with the new code, but by setting
parameters back to default on most 'manual tuned' probably makes your system
even faster. </font><br><font size=3>if you have a Scale Client on 4.2.3+ you really shouldn't
have anything set beside maxfilestocache, pagepool, workerthreads and potential
prefetch , if you are a protocol node, this and settings specific to an
export (e.g. SMB, NFS set some special settings) , pretty much everything
else these days should be set to default so the code can pick the correct
parameters., if its not and you get better performance by manual tweaking
something i like to hear about it.</font><br><font size=3>on the communication side in the next release will eliminate
another set of parameters that are now 'auto set' and we plan to work on
NSD next. </font><br><font size=3>i presented various slides about the communication and
simplicity changes in various forums, latest public non NDA slides i presented
are here --> </font><a href=http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf><font size=3 color=blue><u>http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf</u></font></a><br><br><font size=3>hope this helps . </font><br><br><font size=3>Sven</font><br><br><br><br><font size=3>On Fri, Sep 1, 2017 at 1:56 PM Edward Wahl <</font><a href=mailto:ewahl@osc.edu></a><a href=mailto:ewahl@osc.edu><font size=3 color=blue><u>ewahl@osc.edu</u></font></a><font size=3>>
wrote:</font><br><font size=3>Howdy. Just noticed this change to min RDMA
packet size and I don't seem to<br>see it in any patch notes. Maybe I just skipped the one where this
changed?<br><br> mmlsconfig verbsRdmaMinBytes<br>verbsRdmaMinBytes 16384<br><br>(in case someone thinks we changed it)<br><br>[root@proj-nsd01 ~]# mmlsconfig |grep verbs<br>verbsRdma enable<br>verbsRdma disable<br>verbsRdmasPerConnection 14<br>verbsRdmasPerNode 1024<br>verbsPorts mlx5_3/1<br>verbsPorts mlx4_0<br>verbsPorts mlx5_0<br>verbsPorts mlx5_0 mlx5_1<br>verbsPorts mlx4_1/1<br>verbsPorts mlx4_1/2<br><br><br>Oddly I also see this in config, though I've seen these kinds of things
before.<br>mmdiag --config |grep verbsRdmaMinBytes<br> verbsRdmaMinBytes 8192<br><br>We're on a recent efix.<br>Current GPFS build: "4.2.2.3 efix21 (1028007)".<br><br>--<br><br>Ed Wahl<br>Ohio Supercomputer Center</font><font size=3 color=blue><u><br></u></font><a href="tel:%28614%29%20292-9302" target=_blank><font size=3 color=blue><u>614-292-9302</u></font></a><font size=3><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href=http://spectrumscale.org/ target=_blank><font size=3 color=blue><u>spectrumscale.org</u></font></a><font size=3 color=blue><u><br></u></font><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target=_blank><font size=3 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></a><br><font size=3><br></font><br><tt><font size=3>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=3 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></tt></a><tt><font size=3><br></font></tt><br><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br></font></tt><br><br></div><BR>