[gpfsug-discuss] Change to default for verbsRdmaMinBytes?

Kenneth Waegeman kenneth.waegeman at ugent.be
Wed Sep 6 12:55:20 BST 2017


Hi Sven,

I see two parameters that we have set to non-default values that are not 
in your list of options still to configure.

verbsRdmasPerConnection (256) and
socketMaxListenConnections (1024)

I remember we had to set socketMaxListenConnections because our cluster 
consist of +550 nodes.

Are these settings still needed, or is this also tackled in the code?

Thank you!!

Cheers,
Kenneth


On 02/09/17 00:42, Sven Oehme wrote:
> Hi Ed,
>
> yes the defaults for that have changed for customers who had not 
> overridden the default settings. the reason we did this was that many 
> systems in the field including all ESS systems that come pre-tuned 
> where manually changed to 8k from the 16k default due to better 
> performance that was confirmed in multiple customer engagements and 
> tests with various settings , therefore we change the default to what 
> it should be in the field so people are not bothered to set it anymore 
> (simplification) or get benefits by changing the default to provides 
> better performance.
> all this happened when we did the communication code overhaul that did 
> lead to significant (think factors) of improved RPC performance for 
> RDMA and VERBS workloads.
> there is another round of significant enhancements coming soon , that 
> will make even more parameters either obsolete or change some of the 
> defaults for better out of the box performance.
> i see that we should probably enhance the communication of this 
> changes, not that i think this will have any negative effect compared 
> to what your performance was with the old setting i am actually pretty 
> confident that you get better performance with the new code, but by 
> setting parameters back to default on most 'manual tuned' probably 
> makes your system even faster.
> if you have a Scale Client on 4.2.3+ you really shouldn't have 
> anything set beside maxfilestocache, pagepool, workerthreads and 
> potential prefetch , if you are a protocol node, this and settings 
> specific to an  export (e.g. SMB, NFS set some special settings) , 
> pretty much everything else these days should be set to default so the 
> code can pick the correct parameters., if its not and you get better 
> performance by manual tweaking something i like to hear about it.
> on the communication side in the next release will eliminate another 
> set of parameters that are now 'auto set' and we plan to work on NSD 
> next.
> i presented various slides about the communication and simplicity 
> changes in various forums, latest public non NDA slides i presented 
> are here --> 
> http://files.gpfsug.org/presentations/2017/Manchester/08_Research_Topics.pdf
>
> hope this helps .
>
> Sven
>
>
>
> On Fri, Sep 1, 2017 at 1:56 PM Edward Wahl <ewahl at osc.edu 
> <mailto:ewahl at osc.edu>> wrote:
>
>     Howdy.  Just noticed this change to min RDMA packet size and I
>     don't seem to
>     see it in any patch notes.  Maybe I just skipped the one where
>     this changed?
>
>      mmlsconfig verbsRdmaMinBytes
>     verbsRdmaMinBytes 16384
>
>     (in case someone thinks we changed it)
>
>     [root at proj-nsd01 ~]# mmlsconfig |grep verbs
>     verbsRdma enable
>     verbsRdma disable
>     verbsRdmasPerConnection 14
>     verbsRdmasPerNode 1024
>     verbsPorts mlx5_3/1
>     verbsPorts mlx4_0
>     verbsPorts mlx5_0
>     verbsPorts mlx5_0 mlx5_1
>     verbsPorts mlx4_1/1
>     verbsPorts mlx4_1/2
>
>
>     Oddly I also see this in config, though I've seen these kinds of
>     things before.
>     mmdiag --config |grep verbsRdmaMinBytes
>        verbsRdmaMinBytes 8192
>
>     We're on a recent efix.
>     Current GPFS build: "4.2.2.3 efix21 (1028007)".
>
>     --
>
>     Ed Wahl
>     Ohio Supercomputer Center
>     614-292-9302 <tel:%28614%29%20292-9302>
>     _______________________________________________
>     gpfsug-discuss mailing list
>     gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
>     http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170906/c0192e11/attachment-0002.htm>


More information about the gpfsug-discuss mailing list