<div dir="ltr"><div dir="ltr" class="gmail_msg">its more likely you run out of verbsRdmasPerNode which is the top limit across all connections for a given node.</div><div dir="ltr" class="gmail_msg"><br></div><div dir="ltr" class="gmail_msg">Sven<br><div class="gmail_msg"><br class="gmail_msg"></div></div><br class="gmail_msg"><div class="gmail_quote gmail_msg"><div dir="ltr" class="gmail_msg">On Fri, Feb 24, 2017 at 11:31 AM Aaron Knister <<a href="mailto:aaron.s.knister@nasa.gov" class="gmail_msg" target="_blank">aaron.s.knister@nasa.gov</a>> wrote:<br class="gmail_msg"></div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Interesting, thanks Sven!<br class="gmail_msg">
<br class="gmail_msg">
Could "resources" I'm running out of include NSD server queues?<br class="gmail_msg">
<br class="gmail_msg">
On 2/23/17 12:12 PM, Sven Oehme wrote:<br class="gmail_msg">
> all this waiter shows is that you have more in flight than the node or<br class="gmail_msg">
> connection can currently serve. the reasons for that can be<br class="gmail_msg">
> misconfiguration or you simply run out of resources on the node, not the<br class="gmail_msg">
> connection. with latest code you shouldn't see this anymore for node<br class="gmail_msg">
> limits as the system automatically adjusts the number of maximum RDMA's<br class="gmail_msg">
> according to the systems Node capabilities :<br class="gmail_msg">
><br class="gmail_msg">
> you should see messages in your mmfslog like :<br class="gmail_msg">
><br class="gmail_msg">
> 2017-02-23_06:19:50.056-0800: [I] VERBS RDMA starting with<br class="gmail_msg">
> verbsRdmaCm=no verbsRdmaSend=yes verbsRdmaUseMultiCqThreads=yes<br class="gmail_msg">
> verbsRdmaUseCompVectors=yes<br class="gmail_msg">
> 2017-02-23_06:19:50.078-0800: [I] VERBS RDMA library libibverbs.so<br class="gmail_msg">
> (version >= 1.1) loaded and initialized.<br class="gmail_msg">
> 2017-02-23_06:19:50.078-0800: [I] VERBS RDMA verbsRdmasPerNode increased<br class="gmail_msg">
> from*_3072 to 3740 because verbsRdmasPerNodeOptimize is set to yes._*<br class="gmail_msg">
> 2017-02-23_06:19:50.121-0800: [I] VERBS RDMA discover mlx5_5 port 1<br class="gmail_msg">
> transport IB link IB NUMA node 16 pkey[0] 0xFFFF gid[0] subnet<br class="gmail_msg">
> 0xFEC0000000000013 id 0xE41D2D0300FDB9CD state ACTIVE<br class="gmail_msg">
> 2017-02-23_06:19:50.137-0800: [I] VERBS RDMA discover mlx5_4 port 1<br class="gmail_msg">
> transport IB link IB NUMA node 16 pkey[0] 0xFFFF gid[0] subnet<br class="gmail_msg">
> 0xFEC0000000000015 id 0xE41D2D0300FDB9CC state ACTIVE<br class="gmail_msg">
> 2017-02-23_06:19:50.153-0800: [I] VERBS RDMA discover mlx5_3 port 1<br class="gmail_msg">
> transport IB link IB NUMA node 1 pkey[0] 0xFFFF gid[0] subnet<br class="gmail_msg">
> 0xFEC0000000000013 id 0xE41D2D0300FDB751 state ACTIVE<br class="gmail_msg">
> 2017-02-23_06:19:50.169-0800: [I] VERBS RDMA discover mlx5_2 port 1<br class="gmail_msg">
> transport IB link IB NUMA node 1 pkey[0] 0xFFFF gid[0] subnet<br class="gmail_msg">
> 0xFEC0000000000015 id 0xE41D2D0300FDB750 state ACTIVE<br class="gmail_msg">
> 2017-02-23_06:19:50.185-0800: [I] VERBS RDMA discover mlx5_1 port 1<br class="gmail_msg">
> transport IB link IB NUMA node 0 pkey[0] 0xFFFF gid[0] subnet<br class="gmail_msg">
> 0xFEC0000000000013 id 0xE41D2D0300FDB78D state ACTIVE<br class="gmail_msg">
> 2017-02-23_06:19:50.201-0800: [I] VERBS RDMA discover mlx5_0 port 1<br class="gmail_msg">
> transport IB link IB NUMA node 0 pkey[0] 0xFFFF gid[0] subnet<br class="gmail_msg">
> 0xFEC0000000000015 id 0xE41D2D0300FDB78C state ACTIVE<br class="gmail_msg">
><br class="gmail_msg">
> we want to eliminate all this configurable limits eventually, but this<br class="gmail_msg">
> takes time, but as you can see above, we make progress on each release :-)<br class="gmail_msg">
><br class="gmail_msg">
> Sven<br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
> On Thu, Feb 23, 2017 at 9:05 AM Aaron Knister <<a href="mailto:aaron.s.knister@nasa.gov" class="gmail_msg" target="_blank">aaron.s.knister@nasa.gov</a><br class="gmail_msg">
> <mailto:<a href="mailto:aaron.s.knister@nasa.gov" class="gmail_msg" target="_blank">aaron.s.knister@nasa.gov</a>>> wrote:<br class="gmail_msg">
><br class="gmail_msg">
> On a particularly heavy loaded NSD server I'm seeing a lot of these<br class="gmail_msg">
> messages:<br class="gmail_msg">
><br class="gmail_msg">
> 0x7FFFF08B63E0 ( 15539) waiting 0.004139456 seconds, NSDThread: on<br class="gmail_msg">
> ThCond 0x7FFFA80772C8 (0x7FFFA80772C8) (VERBSEventWaitCondvar), reason<br class="gmail_msg">
> 'waiting for conn rdmas < conn maxrdmas'<br class="gmail_msg">
> 0x7FFFF08EED80 ( 15584) waiting 0.004075718 seconds, NSDThread: on<br class="gmail_msg">
> ThCond 0x7FFF680008F8 (0x7FFF680008F8) (VERBSEventWaitCondvar), reason<br class="gmail_msg">
> 'waiting for conn rdmas < conn maxrdmas'<br class="gmail_msg">
> 0x7FFFF08FDF00 ( 15596) waiting 0.003965504 seconds, NSDThread: on<br class="gmail_msg">
> ThCond 0x7FFF8C00E288 (0x7FFF8C00E288) (VERBSEventWaitCondvar), reason<br class="gmail_msg">
> 'waiting for conn rdmas < conn maxrdmas'<br class="gmail_msg">
> 0x7FFFF09185A0 ( 15617) waiting 0.003916346 seconds, NSDThread: on<br class="gmail_msg">
> ThCond 0x7FFF9000CB18 (0x7FFF9000CB18) (VERBSEventWaitCondvar), reason<br class="gmail_msg">
> 'waiting for conn rdmas < conn maxrdmas'<br class="gmail_msg">
> 0x7FFFF092B380 ( 15632) waiting 0.003659610 seconds, NSDThread: on<br class="gmail_msg">
> ThCond 0x1DB04B8 (0x1DB04B8) (VERBSEventWaitCondvar), reason 'waiting<br class="gmail_msg">
> for conn rdmas < conn maxrdmas'<br class="gmail_msg">
><br class="gmail_msg">
> I've tried tweaking verbsRdmasPerConnection but the issue seems to<br class="gmail_msg">
> persist. Has anyone has encountered this and if so how'd you fix it?<br class="gmail_msg">
><br class="gmail_msg">
> -Aaron<br class="gmail_msg">
><br class="gmail_msg">
> --<br class="gmail_msg">
> Aaron Knister<br class="gmail_msg">
> NASA Center for Climate Simulation (Code 606.2)<br class="gmail_msg">
> Goddard Space Flight Center<br class="gmail_msg">
> <a href="tel:(301)%20286-2776" value="+13012862776" class="gmail_msg" target="_blank">(301) 286-2776</a> <tel:(301)%20286-2776><br class="gmail_msg">
> _______________________________________________<br class="gmail_msg">
> gpfsug-discuss mailing list<br class="gmail_msg">
> gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" class="gmail_msg" target="_blank">spectrumscale.org</a> <<a href="http://spectrumscale.org" rel="noreferrer" class="gmail_msg" target="_blank">http://spectrumscale.org</a>><br class="gmail_msg">
> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" class="gmail_msg" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
> _______________________________________________<br class="gmail_msg">
> gpfsug-discuss mailing list<br class="gmail_msg">
> gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" class="gmail_msg" target="_blank">spectrumscale.org</a><br class="gmail_msg">
> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" class="gmail_msg" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="gmail_msg">
><br class="gmail_msg">
<br class="gmail_msg">
--<br class="gmail_msg">
Aaron Knister<br class="gmail_msg">
NASA Center for Climate Simulation (Code 606.2)<br class="gmail_msg">
Goddard Space Flight Center<br class="gmail_msg">
<a href="tel:(301)%20286-2776" value="+13012862776" class="gmail_msg" target="_blank">(301) 286-2776</a><br class="gmail_msg">
_______________________________________________<br class="gmail_msg">
gpfsug-discuss mailing list<br class="gmail_msg">
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" class="gmail_msg" target="_blank">spectrumscale.org</a><br class="gmail_msg">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" class="gmail_msg" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="gmail_msg">
</blockquote></div></div>