<div dir="ltr"><div>all this waiter shows is that you have more in flight than the node or connection can currently serve. the reasons for that can be misconfiguration or you simply run out of resources on the node, not the connection. with latest code you shouldn't see this anymore for node limits as the system automatically adjusts the number of maximum RDMA's according to the systems Node capabilities : </div><div><br></div><div>you should see messages in your mmfslog like :</div><div><br></div><div><div>2017-02-23_06:19:50.056-0800: [I] VERBS RDMA starting with verbsRdmaCm=no verbsRdmaSend=yes verbsRdmaUseMultiCqThreads=yes verbsRdmaUseCompVectors=yes</div><div>2017-02-23_06:19:50.078-0800: [I] VERBS RDMA library libibverbs.so (version >= 1.1) loaded and initialized.</div><div>2017-02-23_06:19:50.078-0800: [I] VERBS RDMA verbsRdmasPerNode increased from<b><u> 3072 to 3740 because verbsRdmasPerNodeOptimize is set to yes.</u></b></div><div>2017-02-23_06:19:50.121-0800: [I] VERBS RDMA discover mlx5_5 port 1 transport IB link IB NUMA node 16 pkey[0] 0xFFFF gid[0] subnet 0xFEC0000000000013 id 0xE41D2D0300FDB9CD state ACTIVE</div><div>2017-02-23_06:19:50.137-0800: [I] VERBS RDMA discover mlx5_4 port 1 transport IB link IB NUMA node 16 pkey[0] 0xFFFF gid[0] subnet 0xFEC0000000000015 id 0xE41D2D0300FDB9CC state ACTIVE</div><div>2017-02-23_06:19:50.153-0800: [I] VERBS RDMA discover mlx5_3 port 1 transport IB link IB NUMA node 1 pkey[0] 0xFFFF gid[0] subnet 0xFEC0000000000013 id 0xE41D2D0300FDB751 state ACTIVE</div><div>2017-02-23_06:19:50.169-0800: [I] VERBS RDMA discover mlx5_2 port 1 transport IB link IB NUMA node 1 pkey[0] 0xFFFF gid[0] subnet 0xFEC0000000000015 id 0xE41D2D0300FDB750 state ACTIVE</div><div>2017-02-23_06:19:50.185-0800: [I] VERBS RDMA discover mlx5_1 port 1 transport IB link IB NUMA node 0 pkey[0] 0xFFFF gid[0] subnet 0xFEC0000000000013 id 0xE41D2D0300FDB78D state ACTIVE</div><div>2017-02-23_06:19:50.201-0800: [I] VERBS RDMA discover mlx5_0 port 1 transport IB link IB NUMA node 0 pkey[0] 0xFFFF gid[0] subnet 0xFEC0000000000015 id 0xE41D2D0300FDB78C state ACTIVE</div></div><div><br></div><div>we want to eliminate all this configurable limits eventually, but this takes time, but as you can see above, we make progress on each release :-)</div><div><br></div><div>Sven</div><div><br></div><div> <div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Thu, Feb 23, 2017 at 9:05 AM Aaron Knister <<a href="mailto:aaron.s.knister@nasa.gov">aaron.s.knister@nasa.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On a particularly heavy loaded NSD server I'm seeing a lot of these<br class="gmail_msg">
messages:<br class="gmail_msg">
<br class="gmail_msg">
0x7FFFF08B63E0 ( 15539) waiting 0.004139456 seconds, NSDThread: on<br class="gmail_msg">
ThCond 0x7FFFA80772C8 (0x7FFFA80772C8) (VERBSEventWaitCondvar), reason<br class="gmail_msg">
'waiting for conn rdmas < conn maxrdmas'<br class="gmail_msg">
0x7FFFF08EED80 ( 15584) waiting 0.004075718 seconds, NSDThread: on<br class="gmail_msg">
ThCond 0x7FFF680008F8 (0x7FFF680008F8) (VERBSEventWaitCondvar), reason<br class="gmail_msg">
'waiting for conn rdmas < conn maxrdmas'<br class="gmail_msg">
0x7FFFF08FDF00 ( 15596) waiting 0.003965504 seconds, NSDThread: on<br class="gmail_msg">
ThCond 0x7FFF8C00E288 (0x7FFF8C00E288) (VERBSEventWaitCondvar), reason<br class="gmail_msg">
'waiting for conn rdmas < conn maxrdmas'<br class="gmail_msg">
0x7FFFF09185A0 ( 15617) waiting 0.003916346 seconds, NSDThread: on<br class="gmail_msg">
ThCond 0x7FFF9000CB18 (0x7FFF9000CB18) (VERBSEventWaitCondvar), reason<br class="gmail_msg">
'waiting for conn rdmas < conn maxrdmas'<br class="gmail_msg">
0x7FFFF092B380 ( 15632) waiting 0.003659610 seconds, NSDThread: on<br class="gmail_msg">
ThCond 0x1DB04B8 (0x1DB04B8) (VERBSEventWaitCondvar), reason 'waiting<br class="gmail_msg">
for conn rdmas < conn maxrdmas'<br class="gmail_msg">
<br class="gmail_msg">
I've tried tweaking verbsRdmasPerConnection but the issue seems to<br class="gmail_msg">
persist. Has anyone has encountered this and if so how'd you fix it?<br class="gmail_msg">
<br class="gmail_msg">
-Aaron<br class="gmail_msg">
<br class="gmail_msg">
--<br class="gmail_msg">
Aaron Knister<br class="gmail_msg">
NASA Center for Climate Simulation (Code 606.2)<br class="gmail_msg">
Goddard Space Flight Center<br class="gmail_msg">
<a href="tel:(301)%20286-2776" value="+13012862776" class="gmail_msg" target="_blank">(301) 286-2776</a><br class="gmail_msg">
_______________________________________________<br class="gmail_msg">
gpfsug-discuss mailing list<br class="gmail_msg">
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" class="gmail_msg" target="_blank">spectrumscale.org</a><br class="gmail_msg">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" class="gmail_msg" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="gmail_msg">
</blockquote></div>