<font size=2 face="sans-serif">is ib_read_bw working ?</font><br><font size=2 face="sans-serif">just test it between the two nodes ...
</font><br><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Barry Evans <bevans@pixitmedia.com></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug main discussion
list <gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">09/20/2017 03:21 AM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">[gpfsug-discuss]
RoCE not playing ball</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><font size=3>Hi All,</font><br><br><font size=3>Weirdness with a RoCE interface - verbs is not playing
ball and is complaining about the inet6 address not matching up:</font><br><br><tt><font size=3>2017-09-02_07:46:01.376+0100: [I] VERBS RDMA starting
with verbsRdmaCm=yes verbsRdmaSend=no verbsRdmaUseMultiCqThreads=yes verbsRdmaUseCompVectors=yes</font></tt><br><tt><font size=3>2017-09-02_07:46:01.377+0100: [I] VERBS RDMA library
librdmacm.so (version >= 1.1) loaded and initialized.</font></tt><br><tt><font size=3>2017-09-02_07:46:01.377+0100: [I] VERBS RDMA verbsRdmasPerNode
reduced from 1000 to 514 to match (nsdMaxWorkerThreads 512 + (nspdThreadsPerQueue
2 * nspdQueues 1)).</font></tt><br><tt><font size=3>2017-09-02_07:46:01.382+0100: [I] VERBS RDMA discover
mlx4_1 port 1 transport IB link ETH NUMA node 0 pkey[0] 0xFFFF gid[0]
subnet 0xFE80000000000000 id 0x268A07FFFEF981C0 state ACTIVE</font></tt><br><tt><font size=3>2017-09-02_07:46:01.383+0100: [I] VERBS RDMA discover
mlx4_1 port 1 transport IB link ETH NUMA node 0 pkey[0] 0xFFFF gid[1]
subnet 0x0000000000000000 id 0x0000FFFFAC106404 state ACTIVE</font></tt><br><tt><font size=3>2017-09-02_07:46:01.384+0100: [I] VERBS RDMA discover
mlx4_1 port 2 transport IB link ETH NUMA node 0 pkey[0] 0xFFFF gid[0]
subnet 0xFE80000000000000 id 0x248A070001F981E1 state DOWN</font></tt><br><tt><font size=3>2017-09-02_07:46:01.385+0100: [I] VERBS RDMA discover
mlx4_0 port 1 transport IB link ETH NUMA node 0 pkey[0] 0xFFFF gid[0]
subnet 0xFE80000000000000 id 0x268A07FFFEF981C0 state ACTIVE</font></tt><br><tt><font size=3>2017-09-02_07:46:01.385+0100: [I] VERBS RDMA discover
mlx4_0 port 1 transport IB link ETH NUMA node 0 pkey[0] 0xFFFF gid[1]
subnet 0x0000000000000000 id 0x0000FFFFAC106404 state ACTIVE</font></tt><br><tt><font size=3>2017-09-02_07:46:01.386+0100: [I] VERBS RDMA discover
mlx4_0 port 2 transport IB link ETH NUMA node 0 pkey[0] 0xFFFF gid[0]
subnet 0xFE80000000000000 id 0x248A070001F981C1 state ACTIVE</font></tt><br><tt><font size=3>2017-09-02_07:46:01.386+0100: [I] VERBS RDMA discover
mlx4_0 port 2 transport IB link ETH NUMA node 0 pkey[0] 0xFFFF gid[1]
subnet 0x0000000000000000 id 0x0000FFFF0AC20011 state ACTIVE</font></tt><br><tt><font size=3>2017-09-02_07:46:01.387+0100: [I] VERBS RDMA parse
verbsPorts mlx4_0/1</font></tt><br><tt><font size=3>2017-09-02_07:46:01.390+0100: [W] VERBS RDMA parse
error verbsPort mlx4_0/1 ignored due to interface not found
for port 1 of device mlx4_0 with GID c081f9feff078a26. Please check if
the correct inet6 address for the corresponding IP network interface is
set</font></tt><br><tt><font size=3>2017-09-02_07:46:01.390+0100: [E] VERBS RDMA: rdma_get_cm_event
err -1</font></tt><br><tt><font size=3>2017-09-02_07:46:01.391+0100: [I] VERBS RDMA library
librdmacm.so unloaded.</font></tt><br><tt><font size=3>2017-09-02_07:46:01.391+0100: [E] VERBS RDMA failed
to start, no valid verbsPorts defined.</font></tt><br><br><br><font size=3>Anyone run into this before? I have another node imaged
the *exact* same way and no dice. Have tried a variety of drivers, cards,
etc, same result every time.</font><br><br><font size=3>Cheers,</font><br><font size=3>Barry</font><br><br><br><br><br><br><a href=http://pixitmedia.com/ target=_blank></a><font size=1 face="Arial"><br>This email is confidential in that it is intended for the exclusive attention
of the addressee(s) indicated. If you are not the intended recipient, this
email should not be read or disclosed to any other person. Please notify
the sender immediately and delete this email from your computer system.
Any opinions expressed are not necessarily those of the company from which
this email was sent and, whilst to the best of our knowledge no viruses
or defects exist, no responsibility can be accepted for any loss or damage
arising from its receipt or subsequent use of this email.</font><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br></font></tt><br><br><BR>