<div dir="ltr">It has worked, yes, and while the issue has been present. At the moment it's not working, but I'm not entirely surprised with the amount it's been poked at.<div><br></div><div>Cheers,</div><div>Barry</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Sep 19, 2017 at 10:00 PM, Olaf Weiser <span dir="ltr"><<a href="mailto:olaf.weiser@de.ibm.com" target="_blank">olaf.weiser@de.ibm.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><font size="2" face="sans-serif">is ib_read_bw working ?</font><br><font size="2" face="sans-serif">just test it between the two nodes ...
</font><br><br><br><br><br><font size="1" color="#5f5f5f" face="sans-serif">From:
</font><font size="1" face="sans-serif">Barry Evans <<a href="mailto:bevans@pixitmedia.com" target="_blank">bevans@pixitmedia.com</a>></font><br><font size="1" color="#5f5f5f" face="sans-serif">To:
</font><font size="1" face="sans-serif">gpfsug main discussion
list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.<wbr>org</a>></font><br><font size="1" color="#5f5f5f" face="sans-serif">Date:
</font><font size="1" face="sans-serif">09/20/2017 03:21 AM</font><br><font size="1" color="#5f5f5f" face="sans-serif">Subject:
</font><font size="1" face="sans-serif">[gpfsug-discuss]
RoCE not playing ball</font><br><font size="1" color="#5f5f5f" face="sans-serif">Sent by:
</font><font size="1" face="sans-serif"><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@<wbr>spectrumscale.org</a></font><br><hr noshade><br><br><br><font size="3">Hi All,</font><br><br><font size="3">Weirdness with a RoCE interface - verbs is not playing
ball and is complaining about the inet6 address not matching up:</font><br><br><tt><font size="3">2017-09-02_07:46:01.376+0100: [I] VERBS RDMA starting
with verbsRdmaCm=yes verbsRdmaSend=no verbsRdmaUseMultiCqThreads=yes verbsRdmaUseCompVectors=yes</font></tt><br><tt><font size="3">2017-09-02_07:46:01.377+0100: [I] VERBS RDMA library
librdmacm.so (version >= 1.1) loaded and initialized.</font></tt><br><tt><font size="3">2017-09-02_07:46:01.377+0100: [I] VERBS RDMA verbsRdmasPerNode
reduced from 1000 to 514 to match (nsdMaxWorkerThreads 512 + (nspdThreadsPerQueue
2 * nspdQueues 1)).</font></tt><br><tt><font size="3">2017-09-02_07:46:01.382+0100: [I] VERBS RDMA discover
mlx4_1 port 1 transport IB link ETH NUMA node 0 pkey[0] 0xFFFF gid[0]
subnet 0xFE80000000000000 id 0x268A07FFFEF981C0 state ACTIVE</font></tt><br><tt><font size="3">2017-09-02_07:46:01.383+0100: [I] VERBS RDMA discover
mlx4_1 port 1 transport IB link ETH NUMA node 0 pkey[0] 0xFFFF gid[1]
subnet 0x0000000000000000 id 0x0000FFFFAC106404 state ACTIVE</font></tt><br><tt><font size="3">2017-09-02_07:46:01.384+0100: [I] VERBS RDMA discover
mlx4_1 port 2 transport IB link ETH NUMA node 0 pkey[0] 0xFFFF gid[0]
subnet 0xFE80000000000000 id 0x248A070001F981E1 state DOWN</font></tt><br><tt><font size="3">2017-09-02_07:46:01.385+0100: [I] VERBS RDMA discover
mlx4_0 port 1 transport IB link ETH NUMA node 0 pkey[0] 0xFFFF gid[0]
subnet 0xFE80000000000000 id 0x268A07FFFEF981C0 state ACTIVE</font></tt><br><tt><font size="3">2017-09-02_07:46:01.385+0100: [I] VERBS RDMA discover
mlx4_0 port 1 transport IB link ETH NUMA node 0 pkey[0] 0xFFFF gid[1]
subnet 0x0000000000000000 id 0x0000FFFFAC106404 state ACTIVE</font></tt><br><tt><font size="3">2017-09-02_07:46:01.386+0100: [I] VERBS RDMA discover
mlx4_0 port 2 transport IB link ETH NUMA node 0 pkey[0] 0xFFFF gid[0]
subnet 0xFE80000000000000 id 0x248A070001F981C1 state ACTIVE</font></tt><br><tt><font size="3">2017-09-02_07:46:01.386+0100: [I] VERBS RDMA discover
mlx4_0 port 2 transport IB link ETH NUMA node 0 pkey[0] 0xFFFF gid[1]
subnet 0x0000000000000000 id 0x0000FFFF0AC20011 state ACTIVE</font></tt><br><tt><font size="3">2017-09-02_07:46:01.387+0100: [I] VERBS RDMA parse
verbsPorts mlx4_0/1</font></tt><br><tt><font size="3">2017-09-02_07:46:01.390+0100: [W] VERBS RDMA parse
error verbsPort mlx4_0/1 ignored due to interface not found
for port 1 of device mlx4_0 with GID c081f9feff078a26. Please check if
the correct inet6 address for the corresponding IP network interface is
set</font></tt><br><tt><font size="3">2017-09-02_07:46:01.390+0100: [E] VERBS RDMA: rdma_get_cm_event
err -1</font></tt><br><tt><font size="3">2017-09-02_07:46:01.391+0100: [I] VERBS RDMA library
librdmacm.so unloaded.</font></tt><br><tt><font size="3">2017-09-02_07:46:01.391+0100: [E] VERBS RDMA failed
to start, no valid verbsPorts defined.</font></tt><br><br><br><font size="3">Anyone run into this before? I have another node imaged
the *exact* same way and no dice. Have tried a variety of drivers, cards,
etc, same result every time.</font><br><br><font size="3">Cheers,</font><br><font size="3">Barry</font><br><br><br><br><br><br><a href="http://pixitmedia.com/" target="_blank"></a><font size="1" face="Arial"><br>This email is confidential in that it is intended for the exclusive attention
of the addressee(s) indicated. If you are not the intended recipient, this
email should not be read or disclosed to any other person. Please notify
the sender immediately and delete this email from your computer system.
Any opinions expressed are not necessarily those of the company from which
this email was sent and, whilst to the best of our knowledge no viruses
or defects exist, no responsibility can be accepted for any loss or damage
arising from its receipt or subsequent use of this email.</font><tt><font size="2">________________________<wbr>_______________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><tt><font size="2">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</font></tt></a><tt><font size="2"><br></font></tt><br><br><br>
<br>______________________________<wbr>_________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</a><br>
<br></blockquote></div><br></div>
<br>
<div><a href="http://pixitmedia.com" target="_blank"><img src="http://pixitmedia.com/sig/IBC_emailsig17.png"></a><br><font face="Arial, Helvetica, sans-serif" size="1">This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email.</font></div>