[gpfsug-discuss] RoCE not playing ball

[Barry Evans]

It has worked, yes, and while the issue has been present. At the moment
it's not working, but I'm not entirely surprised with the amount it's been
poked at.

Cheers,
Barry

On Tue, Sep 19, 2017 at 10:00 PM, Olaf Weiser <olaf.weiser at de.ibm.com>
wrote:

> is ib_read_bw  working  ?
> just test it between the two nodes ...
>
>
>
>
> From:        Barry Evans <bevans at pixitmedia.com>
> To:        gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
> Date:        09/20/2017 03:21 AM
> Subject:        [gpfsug-discuss] RoCE not playing ball
> Sent by:        gpfsug-discuss-bounces at spectrumscale.org
> ------------------------------
>
>
>
> Hi All,
>
> Weirdness with a RoCE interface - verbs is not playing ball and is
> complaining about the inet6 address not matching up:
>
> 2017-09-02_07:46:01.376+0100: [I] VERBS RDMA starting with verbsRdmaCm=yes
> verbsRdmaSend=no verbsRdmaUseMultiCqThreads=yes verbsRdmaUseCompVectors=yes
> 2017-09-02_07:46:01.377+0100: [I] VERBS RDMA library librdmacm.so (version
> >= 1.1) loaded and initialized.
> 2017-09-02_07:46:01.377+0100: [I] VERBS RDMA verbsRdmasPerNode reduced
> from 1000 to 514 to match (nsdMaxWorkerThreads 512 + (nspdThreadsPerQueue 2
> * nspdQueues 1)).
> 2017-09-02_07:46:01.382+0100: [I] VERBS RDMA discover mlx4_1 port 1
> transport IB link ETH NUMA node  0 pkey[0] 0xFFFF gid[0] subnet
> 0xFE80000000000000 id 0x268A07FFFEF981C0 state ACTIVE
> 2017-09-02_07:46:01.383+0100: [I] VERBS RDMA discover mlx4_1 port 1
> transport IB link ETH NUMA node  0 pkey[0] 0xFFFF gid[1] subnet
> 0x0000000000000000 id 0x0000FFFFAC106404 state ACTIVE
> 2017-09-02_07:46:01.384+0100: [I] VERBS RDMA discover mlx4_1 port 2
> transport IB link ETH NUMA node  0 pkey[0] 0xFFFF gid[0] subnet
> 0xFE80000000000000 id 0x248A070001F981E1 state DOWN
> 2017-09-02_07:46:01.385+0100: [I] VERBS RDMA discover mlx4_0 port 1
> transport IB link ETH NUMA node  0 pkey[0] 0xFFFF gid[0] subnet
> 0xFE80000000000000 id 0x268A07FFFEF981C0 state ACTIVE
> 2017-09-02_07:46:01.385+0100: [I] VERBS RDMA discover mlx4_0 port 1
> transport IB link ETH NUMA node  0 pkey[0] 0xFFFF gid[1] subnet
> 0x0000000000000000 id 0x0000FFFFAC106404 state ACTIVE
> 2017-09-02_07:46:01.386+0100: [I] VERBS RDMA discover mlx4_0 port 2
> transport IB link ETH NUMA node  0 pkey[0] 0xFFFF gid[0] subnet
> 0xFE80000000000000 id 0x248A070001F981C1 state ACTIVE
> 2017-09-02_07:46:01.386+0100: [I] VERBS RDMA discover mlx4_0 port 2
> transport IB link ETH NUMA node  0 pkey[0] 0xFFFF gid[1] subnet
> 0x0000000000000000 id 0x0000FFFF0AC20011 state ACTIVE
> 2017-09-02_07:46:01.387+0100: [I] VERBS RDMA parse verbsPorts mlx4_0/1
> 2017-09-02_07:46:01.390+0100: [W] VERBS RDMA parse error   verbsPort
> mlx4_0/1   ignored due to interface not found for port 1 of device mlx4_0
> with GID c081f9feff078a26. Please check if the correct inet6 address for
> the corresponding IP network interface is set
> 2017-09-02_07:46:01.390+0100: [E] VERBS RDMA: rdma_get_cm_event err -1
> 2017-09-02_07:46:01.391+0100: [I] VERBS RDMA library librdmacm.so unloaded.
> 2017-09-02_07:46:01.391+0100: [E] VERBS RDMA failed to start, no valid
> verbsPorts defined.
>
>
> Anyone run into this before? I have another node imaged the *exact* same
> way and no dice. Have tried a variety of drivers, cards, etc, same result
> every time.
>
> Cheers,
> Barry
>
>
>
>
>
> <http://pixitmedia.com/>
> This email is confidential in that it is intended for the exclusive
> attention of the addressee(s) indicated. If you are not the intended
> recipient, this email should not be read or disclosed to any other person.
> Please notify the sender immediately and delete this email from your
> computer system. Any opinions expressed are not necessarily those of the
> company from which this email was sent and, whilst to the best of our
> knowledge no viruses or defects exist, no responsibility can be accepted
> for any loss or damage arising from its receipt or subsequent use of this
> email._______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>

-- 
 <http://pixitmedia.com>
This email is confidential in that it is intended for the exclusive 
attention of the addressee(s) indicated. If you are not the intended 
recipient, this email should not be read or disclosed to any other person. 
Please notify the sender immediately and delete this email from your 
computer system. Any opinions expressed are not necessarily those of the 
company from which this email was sent and, whilst to the best of our 
knowledge no viruses or defects exist, no responsibility can be accepted 
for any loss or damage arising from its receipt or subsequent use of this 
email.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170920/7e91a2b9/attachment-0002.htm>