[gpfsug-discuss] RoCE not playing ball

Gang Qiu gangqiu at cn.ibm.com
Wed Sep 20 06:58:15 BST 2017


 Do you set ip address for these adapters?

Refer to the description of verbsRdmaCm in ‘Command and Programming 
Reference':

If RDMA CM is enabled for a node, the node will only be able to establish 
RDMA connections
using RDMA CM to other nodes with verbsRdmaCm enabled. RDMA CM enablement 
requires
IPoIB (IP over InfiniBand) with an active IP address for each port. 
Although IPv6 must be
enabled, the GPFS implementation of RDMA CM does not currently support 
IPv6 addresses, so
an IPv4 address must be used.



Regards,
Gang Qiu

********************************************************************************************** 

IBM China Systems & Technology Lab
Tel:   86-10-82452193
Fax:   86-10-82452312
Moble: 132-6134-8284
Email:  gangqiu at cn.ibm.com
Address: Ring Bldg. No.28 Building, Zhong Guan Cun Software Park, No. 8 
Dong Bei Wang West Road, ShangDi, Haidian District, Beijing 100193, 
P.R.China
地址:北京市海淀区东北旺西路8号中关村软件园28号楼环宇大厦邮政编码:100193
**********************************************************************************************



From:   "Olaf Weiser" <olaf.weiser at de.ibm.com>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:   09/20/2017 01:01 PM
Subject:        Re: [gpfsug-discuss] RoCE not playing ball
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



is ib_read_bw  working  ?
just test it between the two nodes ... 




From:        Barry Evans <bevans at pixitmedia.com>
To:        gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:        09/20/2017 03:21 AM
Subject:        [gpfsug-discuss] RoCE not playing ball
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



Hi All,

Weirdness with a RoCE interface - verbs is not playing ball and is 
complaining about the inet6 address not matching up:

2017-09-02_07:46:01.376+0100: [I] VERBS RDMA starting with verbsRdmaCm=yes 
verbsRdmaSend=no verbsRdmaUseMultiCqThreads=yes 
verbsRdmaUseCompVectors=yes
2017-09-02_07:46:01.377+0100: [I] VERBS RDMA library librdmacm.so (version 
>= 1.1) loaded and initialized.
2017-09-02_07:46:01.377+0100: [I] VERBS RDMA verbsRdmasPerNode reduced 
from 1000 to 514 to match (nsdMaxWorkerThreads 512 + (nspdThreadsPerQueue 
2 * nspdQueues 1)).
2017-09-02_07:46:01.382+0100: [I] VERBS RDMA discover mlx4_1 port 1 
transport IB link ETH NUMA node  0 pkey[0] 0xFFFF gid[0] subnet 
0xFE80000000000000 id 0x268A07FFFEF981C0 state ACTIVE
2017-09-02_07:46:01.383+0100: [I] VERBS RDMA discover mlx4_1 port 1 
transport IB link ETH NUMA node  0 pkey[0] 0xFFFF gid[1] subnet 
0x0000000000000000 id 0x0000FFFFAC106404 state ACTIVE
2017-09-02_07:46:01.384+0100: [I] VERBS RDMA discover mlx4_1 port 2 
transport IB link ETH NUMA node  0 pkey[0] 0xFFFF gid[0] subnet 
0xFE80000000000000 id 0x248A070001F981E1 state DOWN
2017-09-02_07:46:01.385+0100: [I] VERBS RDMA discover mlx4_0 port 1 
transport IB link ETH NUMA node  0 pkey[0] 0xFFFF gid[0] subnet 
0xFE80000000000000 id 0x268A07FFFEF981C0 state ACTIVE
2017-09-02_07:46:01.385+0100: [I] VERBS RDMA discover mlx4_0 port 1 
transport IB link ETH NUMA node  0 pkey[0] 0xFFFF gid[1] subnet 
0x0000000000000000 id 0x0000FFFFAC106404 state ACTIVE
2017-09-02_07:46:01.386+0100: [I] VERBS RDMA discover mlx4_0 port 2 
transport IB link ETH NUMA node  0 pkey[0] 0xFFFF gid[0] subnet 
0xFE80000000000000 id 0x248A070001F981C1 state ACTIVE
2017-09-02_07:46:01.386+0100: [I] VERBS RDMA discover mlx4_0 port 2 
transport IB link ETH NUMA node  0 pkey[0] 0xFFFF gid[1] subnet 
0x0000000000000000 id 0x0000FFFF0AC20011 state ACTIVE
2017-09-02_07:46:01.387+0100: [I] VERBS RDMA parse verbsPorts mlx4_0/1
2017-09-02_07:46:01.390+0100: [W] VERBS RDMA parse error   verbsPort 
mlx4_0/1   ignored due to interface not found for port 1 of device mlx4_0 
with GID c081f9feff078a26. Please check if the correct inet6 address for 
the corresponding IP network interface is set
2017-09-02_07:46:01.390+0100: [E] VERBS RDMA: rdma_get_cm_event err -1
2017-09-02_07:46:01.391+0100: [I] VERBS RDMA library librdmacm.so 
unloaded.
2017-09-02_07:46:01.391+0100: [E] VERBS RDMA failed to start, no valid 
verbsPorts defined.


Anyone run into this before? I have another node imaged the *exact* same 
way and no dice. Have tried a variety of drivers, cards, etc, same result 
every time.

Cheers,
Barry






This email is confidential in that it is intended for the exclusive 
attention of the addressee(s) indicated. If you are not the intended 
recipient, this email should not be read or disclosed to any other person. 
Please notify the sender immediately and delete this email from your 
computer system. Any opinions expressed are not necessarily those of the 
company from which this email was sent and, whilst to the best of our 
knowledge no viruses or defects exist, no responsibility can be accepted 
for any loss or damage arising from its receipt or subsequent use of this 
email._______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=NCthMXTjizwdEVDBqoDwAfRswiFbdQVHRb4mzseFLEM&m=u155tVFn5u91gqIsTXSOSVvpbR7GQRPoVpviUDH73R0&s=63nY5ozD8mej1jefNBZjLGCkNOFD9-swr-lc7CRPbrM&e= 





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170920/54161a6e/attachment-0002.htm>


More information about the gpfsug-discuss mailing list