<font size=2 face="sans-serif"> </font><br><br><br><font size=2 face="sans-serif">Do you set ip address for these adapters?</font><br><br><font size=2 face="sans-serif">Refer to the description of verbsRdmaCm
in ‘Command and Programming Reference':</font><br><br><font size=2 face="Palatino-Roman">If RDMA CM is enabled for a node,
the node will only be able to establish RDMA connections</font><br><font size=2 face="Palatino-Roman">using RDMA CM to other nodes with
</font><font size=2 face="Palatino-Bold"><b>verbsRdmaCm </b></font><font size=2 face="Palatino-Roman">enabled.
RDMA CM enablement requires</font><br><font size=2 face="Palatino-Roman">IPoIB (IP over InfiniBand) with
an active IP address for each port. Although IPv6 must be</font><br><font size=2 face="Palatino-Roman">enabled, the GPFS implementation
of RDMA CM does not currently support IPv6 addresses, so</font><br><font size=2 face="Palatino-Roman">an IPv4 address must be used.</font><br><font size=2 face="sans-serif"><br><br><br>Regards,<br>Gang Qiu<br><br>**********************************************************************************************
<br>IBM China Systems & Technology Lab<br>Tel: 86-10-82452193<br>Fax: 86-10-82452312<br>Moble: 132-6134-8284<br>Email: gangqiu@cn.ibm.com<br>Address: Ring Bldg. No.28 Building, Zhong Guan Cun Software Park, No. 8
Dong Bei Wang West Road, ShangDi, Haidian District, Beijing 100193, P.R.China<br>地址：北京市海淀区东北旺西路8号中关村软件园28号楼环宇大厦邮政编码：100193<br>**********************************************************************************************</font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">"Olaf Weiser"
<olaf.weiser@de.ibm.com></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug main discussion
list <gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">09/20/2017 01:01 PM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
RoCE not playing ball</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><font size=2 face="sans-serif">is ib_read_bw working ?<br>just test it between the two nodes ... </font><font size=3><br><br><br><br></font><font size=1 color=#5f5f5f face="sans-serif"><br>From: </font><font size=1 face="sans-serif">Barry
Evans <bevans@pixitmedia.com></font><font size=1 color=#5f5f5f face="sans-serif"><br>To: </font><font size=1 face="sans-serif">gpfsug
main discussion list <gpfsug-discuss@spectrumscale.org></font><font size=1 color=#5f5f5f face="sans-serif"><br>Date: </font><font size=1 face="sans-serif">09/20/2017
03:21 AM</font><font size=1 color=#5f5f5f face="sans-serif"><br>Subject: </font><font size=1 face="sans-serif">[gpfsug-discuss]
RoCE not playing ball</font><font size=1 color=#5f5f5f face="sans-serif"><br>Sent by: </font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><font size=3><br></font><hr noshade><font size=3><br><br><br>Hi All,<br><br>Weirdness with a RoCE interface - verbs is not playing ball and is complaining
about the inet6 address not matching up:<br></font><tt><font size=3><br>2017-09-02_07:46:01.376+0100: [I] VERBS RDMA starting with verbsRdmaCm=yes
verbsRdmaSend=no verbsRdmaUseMultiCqThreads=yes verbsRdmaUseCompVectors=yes<br>2017-09-02_07:46:01.377+0100: [I] VERBS RDMA library librdmacm.so (version
>= 1.1) loaded and initialized.<br>2017-09-02_07:46:01.377+0100: [I] VERBS RDMA verbsRdmasPerNode reduced
from 1000 to 514 to match (nsdMaxWorkerThreads 512 + (nspdThreadsPerQueue
2 * nspdQueues 1)).<br>2017-09-02_07:46:01.382+0100: [I] VERBS RDMA discover mlx4_1 port 1 transport
IB link ETH NUMA node 0 pkey[0] 0xFFFF gid[0] subnet 0xFE80000000000000
id 0x268A07FFFEF981C0 state ACTIVE<br>2017-09-02_07:46:01.383+0100: [I] VERBS RDMA discover mlx4_1 port 1 transport
IB link ETH NUMA node 0 pkey[0] 0xFFFF gid[1] subnet 0x0000000000000000
id 0x0000FFFFAC106404 state ACTIVE<br>2017-09-02_07:46:01.384+0100: [I] VERBS RDMA discover mlx4_1 port 2 transport
IB link ETH NUMA node 0 pkey[0] 0xFFFF gid[0] subnet 0xFE80000000000000
id 0x248A070001F981E1 state DOWN<br>2017-09-02_07:46:01.385+0100: [I] VERBS RDMA discover mlx4_0 port 1 transport
IB link ETH NUMA node 0 pkey[0] 0xFFFF gid[0] subnet 0xFE80000000000000
id 0x268A07FFFEF981C0 state ACTIVE<br>2017-09-02_07:46:01.385+0100: [I] VERBS RDMA discover mlx4_0 port 1 transport
IB link ETH NUMA node 0 pkey[0] 0xFFFF gid[1] subnet 0x0000000000000000
id 0x0000FFFFAC106404 state ACTIVE<br>2017-09-02_07:46:01.386+0100: [I] VERBS RDMA discover mlx4_0 port 2 transport
IB link ETH NUMA node 0 pkey[0] 0xFFFF gid[0] subnet 0xFE80000000000000
id 0x248A070001F981C1 state ACTIVE<br>2017-09-02_07:46:01.386+0100: [I] VERBS RDMA discover mlx4_0 port 2 transport
IB link ETH NUMA node 0 pkey[0] 0xFFFF gid[1] subnet 0x0000000000000000
id 0x0000FFFF0AC20011 state ACTIVE<br>2017-09-02_07:46:01.387+0100: [I] VERBS RDMA parse verbsPorts mlx4_0/1<br>2017-09-02_07:46:01.390+0100: [W] VERBS RDMA parse error verbsPort
mlx4_0/1 ignored due to interface not found for port 1 of device
mlx4_0 with GID c081f9feff078a26. Please check if the correct inet6 address
for the corresponding IP network interface is set<br>2017-09-02_07:46:01.390+0100: [E] VERBS RDMA: rdma_get_cm_event err -1<br>2017-09-02_07:46:01.391+0100: [I] VERBS RDMA library librdmacm.so unloaded.<br>2017-09-02_07:46:01.391+0100: [E] VERBS RDMA failed to start, no valid
verbsPorts defined.</font></tt><font size=3><br><br><br>Anyone run into this before? I have another node imaged the *exact* same
way and no dice. Have tried a variety of drivers, cards, etc, same result
every time.<br><br>Cheers,<br>Barry<br><br><br><br><br></font><font size=3 color=blue><u><br></u></font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__pixitmedia.com_&d=DwMFAg&c=jf_iaSHvJObTbx-siA1ZOg&r=NCthMXTjizwdEVDBqoDwAfRswiFbdQVHRb4mzseFLEM&m=u155tVFn5u91gqIsTXSOSVvpbR7GQRPoVpviUDH73R0&s=Cuqio6URV5SlrAbObWAcbPH081odzTfHQKwjrXCoG60&e=" target=_blank></a><font size=1 face="Arial"><br>This email is confidential in that it is intended for the exclusive attention
of the addressee(s) indicated. If you are not the intended recipient, this
email should not be read or disclosed to any other person. Please notify
the sender immediately and delete this email from your computer system.
Any opinions expressed are not necessarily those of the company from which
this email was sent and, whilst to the best of our knowledge no viruses
or defects exist, no responsibility can be accepted for any loss or damage
arising from its receipt or subsequent use of this email.</font><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org</font></tt><font size=3 color=blue><u><br></u></font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFAg&c=jf_iaSHvJObTbx-siA1ZOg&r=NCthMXTjizwdEVDBqoDwAfRswiFbdQVHRb4mzseFLEM&m=u155tVFn5u91gqIsTXSOSVvpbR7GQRPoVpviUDH73R0&s=63nY5ozD8mej1jefNBZjLGCkNOFD9-swr-lc7CRPbrM&e="><tt><font size=2 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></tt></a><font size=3><br><br><br></font><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=NCthMXTjizwdEVDBqoDwAfRswiFbdQVHRb4mzseFLEM&m=u155tVFn5u91gqIsTXSOSVvpbR7GQRPoVpviUDH73R0&s=63nY5ozD8mej1jefNBZjLGCkNOFD9-swr-lc7CRPbrM&e="><tt><font size=2>https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=NCthMXTjizwdEVDBqoDwAfRswiFbdQVHRb4mzseFLEM&m=u155tVFn5u91gqIsTXSOSVvpbR7GQRPoVpviUDH73R0&s=63nY5ozD8mej1jefNBZjLGCkNOFD9-swr-lc7CRPbrM&e=</font></tt></a><tt><font size=2><br></font></tt><br><br><BR>