[gpfsug-discuss] rmda errors scatter thru gpfs logs

Brian Marshall mimarsh2 at vt.edu
Wed Jan 18 16:58:24 GMT 2017


As background, we recently upgraded GPFS from 4.2.0 to 4.2.1  and updated
the Mellanox OFED on our compute cluster to allow it to move from CentOS
7.1 to 7.2

We do some transient warnings from the Mellanox switch gear about various
port counters that we are tracking down with them.

Jobs and filesystem seem stable, but the logs are concerning.

On Wed, Jan 18, 2017 at 10:22 AM, Aaron Knister <aaron.s.knister at nasa.gov>
wrote:

> I'm curious about this too. We see these messages sometimes when things
> have gone horribly wrong but also sometimes during recovery events. Here's
> a recent one:
>
> loremds20 (manager/nsd node):
> Mon Jan 16 14:19:02.048 2017: [E] VERBS RDMA rdma read error
> IBV_WC_REM_ACCESS_ERR to 10.101.11.6 (lorej006) on mlx5_0 port 1 fabnum 3
> vendor_err 136
> Mon Jan 16 14:19:02.049 2017: [E] VERBS RDMA closed connection to
> 10.101.11.6 (lorej006) on mlx5_0 port 1 fabnum 3 due to RDMA read error
> IBV_WC_REM_ACCESS_ERR index 11
>
> lorej006 (client):
> Mon Jan 16 14:19:01.990 2017: [N] VERBS RDMA closed connection to
> 10.101.53.18 (loremds18) on mlx5_0 port 1 fabnum 3 index 2
> Mon Jan 16 14:19:01.995 2017: [N] VERBS RDMA closed connection to
> 10.101.53.19 (loremds19) on mlx5_0 port 1 fabnum 3 index 0
> Mon Jan 16 14:19:01.997 2017: [I] Recovering nodes: 10.101.53.18
> 10.101.53.19
> Mon Jan 16 14:19:02.047 2017: [W] VERBS RDMA async event
> IBV_EVENT_QP_ACCESS_ERR on mlx5_0 qp 0x7fffe550f1c8.
> Mon Jan 16 14:19:02.051 2017: [E] VERBS RDMA closed connection to
> 10.101.53.20 (loremds20) on mlx5_0 port 1 fabnum 3 error 733 index 1
> Mon Jan 16 14:19:02.071 2017: [I] Recovered 2 nodes for file system tnb32.
> Mon Jan 16 14:19:02.140 2017: [I] VERBS RDMA connecting to 10.101.53.20
> (loremds20) on mlx5_0 port 1 fabnum 3 index 0
> Mon Jan 16 14:19:02.160 2017: [I] VERBS RDMA connected to 10.101.53.20
> (loremds20) on mlx5_0 port 1 fabnum 3 sl 0 index 0
>
> I had just shut down loremds18 and loremds19 so there was certainly
> recovery taking place and during that time is when the error seems to have
> occurred.
>
> I looked up the meaning of IBV_WC_REM_ACCESS_ERR here (
> http://www.rdmamojo.com/2013/02/15/ibv_poll_cq/) and see this:
>
> IBV_WC_REM_ACCESS_ERR (10) - Remote Access Error: a protection error
> occurred on a remote data buffer to be read by an RDMA Read, written by an
> RDMA Write or accessed by an atomic operation. This error is reported only
> on RDMA operations or atomic operations. Relevant for RC QPs.
>
> my take on it during recovery it seems like one end of the connection more
> or less hanging up on the other end (e.g. Connection reset by peer
> /ECONNRESET).
>
> But like I said at the start, we also see this when there something has
> gone awfully wrong.
>
> -Aaron
>
> On 1/18/17 3:59 AM, Simon Thompson (Research Computing - IT Services)
> wrote:
>
>> I'd be inclined to look at something like:
>>
>> ibqueryerrors -s
>> PortXmitWait,LinkDownedCounter,PortXmitDiscards,PortRcvRemot
>> ePhysicalErrors
>> -c
>>
>> And see if you have a high number of symbol errors, might be a cable
>> needs replugging or replacing.
>>
>> Simon
>>
>> From: <gpfsug-discuss-bounces at spectrumscale.org
>> <mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of "J. Eric
>> Wonderley" <eric.wonderley at vt.edu <mailto:eric.wonderley at vt.edu>>
>> Reply-To: "gpfsug-discuss at spectrumscale.org
>> <mailto:gpfsug-discuss at spectrumscale.org>"
>> <gpfsug-discuss at spectrumscale.org <mailto:gpfsug-discuss at spectru
>> mscale.org>>
>> Date: Tuesday, 17 January 2017 at 21:16
>> To: "gpfsug-discuss at spectrumscale.org
>> <mailto:gpfsug-discuss at spectrumscale.org>"
>> <gpfsug-discuss at spectrumscale.org <mailto:gpfsug-discuss at spectru
>> mscale.org>>
>> Subject: [gpfsug-discuss] rmda errors scatter thru gpfs logs
>>
>> I have messages like these frequent my logs:
>> Tue Jan 17 11:25:49.731 2017: [E] VERBS RDMA rdma write error
>> IBV_WC_REM_ACCESS_ERR to 10.51.10.5 (cl005) on mlx5_0 port 1 fabnum 0
>> vendor_err 136
>> Tue Jan 17 11:25:49.732 2017: [E] VERBS RDMA closed connection to
>> 10.51.10.5 (cl005) on mlx5_0 port 1 fabnum 0 due to RDMA write error
>> IBV_WC_REM_ACCESS_ERR index 23
>>
>> Any ideas on cause..?
>>
>>
>>
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>
>>
> --
> Aaron Knister
> NASA Center for Climate Simulation (Code 606.2)
> Goddard Space Flight Center
> (301) 286-2776
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170118/a1976845/attachment-0002.htm>


More information about the gpfsug-discuss mailing list