<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p>Hi,</p>
<p><br>
</p>
<p>You were right, ibv_devinfo -v doesn't return something if both card are connected. I didn't checked ibv_* tools, I supposed once IP stack and ibstat OK, the rest should work. I'm stupid
<span>😊</span></p>
<p><br>
</p>
<p><span></span>Anyway, once I disconnect one card, ibv_devinfo show me input but with both cards, I don't have any input except "device not found".</p>
<p>And what is weird here, it's that it work only when one card are connected, no matter the card (both are similar: model, firmware, revision, company)... Really strange, I will dig more about the issue.</p>
<p><br>
</p>
<p>Stupid and bad workaround: connected a dual port Infiniband. But production system doesn't wait..</p>
<p><br>
</p>
Thank for your help,
<div>Frank<br>
<br>
<div style="color: rgb(0, 0, 0);">
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Aaron Knister <aaron.knister@gmail.com><br>
<b>Sent:</b> Saturday, June 10, 2017 2:05 PM<br>
<b>To:</b> gpfsug main discussion list<br>
<b>Subject:</b> Re: [gpfsug-discuss] Infiniband: device mlx4_0 not found</font>
<div> </div>
</div>
<div>
<div>Out of curiosity could you send us the output of "ibv_devinfo -v"?</div>
<div id="AppleMailSignature"><br>
</div>
<div id="AppleMailSignature">-Aaron<br>
<br>
Sent from my iPhone</div>
<div><br>
On Jun 10, 2017, at 06:55, Frank Tower <<a href="mailto:frank.tower@outlook.com">frank.tower@outlook.com</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Arial,Helvetica,sans-serif">
<p>Hi everybody, </p>
<p><br>
</p>
<p>I don't get why one of our compute node cannot start GPFS over IB.</p>
<p><br>
</p>
<p>I have the following error: </p>
<p><br>
</p>
<p>[I] VERBS RDMA starting with verbsRdmaCm=no verbsRdmaSend=no verbsRdmaUseMultiCqThreads=yes verbsRdmaUseCompVectors=yes<br>
</p>
<p>[I] VERBS RDMA library libibverbs.so (version >= 1.1) loaded and initialized.<br>
</p>
<p>[I] VERBS RDMA verbsRdmasPerNode reduced from 1000 to 514 to match (nsdMaxWorkerThreads 512 + (nspdThreadsPerQueue 2 * nspdQueues 1)).</p>
<p>[I] VERBS RDMA parse verbsPorts mlx4_0/1</p>
<p>[W] VERBS RDMA parse error verbsPort mlx4_0/1 ignored due to device mlx4_0 not found</p>
<p>[I] VERBS RDMA library libibverbs.so unloaded.</p>
<p>[E] VERBS RDMA failed to start, no valid verbsPorts defined.</p>
<p><br>
</p>
<p><br>
</p>
<p>I'm using Centos 7.3, Kernel 3.10.0-514.21.1.el7.x86_64.</p>
<p><br>
</p>
<p>I have 2 infinibands card, both have an IP and working well.</p>
<p><br>
</p>
<p>[root@rdx110 ~]# ibstat -l</p>
<p>mlx4_0</p>
<p>mlx4_1</p>
<p>[root@rdx110 ~]#<br>
</p>
<p><br>
</p>
<p>I tried configuration with both card, and no one work with GPFS.</p>
<p><span style="font-size:12pt"><br>
</span></p>
<p><span style="font-size:12pt">I also tried with </span><span style="font-size:12pt">mlx4_0/1, but same problem.</span><br>
</p>
<p><span style="font-size:12pt"></span></p>
<p><span style="font-size:12pt"><br>
</span></p>
<p>Someone already have the issue ?</p>
<p><br>
</p>
<p>Kind Regards,</p>
<p>Frank</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<p></p>
</div>
</div>
</blockquote>
<blockquote type="cite">
<div><span>_______________________________________________</span><br>
<span>gpfsug-discuss mailing list</span><br>
<span>gpfsug-discuss at <a href="http://spectrumscale.org">spectrumscale.org</a></span><br>
<span><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></span><br>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</body>
</html>