<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Lucida Grande";}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
p.null, li.null, div.null
{mso-style-name:null;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.null1
{mso-style-name:null1;}
span.E-postmall20
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.E-postmall21
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Thanks!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">I’ve double checked the selinux state, and it is disabled on both all the ESS nodes and on all the client nodes.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">mmfsd is running as root on all nodes as well.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">It seem a bit strange that this would be an issue of permissions though. I forgot to state this in my original question, but the issue
comes and goes, and can affect some clients while not affecting others at the same time, and which clients are affected at any given time changes as well.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Just a thought; How does infiniband queue pairs react to time skew between nodes?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">For future reference, where did you find the specification of ibv_create_qp error 13?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">I must have been looking in all the wrong places, because I’ve been unable to find the meaning of this error.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Andreas<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="color:black">_____________________________________________<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Lucida Grande";color:black"><img width="121" height="65" style="width:1.2604in;height:.677in" id="Bild_x0020_2" src="cid:image001.png@01D36DD7.DA979600" alt="cid:part1.08040705.03090509@maxiv.lu.se"><br>
</span><b><span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:black"> <br>
Andreas Mattsson<br>
</span></b><span style="font-size:8.0pt;font-family:"Calibri",sans-serif;color:black">Systems Engineer</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:8.0pt;font-family:"Calibri",sans-serif;color:black">MAX IV Laboratory<br>
Lund University<br>
</span><span style="font-size:8.5pt;color:black">P.O. Box 118, SE-221 00 Lund, Sweden<br>
Visiting address: Fotongatan 2, 225 94 Lund<br>
</span><span style="font-size:8.0pt;font-family:"Calibri",sans-serif;color:black">Mobile: +46 706 64 95 44<br>
</span><a href="mailto:andreas.mattsson@maxiv.se"><span style="font-size:8.0pt;font-family:"Calibri",sans-serif">andreas.mattsson@maxiv.se</span></a><span style="font-size:8.0pt;font-family:"Calibri",sans-serif;color:black"><br>
</span><a href="http://www.maxiv.se/"><span style="font-size:8.0pt;font-family:"Calibri",sans-serif">www.maxiv.se</span></a><span style="font-size:8.0pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-left:36.0pt"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Från:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> gpfsug-discuss-bounces@spectrumscale.org [mailto:gpfsug-discuss-bounces@spectrumscale.org]
<b>För </b>Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]<br>
<b>Skickat:</b> den 5 december 2017 14:24<br>
<b>Till:</b> gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
<b>Ämne:</b> Re: [gpfsug-discuss] Infiniband connection rejected, ibv_create_qp err 13<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:36.0pt">
<br>
<br>
Looks like 13 is EPERM which means apparently permissions didn’t exist to create the QP of the desired type which is odd since mmfsd runs as root. Is there any remote chance SELinux is enabled (e.g. sestatus)? Although I’d think mmfsd would run unconfined in
the default policy, but maybe it didn’t transition correctly. <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">On December 5, 2017 at 08:16:49 EST, Andreas Mattsson <<a href="mailto:andreas.mattsson@maxiv.lu.se">andreas.mattsson@maxiv.lu.se</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid windowtext 1.0pt;padding:0cm 0cm 0cm 8.0pt;margin-left:0cm;margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<p class="null" style="margin-left:36.0pt"><span class="null1">Hi.</span><o:p></o:p></p>
<p class="null" style="margin-left:36.0pt"><span class="null1"> </span><o:p></o:p></p>
<p class="null" style="margin-left:36.0pt">Have anyone here experienced having VERBS RDMA connection request rejects on Scale NSD servers with the error message “ibv_create_qp err 13”?<o:p></o:p></p>
<p class="null" style="margin-left:36.0pt">I’m having issues with this on a IBM ESS system.<o:p></o:p></p>
<p class="null" style="margin-left:36.0pt"> <o:p></o:p></p>
<p class="null" style="margin-left:36.0pt">The error mostly affects only one of the two GSSIO-nodes, and moves with the node even if I put all four of the infiniband links on the same infiniband switch as the working node is connected to.<o:p></o:p></p>
<p class="null" style="margin-left:36.0pt">The issue affects client nodes in different blade-chassis, going through different Infiniband swithes and cables, and also non-blade nodes running a slightly different os-setup and different infiniband HCAs.<o:p></o:p></p>
<p class="null" style="margin-left:36.0pt">MPI-jobs on the client nodes can communicate over the infiniband fabric without issues.<o:p></o:p></p>
<p class="null" style="margin-left:36.0pt">Upgrading all switches and HCAs to the latest firmware and making sure that client nodes have the same OFED-version as the ESS has had no impact on the issue.<o:p></o:p></p>
<p class="null" style="margin-left:36.0pt">When the issue is there, I can still do ibping between the nodes, ibroute gives me a working and correct path between the nodes that get connection rejects, and if I set up IPoIB, ip traffic works on the afflicted
interfaces.<o:p></o:p></p>
<p class="null" style="margin-left:36.0pt"> <o:p></o:p></p>
<p class="null" style="margin-left:36.0pt">I have opened a PMR with IBM on the issue, so asking here is a parallel track for trying to find a solution to this.<o:p></o:p></p>
<p class="null" style="margin-left:36.0pt"> <o:p></o:p></p>
<p class="null" style="margin-left:36.0pt">Any help or suggestions is appreciated.<o:p></o:p></p>
<p class="null" style="margin-left:36.0pt">Regards,<o:p></o:p></p>
<p class="null" style="margin-left:36.0pt">Andreas Mattsson<o:p></o:p></p>
<p class="null" style="margin-left:36.0pt"><span class="null1"><span style="color:black">_____________________________________________</span></span><o:p></o:p></p>
<p class="null" style="margin-left:36.0pt"><span style="font-family:"Lucida Grande";color:black"><img border="0" width="121" height="65" style="width:1.2604in;height:.677in" id="_x0036_b0a11de-89d6-4863-b8e8-188f1e1c5a0c" src="cid:image001.png@01D36DD7.DA979600" alt="mid:d8d07f7e01ec4fcca5ae124f40c2d457@maxiv.lu.se/part1.08040705.03090509@maxiv.lu.se"></span><span style="font-family:"Lucida Grande";color:black"><br>
</span><span class="null1"><b><span style="color:black"> </span></b></span><b><span style="color:black"><br>
<span class="null1">Andreas Mattsson</span><br>
</span></b><span class="null1"><span style="color:black">Systems Engineer</span></span><o:p></o:p></p>
<p class="null" style="margin-left:36.0pt"><span class="null1"><span style="color:black"> </span></span><o:p></o:p></p>
<p class="null" style="mso-margin-top-alt:5.0pt;margin-right:0cm;margin-bottom:12.0pt;margin-left:36.0pt">
<span class="null1"><span style="color:black">MAX IV Laboratory</span></span><span style="color:black"><br>
<span class="null1">Lund University</span><br>
<span class="null1">P.O. Box 118, SE-221 00 Lund, Sweden</span><br>
<span class="null1">Visiting address: Fotongatan 2, 225 94 Lund</span><br>
<span class="null1">Mobile: +46 706 64 95 44</span><br>
</span><a href="mailto:andreas.mattsson@maxiv.se"><span class="null1">andreas.mattsson@maxiv.se</span></a><span style="color:black"><br>
</span><a href="http://www.maxiv.se/"><span class="null1">www.maxiv.se</span></a><o:p></o:p></p>
<p class="null" style="margin-left:36.0pt"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>