<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Arial;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Mangal;
panose-1:2 4 5 3 5 2 3 3 2 2;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:Calibri;}
h1
{mso-style-priority:9;
mso-style-link:"Heading 1 Char";
margin-top:12.0pt;
margin-right:0cm;
margin-bottom:0cm;
margin-left:0cm;
margin-bottom:.0001pt;
page-break-after:avoid;
font-size:16.0pt;
font-family:"Calibri Light";
color:#2F5496;
font-weight:normal;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:Calibri;
color:windowtext;}
span.Heading1Char
{mso-style-name:"Heading 1 Char";
mso-style-priority:9;
mso-style-link:"Heading 1";
font-family:"Calibri Light";
color:#2F5496;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:Calibri;}
@page WordSection1
{size:595.0pt 842.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Hello Team,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We are facing a lot of messages waiters related to
<b><a href="https://www.mail-archive.com/search?l=gpfsug-discuss@spectrumscale.org&q=subject:%22Re%5C%3A+%5C%5Bgpfsug%5C-discuss%5C%5D+waiting+for+conn+rdmas+%3C+conn+maxrdmas%22&o=newest">waiting for conn rdmas < conn maxrdmas</a><o:p></o:p></b></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Is there some recommended settings to resolve this issue.?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Our config for RDMA is as follows for 140 nodes(32 cores each)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">VERBS RDMA Configuration:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Status : started<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Start time : Thu
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Stats reset time : Thu
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Dump time : Sun<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> mmfs verbsRdma : enable<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> mmfs verbsRdmaCm : disable<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> mmfs verbsPorts : mlx4_0/1 mlx4_0/2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> mmfs verbsRdmasPerNode : 3200<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> mmfs verbsRdmasPerNode (max) : 3200<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> mmfs verbsRdmasPerNodeOptimize : yes<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> mmfs verbsRdmasPerConnection : 16<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> mmfs verbsRdmasPerConnection (max) : 16<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> mmfs verbsRdmaMinBytes : 16384<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> mmfs verbsRdmaRoCEToS : -1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> mmfs verbsRdmaQpRtrMinRnrTimer : 18<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> mmfs verbsRdmaQpRtrPathMtu : 2048<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> mmfs verbsRdmaQpRtrSl : 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> mmfs verbsRdmaQpRtrSlDynamic : no<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> mmfs verbsRdmaQpRtrSlDynamicTimeout : 10<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> mmfs verbsRdmaQpRtsRnrRetry : 6<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> mmfs verbsRdmaQpRtsRetryCnt : 6<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> mmfs verbsRdmaQpRtsTimeout : 18<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> mmfs verbsRdmaMaxSendBytes : 16777216<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> mmfs verbsRdmaMaxSendSge : 27<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> mmfs verbsRdmaSend : yes<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> mmfs verbsRdmaSerializeRecv : no<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> mmfs verbsRdmaSerializeSend : no<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> mmfs verbsRdmaUseMultiCqThreads : yes<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> mmfs verbsSendBufferMemoryMB : 1024<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> mmfs verbsLibName : libibverbs.so<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> mmfs verbsRdmaCmLibName : librdmacm.so<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> mmfs verbsRdmaMaxReconnectInterval : 60<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> mmfs verbsRdmaMaxReconnectRetries : -1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> mmfs verbsRdmaReconnectAction : disable<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> mmfs verbsRdmaReconnectThreads : 32<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> mmfs verbsHungRdmaTimeout : 90<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> ibv_fork_support : true<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Max connections : 196608<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Max RDMA size : 16777216<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Target number of vsend buffs : 16384<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Initial vsend buffs per conn : 59<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> nQPs : 140<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> nCQs : 282<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> nCMIDs : 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> nDtoThreads : 2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> nextIndex : 141<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Number of Devices opened : 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Device : mlx4_0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> vendor_id : 713<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Device vendor_part_id : 4099<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Device mem register chunk : 8589934592 (0x200000000)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Device max_sge : 32<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Adjusted max_sge : 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Adjusted max_sge vsend : 30<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Device max_qp_wr : 16351<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Device max_qp_rd_atom : 16<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Open Connect Ports : 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> verbsConnectPorts[0] : mlx4_0/1/0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> lid : 129<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> state : IBV_PORT_ACTIVE<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> path_mtu : 2048<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> interface ID : 0xe41d2d030073b9d1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> sendChannel.ib_channel : 0x7FA6CB816200<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> sendChannel.dtoThreadP : 0x7FA6CB821870<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> sendChannel.dtoThreadId : 12540<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> sendChannel.nFreeCq : 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> recvChannel.ib_channel : 0x7FA6CB81D590<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> recvChannel.dtoThreadP : 0x7FA6CB822BA0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> recvChannel.dtoThreadId : 12541<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> recvChannel.nFreeCq : 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> ibv_cq : 0x7FA2724C81F8<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> ibv_cq.cqP : 0x0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> ibv_cq.nEvents : 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> ibv_cq.contextP : 0x0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> ibv_cq.ib_channel : 0x0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Times New Roman""><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:Arial;color:#AC7F00">Tushar B Pathare MBA IT,BE IT<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Arial;color:gray">Bigdata & GPFS<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Arial;color:gray">Software Development & Databases<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Arial;color:gray">Scientific Computing</span><span style="font-family:"Times New Roman";color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Arial;color:gray">Bioinformatics Division</span><span style="font-size:11.0pt;font-family:"Times New Roman";color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Arial;color:gray">Research</span><span style="font-size:11.0pt;font-family:"Times New Roman";color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Times New Roman";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Arial;color:#AFABAB">"What ever the mind of man can conceive and believe, drill can query"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Times New Roman";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:Arial;color:#AC7F00">Sidra Medical and Research Centre</span></b><span style="font-size:11.0pt;font-family:"Times New Roman";color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:Arial;color:#AC7F00">Sidra OPC Building</span></b><span style="font-size:11.0pt;font-family:"Times New Roman";color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Arial;color:gray">Sidra Medical & Research Center<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Arial;color:gray">PO Box 26999<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Arial;color:gray">Al Luqta Street<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Arial;color:gray">Education City North Campus<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Arial;color:gray">​Qatar Foundation, Doha, Qatar<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Arial;color:gray">Office 4003 3333 ext 37443 | M +974 74793547</span><span style="font-size:11.0pt;font-family:"Times New Roman";color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="FR" style="font-size:10.0pt;font-family:Arial;color:gray"><a href="mailto:tpathare@sidra.org"><span style="color:purple">tpathare@sidra.org</span></a> | </span><span lang="FR" style="font-size:10.0pt;font-family:Arial;color:blue"><a href="http://www.sidra.org/"><span style="color:purple">www.sidra.org</span></a></span><span style="font-size:11.0pt;font-family:"Times New Roman";color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
Disclaimer: This email and its attachments may be confidential and are intended solely for the use of the individual to whom it is addressed. If you are not the intended recipient, any reading, printing, storage, disclosure, copying or any other action taken
in respect of this e-mail is prohibited and may be unlawful. If you are not the intended recipient, please notify the sender immediately by using the reply function and then permanently delete what you have received. Any views or opinions expressed are solely
those of the author and do not necessarily represent those of Sidra Medical and Research Center.
</body>
</html>