<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif, Helvetica, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;" dir="ltr">
<p style="margin-top:0;margin-bottom:0">Hi Folks, </p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">As this is my first post to the group, let me start by saying I applaud the commentary from the user group as it has been a resource to those of us watching from the sidelines.</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">That said, we have a GPFS layered on IPoIB, and recently, we started having some issues on our IB FDR fabric which manifested when GPFS began sending persistent expel messages to particular nodes. </p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">Shortly after, we embarked on a tuning exercise using
<a href="https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Welcome%20to%20High%20Performance%20Computing%20%28HPC%29%20Central/page/Linux%20System%20Tuning%20Recommendations" class="OWAAutoLink" id="LPlnk20027" previewremoved="true">
IBM tuning recommendations</a> but this page is quite old and we've run into some snags, specifically with setting 4k MTUs using mlx4_core/mlx4_en module options. </p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">While setting 4k MTUs as the guide recommends is our general inclination, I'd like to solicit some advice as to whether 4k MTUs are a good idea and any hitch-free steps to accomplishing this. I'm getting some conflicting
remarks from Mellanox support asking why we'd want to use 4k MTUs with Unreliable Datagram mode.</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">Also, any pointers to best practices or resources for network configurations for heavy I/O clusters would be much appreciated.</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif, Helvetica, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;">
<p></p>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<font face="Calibri,Arial,Helvetica,sans-serif">Thanks,</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<font face="Calibri,Arial,Helvetica,sans-serif"> </font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<font face="Calibri,Arial,Helvetica,sans-serif">Siji Saula</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<font face="Calibri,Arial,Helvetica,sans-serif">HPC System Administrator</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<font face="Calibri,Arial,Helvetica,sans-serif">Center for Computationally Assisted Science & Technology</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<b>NORTH DAKOTA STATE UNIVERSITY</b></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<a href="https://www.ndsu.edu/alphaindex/buildings/Building::395" target="_blank" id="LPNoLP"><span id="LPNoLP"></span></a><span id="LPNoLP" style="font-family:Calibri,Arial,Helvetica,sans-serif,serif,EmojiFont; font-size:16px"><a href="https://www.ndsu.edu/alphaindex/buildings/Building::396" target="_blank" rel="noopener noreferrer" id="LPNoLP"><span style="color:rgb(0,111,201)">Research
2 Building</span></a><a href="https://www.ndsu.edu/alphaindex/buildings/Building::395" target="_blank" rel="noopener noreferrer" id="LPNoLP"></a></span><span style="font-family:Calibri,Arial,Helvetica,sans-serif,serif,EmojiFont; font-size:16px"> – </span><span class="contextualExtensionHighlight ms-font-color-themePrimary ms-border-color-themePrimary ident_1641_1697" style="font-family:Calibri,Arial,Helvetica,sans-serif,serif,EmojiFont; font-size:16px; color:rgb(0,0,0)" tabindex="0">Room
220B</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
Dept 4100, PO Box 6050 / Fargo, ND 58108-6050</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
p:701.231.7749</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<a href="file://composeviewinternalloadurl/www.ccast.ndsu.edu" target="_blank" id="LPNoLP"><span id="LPNoLP">www.ccast.ndsu.edu</span></a> | <a href="file://composeviewinternalloadurl/www.ndsu.edu" target="_blank" id="LPNoLP"><span id="LPNoLP">www.ndsu.edu</span></a></div>
<br>
<p></p>
</div>
</div>
</div>
</body>
</html>