<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; background-color: white;">
Hi, Saula,<br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; background-color: white;">
This sounds like the problem with the jumbo frame. <br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; background-color: white;">
Ping or metadata query use small packets, so any time you can ping or ls file. <br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; background-color: white;">
However, data transferring use large packets, the MTU size. Your MTU 65536 nodes send out large packets, but they get dropped to the 2044 nodes, because the packet size cannot fit in 2044 size limit. The reverse is ok.<br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; background-color: white;">
I think the gpfs client nodes always communicate with each other to sync the sdr repo files, or other user job mpi communications if there are any. I think all the nodes should agree on a single MTU. I guess ipoib supports up to 4096.<br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; background-color: white;">
I might missed your Ethernet network switch part whether jumbo frame is enabled or not, if you are using any.<br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; background-color: white;">
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; background-color: white;">
Wei Guo<br>
<br>
</div>
<br>
<br>
</div>
<br>
<br>
<br>
<div class="gmail_quote">On Sat, Mar 10, 2018 at 8:29 AM -0600, "Saula, Oluwasijibomi"
<span dir="ltr"><<a href="mailto:oluwasijibomi.saula@ndsu.edu" target="_blank">oluwasijibomi.saula@ndsu.edu</a>></span> wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="3D"ltr""><style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
<div id="divtagdefaultwrapper" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif, Helvetica, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;" dir="ltr">
<div id="divtagdefaultwrapper" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif, Helvetica, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;" dir="ltr">
<p style="margin-top:0;margin-bottom:0">Wei - So the expelled node could ping the rest of the cluster just fine. In fact, after adding this new node to the cluster I could traverse the filesystem for simple lookups, however, heavy data moves in or out of the
filesystem seemed to trigger the expel messages to the new node.</p>
<p style="margin-top:0;margin-bottom:0"><span style="font-size: 12pt;"><br>
</span></p>
<p style="margin-top:0;margin-bottom:0"><span style="font-size: 12pt;">This experience prompted my tunning exercise on the node and has since resolved the expel messages to node even during times of high I/O activity.</span></p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">Nevertheless, I still have this nagging feeling that the IPoIB tuning for GPFS may not be optimal.</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">To answer your questions, Ed - <span style="font-size: 12pt;">IB supports both administrative and daemon communications, and we have verbsRdma configured.</span></p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">Currently, we have both 2044 and 65520 MTU nodes on our IB network and I've been told this should not be the case. I'm hoping to settle on 4096 MTU nodes for the entire cluster but I fear there may be some caveats - any
thoughts on this? </p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">(Oh, Ed - Hideaki was my mentor for a short while when I began my HPC career with NDSU but he left us shortly after. Maybe like you I can tune up my Japanese as well once my GPFS issues are put to rest!
<span>😊 </span>)</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif, Helvetica, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;">
<p></p>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<font face="Calibri,Arial,Helvetica,sans-serif">Thanks,</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<font face="Calibri,Arial,Helvetica,sans-serif"> </font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<font face="Calibri,Arial,Helvetica,sans-serif">Siji Saula</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<font face="Calibri,Arial,Helvetica,sans-serif">HPC System Administrator</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<font face="Calibri,Arial,Helvetica,sans-serif">Center for Computationally Assisted Science & Technology</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<b>NORTH DAKOTA STATE UNIVERSITY</b></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<a href="https://www.ndsu.edu/alphaindex/buildings/Building::395" target="_blank" id="LPNoLP"><span id="LPNoLP"></span></a><span id="LPNoLP" style="font-family:Calibri,Arial,Helvetica,sans-serif,serif,EmojiFont; font-size:16px"><a href="https://www.ndsu.edu/alphaindex/buildings/Building::396" target="_blank" rel="noopener noreferrer" id="LPNoLP"><span style="color:rgb(0,111,201)">Research
2 Building</span></a><a href="https://www.ndsu.edu/alphaindex/buildings/Building::395" target="_blank" rel="noopener noreferrer" id="LPNoLP"></a></span><span style="font-family:Calibri,Arial,Helvetica,sans-serif,serif,EmojiFont; font-size:16px"> – </span><span tabindex="0" class="contextualExtensionHighlight ms-font-color-themePrimary ms-border-color-themePrimary ident_1641_1697" style="font-family:Calibri,Arial,Helvetica,sans-serif,serif,EmojiFont; font-size:16px; color:rgb(0,0,0)">Room
220B</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
Dept 4100, PO Box 6050 / Fargo, ND 58108-6050</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
p:701.231.7749</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<a href="file://composeviewinternalloadurl/www.ccast.ndsu.edu" target="_blank" id="LPNoLP"><span id="LPNoLP">www.ccast.ndsu.edu</span></a> | <a href="file://composeviewinternalloadurl/www.ndsu.edu" target="_blank" id="LPNoLP"><span id="LPNoLP">www.ndsu.edu</span></a></div>
<br>
<p></p>
</div>
</div>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Edward Wahl <ewahl@osc.edu><br>
<b>Sent:</b> Friday, March 9, 2018 8:19:10 AM<br>
<b>To:</b> gpfsug-discuss@spectrumscale.org<br>
<b>Cc:</b> Saula, Oluwasijibomi<br>
<b>Subject:</b> Re: [gpfsug-discuss] Thoughts on GPFS on IB & MTU sizes</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText"><br>
Welcome to the list. <br>
<br>
If Hideaki Kikuchi is still around CCAST, say "Oh hisashiburi, des ne?" for me.<br>
Though I recall he may have left.<br>
<br>
<br>
A couple of questions as I, unfortunately, have a good deal of expel experience.<br>
<br>
-Are you set up to use verbs or only IPoIB? "mmlsconfig verbsRdma"<br>
<br>
-Are you using the IB as the administrative IP network?<br>
<br>
-As Wei asked, can nodes sending the expel requests ping the victim over<br>
whatever interface is being used administratively? Other interfaces do NOT<br>
matter for expels. Nodes that cannot even mount the file systems can still<br>
request expels. Many many things can cause issues here from routing and<br>
firewalls to bad switch software which will not update ARP tables, and you get<br>
nodes trying to expel each other.<br>
<br>
-are your NSDs logging the expels in /tmp/mmfs? You can mmchconfig<br>
expelDataCollectionDailyLimit if you need more captures to narrow down what is<br>
happening outside the mmfs.log.latest. Just be wary of the disk space if you<br>
have "expel storms".<br>
<br>
-That tuning page is very out of date and appears to be mostly focused on GPFS<br>
3.5.x tuning. While there is also a Spectrum Scale wiki, it's Linux tuning<br>
page does not appear to be kernel and network focused and is dated even older.<br>
<br>
<br>
Ed<br>
<br>
<br>
<br>
On Thu, 8 Mar 2018 15:06:03 +0000<br>
"Saula, Oluwasijibomi" <oluwasijibomi.saula@ndsu.edu> wrote:<br>
<br>
> Hi Folks,<br>
> <br>
> <br>
> As this is my first post to the group, let me start by saying I applaud the<br>
> commentary from the user group as it has been a resource to those of us<br>
> watching from the sidelines.<br>
> <br>
> <br>
> That said, we have a GPFS layered on IPoIB, and recently, we started having<br>
> some issues on our IB FDR fabric which manifested when GPFS began sending<br>
> persistent expel messages to particular nodes.<br>
> <br>
> <br>
> Shortly after, we embarked on a tuning exercise using IBM tuning<br>
> recommendations<<a href="https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Welcome%20to%20High%20Performance%20Computing%20%28HPC%29%20Central/page/Linux%20System%20Tuning%20Recommendations">https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Welcome%20to%20High%20Performance%20Computing%20%28HPC%29%20Central/page/Linux%20System%20Tuning%20Recommendations</a>><br>
> but this page is quite old and we've run into some snags, specifically with<br>
> setting 4k MTUs using mlx4_core/mlx4_en module options.<br>
> <br>
> <br>
> While setting 4k MTUs as the guide recommends is our general inclination, I'd<br>
> like to solicit some advice as to whether 4k MTUs are a good idea and any<br>
> hitch-free steps to accomplishing this. I'm getting some conflicting remarks<br>
> from Mellanox support asking why we'd want to use 4k MTUs with Unreliable<br>
> Datagram mode.<br>
> <br>
> <br>
> Also, any pointers to best practices or resources for network configurations<br>
> for heavy I/O clusters would be much appreciated.<br>
> <br>
> <br>
> Thanks,<br>
> <br>
> Siji Saula<br>
> HPC System Administrator<br>
> Center for Computationally Assisted Science & Technology<br>
> NORTH DAKOTA STATE UNIVERSITY<br>
> <br>
> <br>
> <<a href="https://www.ndsu.edu/alphaindex/buildings/Building::395">https://www.ndsu.edu/alphaindex/buildings/Building::395</a>>Research 2<br>
> Building<<a href="https://www.ndsu.edu/alphaindex/buildings/Building::396><https://www.ndsu.edu/alphaindex/buildings/Building::395">https://www.ndsu.edu/alphaindex/buildings/Building::396><https://www.ndsu.edu/alphaindex/buildings/Building::395</a>><br>
> – Room 220B Dept 4100, PO Box 6050 / Fargo, ND 58108-6050 p:701.231.7749<br>
> www.ccast.ndsu.edu<<a href="file://composeviewinternalloadurl/www.ccast.ndsu.edu">file://composeviewinternalloadurl/www.ccast.ndsu.edu</a>> |<br>
> www.ndsu.edu<<a href="file://composeviewinternalloadurl/www.ndsu.edu">file://composeviewinternalloadurl/www.ndsu.edu</a>><br>
> <br>
<br>
<br>
<br>
-- <br>
<br>
Ed Wahl<br>
Ohio Supercomputer Center<br>
614-292-9302<br>
</div>
</span></font></div>
</div>
</div>
</blockquote>
</div>
<br>
<hr>
<table border="0" cellpadding="0">
<tbody>
<tr>
<td><!-- Copyright 2015. UT Southwestern Medical Center 12082014v1 -->
<p align="right" style="text-align:right"><span style="font-size:17pt;letter-spacing:-3px;font-family:"Helvetica","sans-serif";color:#0054A0;font-weight:bolder;">UT</span><span style="font-size:17pt;letter-spacing:-0px;font-family:"Helvetica","sans-serif";color:#0054A0;font-weight:bolder;">
Southwestern</span> </p>
</td>
</tr>
<tr>
<td>
<p align="right" style="text-align:right"><span style="font-size:13pt;font-family:"Helvetica","sans-serif";color:#666D70;">Medical Center</span></p>
</td>
</tr>
<tr>
<td></td>
</tr>
<tr>
<td>
<p style="text-align:right"><span style="font-size:10pt;font-family:"Helvetica","sans-serif"; color:#666D70;">The future of medicine, today.</span></p>
</td>
</tr>
</tbody>
</table>
</body>
</html>