<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hello Folks,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
So we are experiencing a mind-boggling issue where just a couple of nodes in our cluster, at GPFS boot up, get hung so badly that the node must be power reset.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
These AMD client nodes are diskless in nature and have at least 256G of memory. We have other AMD nodes that are working just fine in a separate GPFS cluster albeit on RHEL7.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Just before GPFS (or related processes) seize up the node, the following lines of /var/mmfs/gen/mmfslog are noted:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<p class="p1" style="margin:0px;font:16px Menlo;background-color:rgb(254, 244, 139)">
<span class="s1" style="font-variant-ligatures:no-common-ligatures">2021-03-29_12:47:37.343-0500: [N] mmfsd ready</span></p>
<p class="p1" style="margin:0px;font:16px Menlo;background-color:rgb(254, 244, 139)">
<span class="s1" style="font-variant-ligatures:no-common-ligatures">2021-03-29_12:47:37.426-0500: mmcommon mmfsup invoked. Parameters: 10.12.50.47 10.12.50.242 all</span></p>
<p class="p1" style="margin:0px;font:16px Menlo;background-color:rgb(254, 244, 139)">
<span class="s1" style="font-variant-ligatures:no-common-ligatures">2021-03-29_12:47:37.587-0500: mounting /dev/mmfs1</span></p>
<p class="p1" style="margin:0px;font:16px Menlo;background-color:rgb(254, 244, 139)">
<span class="s1" style="font-variant-ligatures:no-common-ligatures">2021-03-29_12:47:37.590-0500: [I] Command: mount mmfs1</span></p>
<p class="p1" style="margin:0px;font:16px Menlo;background-color:rgb(254, 244, 139)">
<span class="s1" style="font-variant-ligatures:no-common-ligatures">2021-03-29_12:47:37.859-0500: [N] Connecting to 10.12.50.243 tier1-sn-02.pixstor <c0n2></span></p>
<p class="p1" style="margin:0px;font:16px Menlo;background-color:rgb(254, 244, 139)">
<span class="s1" style="font-variant-ligatures:no-common-ligatures">2021-03-29_12:47:37.864-0500: [I] VERBS RDMA connecting to 10.12.50.242 (tier1-sn-01.pixstor) on mlx5_0 port 1 fabnum 0 sl 0 index 0</span></p>
<p class="p1" style="margin:0px;font:16px Menlo;background-color:rgb(254, 244, 139)">
<span class="s1" style="font-variant-ligatures:no-common-ligatures">2021-03-29_12:47:37.864-0500: [I] VERBS RDMA connecting to 10.12.50.242 (tier1-sn-01) on mlx5_0 port 1 fabnum 0 sl 0 index 1</span></p>
<p class="p1" style="margin:0px;font:16px Menlo;background-color:rgb(254, 244, 139)">
<span class="s1" style="font-variant-ligatures:no-common-ligatures">2021-03-29_12:47:37.866-0500: [I] VERBS RDMA connected to 10.12.50.242 (tier1-sn-01) on mlx5_0 port 1 fabnum 0 sl 0 index 0</span></p>
<p class="p1" style="margin:0px;font:16px Menlo;background-color:rgb(254, 244, 139)">
<span class="s1" style="font-variant-ligatures:no-common-ligatures">2021-03-29_12:47:37.867-0500: [I] VERBS RDMA connected to 10.12.50.242 (tier1-sn-01) on mlx5_0 port 1 fabnum 0 sl 0 index 1</span></p>
<p class="p1" style="margin:0px;font:16px Menlo;background-color:rgb(254, 244, 139)">
<span class="s1" style="font-variant-ligatures:no-common-ligatures">2021-03-29_12:47:37.868-0500: [I] Connected to 10.12.50.243 tier1-sn-02 <c0n2></span></p>
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
There have been hunches that this might be a network issue, however, other nodes connected to the IB network switch are mounting the filesystem without incident.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I'm inclined to believe there's a GPFS/OS-specific setting that might be causing these crashes especially when we note that disabling the automount on the client node doesn't result in the node hanging. However, once we issue mmmount, we see the node seize
up shortly...</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Please let me know if you have any thoughts on where to look for root-causes as I and a few fellows are stuck here
<span id="🙁">🙁</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature">
<div>
<div></div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Arial,Helvetica,sans-serif">
<p style="margin-top:0px; margin-bottom:0px"></p>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<span style="font-family:Arial,sans-serif; font-size:13.3333px">Thanks,</span><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<font face="Calibri,Arial,Helvetica,sans-serif"><br>
</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<font face="Calibri,Arial,Helvetica,sans-serif">
<p style="color:rgb(32,31,30); text-align:start; background-color:rgb(255,255,255); font-size:11pt; font-family:Calibri,sans-serif; margin:0px">
<b><span style="margin:0px; font-size:10pt; font-family:Arial,sans-serif; color:black">Oluwasijibomi (Siji) Saula</span></b></p>
<p style="color:rgb(32,31,30); text-align:start; background-color:rgb(255,255,255); font-size:11pt; font-family:Calibri,sans-serif; margin:0px">
<span style="margin:0px; font-size:10pt; font-family:Arial,sans-serif; color:black">HPC Systems Administrator / Information Technology</span><span style="margin:0px; font-family:"Times New Roman",serif; color:black"></span></p>
<p style="color:rgb(32,31,30); text-align:start; background-color:rgb(255,255,255); font-size:11pt; font-family:Calibri,sans-serif; margin:0px">
<span style="margin:0px; font-size:10pt; font-family:Arial,sans-serif; color:black"> </span><span style="margin:0px; font-family:"Times New Roman",serif; color:black"></span></p>
<p style="color:rgb(32,31,30); text-align:start; background-color:rgb(255,255,255); font-size:11pt; font-family:Calibri,sans-serif; margin:0px">
<span style="margin:0px; font-size:10pt; font-family:Arial,sans-serif; color:black"><span style="font-family:Arial,sans-serif; background-color:rgb(255,255,255); display:inline!important">Research 2 Building</span> 220B / Fargo ND 58108-6050</span></p>
<p style="color:rgb(32,31,30); text-align:start; background-color:rgb(255,255,255); font-size:11pt; font-family:Calibri,sans-serif; margin:0px">
<span style="margin:0px; font-size:10pt; font-family:Arial,sans-serif; color:black">p: 701.231.7749 /<span> </span><a href="http://www.ndsu.edu/" target="_blank" rel="noopener noreferrer" style="margin:0px"><span style="margin:0px; color:purple">www.ndsu.edu</span></a></span><span style="margin:0px; font-family:"Times New Roman",serif; color:black"></span></p>
<p style="color:rgb(32,31,30); text-align:start; background-color:rgb(255,255,255); font-size:11pt; font-family:Calibri,sans-serif; margin:0px">
<span style="margin:0px; font-size:10pt; font-family:Arial,sans-serif; color:black"> </span><span style="margin:0px; font-family:"Times New Roman",serif; color:black"></span></p>
<p style="color:rgb(32,31,30); text-align:start; background-color:rgb(255,255,255); font-size:11pt; font-family:Calibri,sans-serif; margin:0px">
<span style="margin:0px; color:rgb(31,73,125)"><img class="EmojiInsert" style="margin:0px; width:140.99pt; height:26.24pt" data-outlook-trace="F:0|T:1" src="cid:image001.gif@01D57DE0.91C300C0"></span></p>
</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<br>
</div>
<br>
<p style="margin-top:0px; margin-bottom:0px"></p>
</div>
</div>
</div>
</div>
</body>
</html>