<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hey Olaf,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
We'll investigate as suggested. I'm hopeful the journald logs would provide some additional insight.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
As for OFED versions, we use the same Mellanox version across the cluster and haven't seen any issues with working nodes that mount the filesystem.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
We also have a PMR open with IBM but we'll send a follow-up if we discover something more for group discussion.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div id="Signature">
<div>
<div></div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Arial,Helvetica,sans-serif">
<p style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px">
</p>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<span style="font-family:Arial,sans-serif; font-size:13.3333px">Thanks,</span><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<font face="Calibri,Arial,Helvetica,sans-serif"><br>
</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<font face="Calibri,Arial,Helvetica,sans-serif">
<p style="margin-top: 0px; margin-bottom: 0px;color:rgb(32,31,30); text-align:start; background-color:rgb(255,255,255); font-size:11pt; font-family:Calibri,sans-serif; margin:0px">
<b><span style="margin:0px; font-size:10pt; font-family:Arial,sans-serif; color:black">Oluwasijibomi (Siji) Saula</span></b></p>
<p style="margin-top: 0px; margin-bottom: 0px;color:rgb(32,31,30); text-align:start; background-color:rgb(255,255,255); font-size:11pt; font-family:Calibri,sans-serif; margin:0px">
<span style="margin:0px; font-size:10pt; font-family:Arial,sans-serif; color:black">HPC Systems Administrator / Information Technology</span><span style="margin:0px; font-family:"Times New Roman",serif; color:black"></span></p>
<p style="margin-top: 0px; margin-bottom: 0px;color:rgb(32,31,30); text-align:start; background-color:rgb(255,255,255); font-size:11pt; font-family:Calibri,sans-serif; margin:0px">
<span style="margin:0px; font-size:10pt; font-family:Arial,sans-serif; color:black"> </span><span style="margin:0px; font-family:"Times New Roman",serif; color:black"></span></p>
<p style="margin-top: 0px; margin-bottom: 0px;color:rgb(32,31,30); text-align:start; background-color:rgb(255,255,255); font-size:11pt; font-family:Calibri,sans-serif; margin:0px">
<span style="margin:0px; font-size:10pt; font-family:Arial,sans-serif; color:black"><span style="font-family:Arial,sans-serif; background-color:rgb(255,255,255); display:inline!important">Research 2 Building</span> 220B / Fargo ND 58108-6050</span></p>
<p style="margin-top: 0px; margin-bottom: 0px;color:rgb(32,31,30); text-align:start; background-color:rgb(255,255,255); font-size:11pt; font-family:Calibri,sans-serif; margin:0px">
<span style="margin:0px; font-size:10pt; font-family:Arial,sans-serif; color:black">p: 701.231.7749 /<span> </span><a href="http://www.ndsu.edu/" target="_blank" rel="noopener noreferrer" style="margin:0px"><span style="margin:0px; color:purple">www.ndsu.edu</span></a></span><span style="margin:0px; font-family:"Times New Roman",serif; color:black"></span></p>
<p style="margin-top: 0px; margin-bottom: 0px;color:rgb(32,31,30); text-align:start; background-color:rgb(255,255,255); font-size:11pt; font-family:Calibri,sans-serif; margin:0px">
<span style="margin:0px; font-size:10pt; font-family:Arial,sans-serif; color:black"> </span><span style="margin:0px; font-family:"Times New Roman",serif; color:black"></span></p>
<p style="margin-top: 0px; margin-bottom: 0px;color:rgb(32,31,30); text-align:start; background-color:rgb(255,255,255); font-size:11pt; font-family:Calibri,sans-serif; margin:0px">
<span style="margin:0px; color:rgb(31,73,125)"><img class="EmojiInsert" style="margin:0px; width:140.99pt; height:26.24pt" data-outlook-trace="F:0|T:1" src="cid:image001.gif@01D57DE0.91C300C0"></span></p>
</font></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; margin-top:0px; margin-bottom:0px">
<br>
</div>
<br>
<p style="margin-top: 0px; margin-bottom: 0px;margin-top:0px; margin-bottom:0px">
</p>
</div>
</div>
</div>
</div>
<div id="appendonsend"></div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> gpfsug-discuss-bounces@spectrumscale.org <gpfsug-discuss-bounces@spectrumscale.org> on behalf of gpfsug-discuss-request@spectrumscale.org
<gpfsug-discuss-request@spectrumscale.org><br>
<b>Sent:</b> Tuesday, March 30, 2021 1:07 AM<br>
<b>To:</b> gpfsug-discuss@spectrumscale.org <gpfsug-discuss@spectrumscale.org><br>
<b>Subject:</b> gpfsug-discuss Digest, Vol 110, Issue 34</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt">
<div class="PlainText">Send gpfsug-discuss mailing list submissions to<br>
gpfsug-discuss@spectrumscale.org<br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
or, via email, send a message with subject or body 'help' to<br>
gpfsug-discuss-request@spectrumscale.org<br>
<br>
You can reach the person managing the list at<br>
gpfsug-discuss-owner@spectrumscale.org<br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of gpfsug-discuss digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Filesystem mount attempt hangs GPFS client node<br>
(Saula, Oluwasijibomi)<br>
2. Re: Filesystem mount attempt hangs GPFS client node (Olaf Weiser)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Mon, 29 Mar 2021 18:38:00 +0000<br>
From: "Saula, Oluwasijibomi" <oluwasijibomi.saula@ndsu.edu><br>
To: "gpfsug-discuss@spectrumscale.org"<br>
<gpfsug-discuss@spectrumscale.org><br>
Subject: [gpfsug-discuss] Filesystem mount attempt hangs GPFS client<br>
node<br>
Message-ID:<br>
<PH0PR08MB6598F1A7BC557225D417D8C9987E9@PH0PR08MB6598.namprd08.prod.outlook.com><br>
<br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Hello Folks,<br>
<br>
So we are experiencing a mind-boggling issue where just a couple of nodes in our cluster, at GPFS boot up, get hung so badly that the node must be power reset.<br>
<br>
These AMD client nodes are diskless in nature and have at least 256G of memory. We have other AMD nodes that are working just fine in a separate GPFS cluster albeit on RHEL7.<br>
<br>
Just before GPFS (or related processes) seize up the node, the following lines of /var/mmfs/gen/mmfslog are noted:<br>
<br>
<br>
2021-03-29_12:47:37.343-0500: [N] mmfsd ready<br>
<br>
2021-03-29_12:47:37.426-0500: mmcommon mmfsup invoked. Parameters: 10.12.50.47 10.12.50.242 all<br>
<br>
2021-03-29_12:47:37.587-0500: mounting /dev/mmfs1<br>
<br>
2021-03-29_12:47:37.590-0500: [I] Command: mount mmfs1<br>
<br>
2021-03-29_12:47:37.859-0500: [N] Connecting to 10.12.50.243 tier1-sn-02.pixstor <c0n2><br>
<br>
2021-03-29_12:47:37.864-0500: [I] VERBS RDMA connecting to 10.12.50.242 (tier1-sn-01.pixstor) on mlx5_0 port 1 fabnum 0 sl 0 index 0<br>
<br>
2021-03-29_12:47:37.864-0500: [I] VERBS RDMA connecting to 10.12.50.242 (tier1-sn-01) on mlx5_0 port 1 fabnum 0 sl 0 index 1<br>
<br>
2021-03-29_12:47:37.866-0500: [I] VERBS RDMA connected to 10.12.50.242 (tier1-sn-01) on mlx5_0 port 1 fabnum 0 sl 0 index 0<br>
<br>
2021-03-29_12:47:37.867-0500: [I] VERBS RDMA connected to 10.12.50.242 (tier1-sn-01) on mlx5_0 port 1 fabnum 0 sl 0 index 1<br>
<br>
2021-03-29_12:47:37.868-0500: [I] Connected to 10.12.50.243 tier1-sn-02 <c0n2><br>
<br>
There have been hunches that this might be a network issue, however, other nodes connected to the IB network switch are mounting the filesystem without incident.<br>
<br>
I'm inclined to believe there's a GPFS/OS-specific setting that might be causing these crashes especially when we note that disabling the automount on the client node doesn't result in the node hanging. However, once we issue mmmount, we see the node seize
up shortly...<br>
<br>
Please let me know if you have any thoughts on where to look for root-causes as I and a few fellows are stuck here ?<br>
<br>
<br>
<br>
Thanks,<br>
<br>
<br>
Oluwasijibomi (Siji) Saula<br>
<br>
HPC Systems Administrator / Information Technology<br>
<br>
<br>
<br>
Research 2 Building 220B / Fargo ND 58108-6050<br>
<br>
p: 701.231.7749 / www.ndsu.edu<<a href="http://www.ndsu.edu/">http://www.ndsu.edu/</a>><br>
<br>
<br>
<br>
[cid:image001.gif@01D57DE0.91C300C0]<br>
<br>
<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20210329/4ce36267/attachment-0001.html">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20210329/4ce36267/attachment-0001.html</a>><br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Tue, 30 Mar 2021 06:06:54 +0000<br>
From: "Olaf Weiser" <olaf.weiser@de.ibm.com><br>
To: gpfsug-discuss@spectrumscale.org<br>
Cc: gpfsug-discuss@spectrumscale.org<br>
Subject: Re: [gpfsug-discuss] Filesystem mount attempt hangs GPFS<br>
client node<br>
Message-ID:<br>
<OF4FF5120B.5E2B3DE7-ON002586A8.0021023A-002586A8.0021976C@notes.na.collabserv.com><br>
<br>
Content-Type: text/plain; charset="us-ascii"<br>
<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20210330/ae3c3cdd/attachment.html">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20210330/ae3c3cdd/attachment.html</a>><br>
<br>
------------------------------<br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at spectrumscale.org<br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
<br>
<br>
End of gpfsug-discuss Digest, Vol 110, Issue 34<br>
***********************************************<br>
</div>
</span></font></div>
</body>
</html>