<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif, "EmojiFont", "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;">
<p>Hi Norbert,</p>
<p><br>
</p>
<p>thanks a lot for for answering. <br>
</p>
<p><br>
</p>
<p>The nodes are running RHEL7.7 (Kernel <span>3.10.0-1062.12.1.el7.x86_64</span>). The previous version was 5.0.3-2. <br>
</p>
<p>I restarted mmsysmoncontrol (I kept usesharedlib=1 as this is RHEL). Restarting it, it cleans mmhealth messages as expected, let's see whether this is repeated or not but it might take several minutes.</p>
<p><br>
</p>
<p>Just add that when I had a mix of 5.0.3-2 and 5.0.4-2 I received some '<span>stale_mount</span>' messages (from GPFSGUI) for a remote cluster filesystem mountpoints, but apparently everything worked fine. After upgrading everything to v5.0.4-2 looks like
the same nodes report the '<font face="Calibri">csm_resync_needed</font>' instead (no more 'stale_mount' errors seen since then). I am not sure whether this is related or not but might be a hint if this is related.<br>
</p>
<p><br>
</p>
<p>Best regards,</p>
<p>Marc <br>
</p>
<div id="Signature">
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px">
<div class="BodyFragment"><font size="2"><span style="font-size:10pt">
<div class="PlainText">_________________________________________________________<br>
Paul Scherrer Institut <br>
High Performance Computing & Emerging Technologies<br>
Marc Caubet Serrabou<br>
Building/Room: OHSA/014</div>
<div class="PlainText"><font size="2"><span style="font-size:10pt">Forschungsstrasse, 111</span></font></div>
<div class="PlainText">5232 Villigen PSI<br>
Switzerland<br>
<br>
Telephone: +41 56 310 46 67<br>
E-Mail: marc.caubet@psi.ch</div>
</span></font></div>
</div>
</div>
</div>
</div>
</div>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> gpfsug-discuss-bounces@spectrumscale.org <gpfsug-discuss-bounces@spectrumscale.org> on behalf of Norbert Schuld <NSCHULD@de.ibm.com><br>
<b>Sent:</b> Monday, April 6, 2020 2:25:22 PM<br>
<b>To:</b> gpfsug main discussion list<br>
<b>Subject:</b> Re: [gpfsug-discuss] "csm_resync_needed" after upgrading to GPFS v5.0.4-2</font>
<div> </div>
</div>
<div>
<p><font size="2">Hi,</font><br>
<br>
<font size="2">are the nodes running on AIX? If so my advice would be to change /var/mmfs/mmsysmon/mmsysmonitor.conf to read</font><br>
<font size="2">[InterNodeEventing]</font><br>
<font size="2">usesharedlib = 0</font><br>
<font size="2"><br>
and the do a "mmsysmoncontrol restart".</font><br>
<br>
<font size="2">What was the min. release level before the upgrade?</font><br>
<br>
<font size="2">For most other cases a "mmsysmoncontrol restart" on the affected nodes + cluster manager node should do.</font><br>
<br>
<font size="2" face="Arial">Mit freundlichen Grüßen / Kind regards</font><br>
<br>
<b><font face="Arial">Norbert Schuld</font></b><br>
<br>
<br>
<br>
<img naturalheight="16" naturalwidth="16" alt="Inactive hide details for "Caubet Serrabou Marc (PSI)" ---06.04.2020 13:36:28---Hi all, after upgrading one of the clusters to " width="16" height="16" border="0" src="cid:1__=4EBB0FD1DFD04DEF8f9e8a93df938690918c4EB@"><font size="2" color="#424282">"Caubet
Serrabou Marc (PSI)" ---06.04.2020 13:36:28---Hi all, after upgrading one of the clusters to GPFS v5.0.4-2 and setting "minReleaseLevel 5.0.4.0" I</font><br>
<br>
<font size="2" color="#5F5F5F">From: </font><font size="2">"Caubet Serrabou Marc (PSI)" <marc.caubet@psi.ch></font><br>
<font size="2" color="#5F5F5F">To: </font><font size="2">"gpfsug-discuss@spectrumscale.org" <gpfsug-discuss@spectrumscale.org></font><br>
<font size="2" color="#5F5F5F">Date: </font><font size="2">06.04.2020 13:36</font><br>
<font size="2" color="#5F5F5F">Subject: </font><font size="2">[EXTERNAL] [gpfsug-discuss] "csm_resync_needed" after upgrading to GPFS v5.0.4-2</font><br>
<font size="2" color="#5F5F5F">Sent by: </font><font size="2">gpfsug-discuss-bounces@spectrumscale.org</font><br>
</p>
<hr style="color:#8091A5" width="100%" size="2" noshade="" align="left">
<br>
<br>
<br>
<font face="Calibri">Hi all,</font><br>
<br>
<font face="Calibri">after upgrading one of the clusters to GPFS v5.0.4-2 and setting "minReleaseLevel 5.0.4.0" I started to see random "csm_resync_needed" errors on some nodes. This can be easily cleared with "mmhealth node show --resync", however after some
minutes the error re-appears.</font><br>
<font face="Calibri">Apparently, no errors in the log files and no apparent problems other than the "csm_resync_needed" error.
</font><br>
<br>
<font face="Calibri">Before opening a support case, any hints about what could be the reason of that and whether I should worry about it? I would like to clarify what's going on before upgrading the main cluster.</font><br>
<br>
<font face="Calibri">Thanks a lot,</font><br>
<font face="Calibri">Marc </font>
<p><font size="2" face="Tahoma">_________________________________________________________<br>
Paul Scherrer Institut <br>
High Performance Computing & Emerging Technologies<br>
Marc Caubet Serrabou<br>
Building/Room: OHSA/014</font><br>
<font size="2" face="Tahoma">Forschungsstrasse, 111</font><br>
<font size="2" face="Tahoma">5232 Villigen PSI<br>
Switzerland<br>
<br>
Telephone: +41 56 310 46 67<br>
E-Mail: marc.caubet@psi.ch</font><tt><font size="2">_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at spectrumscale.org<br>
</font></tt><tt><font size="2"><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" id="LPlnk585276" previewremoved="true">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></tt><tt><font size="2"> <br>
</font></tt><br>
<br>
<br>
</p>
</div>
</div>
</body>
</html>