<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" >Simon,</div>
<div dir="ltr" > </div>
<div dir="ltr" >are you using Intel 10Gb Network Adapters with RH 7.6 by anychance?</div>
<div dir="ltr" > </div>
<div dir="ltr" >regards</div>
<div dir="ltr" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div dir="ltr" style="margin-top: 20px;" ><div style="font-size: 12pt; font-weight: bold; font-family: sans-serif; color: #7C7C5F;" >Andrew Beattie</div>
<div style="font-size: 10pt; font-weight: bold; font-family: sans-serif;" >File and Object Storage Technical Specialist - A/NZ</div>
<div style="font-size: 10pt; font-weight: bold; font-family: sans-serif;" >IBM Systems - Storage</div>
<div style="font-size: 8pt; font-family: sans-serif; margin-top: 10px;" ><div><span style="font-weight: bold; color: #336699;" >Phone: </span>614-2133-7927</div>
<div><span style="font-weight: bold; color: #336699;" >E-mail: </span><a href="mailto:abeattie@au1.ibm.com" style="color: #555">abeattie@au1.ibm.com</a></div></div></div></div></div></div></div></div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Original message -----<br>From: Simon Thompson <S.J.Thompson@bham.ac.uk><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: "gpfsug-discuss@spectrumscale.org" <gpfsug-discuss@spectrumscale.org><br>Cc:<br>Subject: [EXTERNAL] [gpfsug-discuss] GPFS and POWER9<br>Date: Thu, Sep 19, 2019 8:42 PM<br> <br><!--Notes ACF
<meta http-equiv="Content-Type" content="text/html; charset=utf8" >-->
<div><p style="margin: 0px;" ><span style="font-size:11.0pt" >Recently we’ve been having some issues with some of our POWER9 systems. They are occasionally handing or rebooting, in one case, we’ve found we can cause them to do it by running some MPI IOR workload to GPFS. Every instance we’ve seen which has logged something to syslog has had mmfsd referenced, but we don’t know if that is a symptom or a cause. (sometimes they just hang and we don’t see such a message) We see the following in the kern log:<o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:11.0pt" ><o:p> </o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:11.0pt" >Sep 18 18:45:14 bear-pg0306u11a kernel: Hypervisor Maintenance interrupt [Recovered]<o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:11.0pt" >Sep 18 18:45:14 bear-pg0306u11a kernel: Error detail: Malfunction Alert<o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:11.0pt" >Sep 18 18:45:14 bear-pg0306u11a kernel: #011HMER: 8040000000000000<o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:11.0pt" >Sep 18 18:45:14 bear-pg0306u11a kernel: #011Unknown Malfunction Alert of type 3<o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:11.0pt" >Sep 18 18:45:14 bear-pg0306u11a kernel: Hypervisor Maintenance interrupt [Recovered]<o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:11.0pt" >Sep 18 18:45:14 bear-pg0306u11a kernel: Error detail: Malfunction Alert<o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:11.0pt" >Sep 18 18:45:14 bear-pg0306u11a kernel: #011HMER: 8040000000000000<o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:11.0pt" >Sep 18 18:45:14 bear-pg0306u11a kernel: Severe Machine check interrupt [Not recovered]<o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:11.0pt" >Sep 18 18:45:14 bear-pg0306u11a kernel: NIP: [00000000115a2478] PID: 141380 Comm: mmfsd<o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:11.0pt" >Sep 18 18:45:14 bear-pg0306u11a kernel: Initiator: CPU<o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:11.0pt" >Sep 18 18:45:14 bear-pg0306u11a kernel: Error type: UE [Load/Store]<o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:11.0pt" >Sep 18 18:45:14 bear-pg0306u11a kernel: Effective address: 000003002a2a8400<o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:11.0pt" >Sep 18 18:45:14 bear-pg0306u11a kernel: Physical address: 000003c016590000<o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:11.0pt" >Sep 18 18:45:14 bear-pg0306u11a kernel: Severe Machine check interrupt [Not recovered]<o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:11.0pt" >Sep 18 18:45:14 bear-pg0306u11a kernel: NIP: [000000001150b160] PID: 141380 Comm: mmfsd<o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:11.0pt" >Sep 18 18:45:14 bear-pg0306u11a kernel: Initiator: CPU<o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:11.0pt" >Sep 18 18:45:14 bear-pg0306u11a kernel: Error type: UE [Instruction fetch]<o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:11.0pt" >Sep 18 18:45:14 bear-pg0306u11a kernel: Effective address: 000000001150b160<o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:11.0pt" >Sep 18 18:45:14 bear-pg0306u11a kernel: Physical address: 000003c01fe80000<o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:11.0pt" >Sep 18 18:45:14 bear-pg0306u11a kernel: Severe Machine check interrupt [Not recovered]<o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:11.0pt" >Sep 18 18:45:14 bear-pg0306u11a kernel: NIP: [000000001086a7f0] PID: 25926 Comm: mmfsd<o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:11.0pt" >Sep 18 18:45:14 bear-pg0306u11a kernel: Initiator: CPU<o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:11.0pt" >Sep 18 18:45:14 bear-pg0306u11a kernel: Error type: UE [Instruction fetch]<o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:11.0pt" >Sep 18 18:45:14 bear-pg0306u11a kernel: Effective address: 000000001086a7f0<o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:11.0pt" >Sep 18 18:45:14 bear-pg0306u11a kernel: Physical address: 000003c00fe70000<o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:11.0pt" >Sep 18 18:45:14 bear-pg0306u11a kernel: mmfsd[25926]: unhandled signal 7 at 000000001086a7f0 nip 000000001086a7f0 lr 000000001086a7f0 code 4<o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:11.0pt" ><o:p> </o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:11.0pt" >I’ve raised a hardware ticket with IBM, as traditionally a machine check exception would likely be a hardware/firmware issue. Anyone else seen this sort of behaviour? Its multiple boxes doing this, but they do all have the same firmware/rhel/gpfs stack installed.<o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:11.0pt" ><o:p> </o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:11.0pt" >Asking here as they always reference mmfsd PIDs … (but maybe it’s a symptom rather than cause)…<o:p></o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:11.0pt" ><o:p> </o:p></span></p>
<p style="margin: 0px;" ><span style="font-size:11.0pt" >Simon<o:p></o:p></span></p></div>
<div><font size="2" face="Default Monospace,Courier New,Courier,monospace" >_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> </font></div></blockquote>
<div dir="ltr" > </div></div><BR>