<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
p.CODE, li.CODE, div.CODE
{mso-style-name:CODE;
mso-style-priority:99;
mso-style-link:"CODE Char";
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
background:#F2F2F2;
border:none;
padding:0cm;
font-size:9.0pt;
font-family:Consolas;}
span.CODEChar
{mso-style-name:"CODE Char";
mso-style-link:CODE;
font-family:Consolas;
background:#F2F2F2;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">We think our issue was down to numa settings actually - making mmfsd allocate GPU memory. Makes sense given the type of error.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Tomer suggested to Simon we set numactlOptioni to "0 8", as per:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><a href="https://www-01.ibm.com/support/docview.wss?uid=isg1IJ02794">https://www-01.ibm.com/support/docview.wss?uid=isg1IJ02794</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Our tests are not crashing since setting then – we need to roll it out on all nodes to confirm its fixed all our hangs/reboots.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Cheers,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Luke<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Calibri",sans-serif;color:#1F497D">--
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Luke Sudbery<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Architecture, Infrastructure and Systems<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Advanced Research Computing, IT Services<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Room 132, Computer Centre G5, Elms Road<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:9.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Please note I don’t work on Monday and work from home on Friday.<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> gpfsug-discuss-bounces@spectrumscale.org <gpfsug-discuss-bounces@spectrumscale.org>
<b>On Behalf Of </b>abeattie@au1.ibm.com<br>
<b>Sent:</b> 19 September 2019 22:35<br>
<b>To:</b> gpfsug-discuss@spectrumscale.org<br>
<b>Cc:</b> gpfsug-discuss@spectrumscale.org<br>
<b>Subject:</b> Re: [gpfsug-discuss] GPFS and POWER9<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Simon,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">I have an open support call that required Redhat to create a kernel patch for RH 7.6 because of issues with the Intel x710 network adapter - I can't tell you if its related to
your issue or not<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">but it would cause the GPFS cluster to reboot and the affected node to reboot if we tried to do almost anything with that intel adapter<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">regards,<o:p></o:p></span></p>
</div>
<div>
<div>
<div>
<div>
<div>
<div style="margin-top:15.0pt">
<div>
<p class="MsoNormal"><b><span style="font-family:"Arial",sans-serif;color:#7C7C5F">Andrew Beattie<o:p></o:p></span></b></p>
</div>
<div>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif">File and Object Storage Technical Specialist - A/NZ<o:p></o:p></span></b></p>
</div>
<div>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif">IBM Systems - Storage<o:p></o:p></span></b></p>
</div>
<div style="margin-top:7.5pt">
<div>
<p class="MsoNormal"><b><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#336699">Phone:
</span></b><span style="font-size:8.0pt;font-family:"Arial",sans-serif">614-2133-7927<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><b><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#336699">E-mail:
</span></b><a href="mailto:abeattie@au1.ibm.com"><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#555555">abeattie@au1.ibm.com</span></a><span style="font-size:8.0pt;font-family:"Arial",sans-serif"><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif"> <o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #AAAAAA 1.5pt;padding:0cm 0cm 0cm 4.0pt;margin-left:3.75pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">----- Original message -----<br>
From: Simon Thompson <</span><a href="mailto:S.J.Thompson@bham.ac.uk"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">S.J.Thompson@bham.ac.uk</span></a><span style="font-size:10.0pt;font-family:"Arial",sans-serif">><br>
Sent by: </span><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">gpfsug-discuss-bounces@spectrumscale.org</span></a><span style="font-size:10.0pt;font-family:"Arial",sans-serif"><br>
To: gpfsug main discussion list <</span><a href="mailto:gpfsug-discuss@spectrumscale.org"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">gpfsug-discuss@spectrumscale.org</span></a><span style="font-size:10.0pt;font-family:"Arial",sans-serif">><br>
Cc:<br>
Subject: [EXTERNAL] Re: [gpfsug-discuss] GPFS and POWER9<br>
Date: Fri, Sep 20, 2019 1:18 AM<br>
<o:p></o:p></span></p>
<div>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Hi Andrew,<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif"> <o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Yes, but not only. We use the two SFP+ ports from the Broadcom supplied card + the bifurcated Mellanox card in them.<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif"> <o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Simon<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif"> <o:p></o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p style="margin:0cm;margin-bottom:.0001pt"><b><span style="font-family:"Arial",sans-serif;color:black">From:
</span></b><span style="font-family:"Arial",sans-serif;color:black"><</span><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><span style="font-family:"Arial",sans-serif">gpfsug-discuss-bounces@spectrumscale.org</span></a><span style="font-family:"Arial",sans-serif;color:black">>
on behalf of "</span><a href="mailto:abeattie@au1.ibm.com"><span style="font-family:"Arial",sans-serif">abeattie@au1.ibm.com</span></a><span style="font-family:"Arial",sans-serif;color:black">" <</span><a href="mailto:abeattie@au1.ibm.com"><span style="font-family:"Arial",sans-serif">abeattie@au1.ibm.com</span></a><span style="font-family:"Arial",sans-serif;color:black">><br>
<b>Reply-To: </b>"</span><a href="mailto:gpfsug-discuss@spectrumscale.org"><span style="font-family:"Arial",sans-serif">gpfsug-discuss@spectrumscale.org</span></a><span style="font-family:"Arial",sans-serif;color:black">" <</span><a href="mailto:gpfsug-discuss@spectrumscale.org"><span style="font-family:"Arial",sans-serif">gpfsug-discuss@spectrumscale.org</span></a><span style="font-family:"Arial",sans-serif;color:black">><br>
<b>Date: </b>Thursday, 19 September 2019 at 11:45<br>
<b>To: </b>"</span><a href="mailto:gpfsug-discuss@spectrumscale.org"><span style="font-family:"Arial",sans-serif">gpfsug-discuss@spectrumscale.org</span></a><span style="font-family:"Arial",sans-serif;color:black">" <</span><a href="mailto:gpfsug-discuss@spectrumscale.org"><span style="font-family:"Arial",sans-serif">gpfsug-discuss@spectrumscale.org</span></a><span style="font-family:"Arial",sans-serif;color:black">><br>
<b>Subject: </b>Re: [gpfsug-discuss] GPFS and POWER9</span><span style="font-size:10.0pt;font-family:"Arial",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif"> <o:p></o:p></span></p>
</div>
<div>
<div>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Simon,<o:p></o:p></span></p>
</div>
<div>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif"> <o:p></o:p></span></p>
</div>
<div>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">are you using Intel 10Gb Network Adapters with RH 7.6 by anychance?<o:p></o:p></span></p>
</div>
<div>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif"> <o:p></o:p></span></p>
</div>
<div>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">regards<o:p></o:p></span></p>
</div>
<div>
<div>
<div>
<div>
<div>
<div style="margin-top:15.0pt">
<div>
<p style="margin:0cm;margin-bottom:.0001pt"><b><span style="font-family:"Arial",sans-serif;color:#7C7C5F">Andrew Beattie</span></b><span style="font-size:10.0pt;font-family:"Arial",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p style="margin:0cm;margin-bottom:.0001pt"><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif">File and Object Storage Technical Specialist - A/NZ</span></b><span style="font-size:10.0pt;font-family:"Arial",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p style="margin:0cm;margin-bottom:.0001pt"><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif">IBM Systems - Storage</span></b><span style="font-size:10.0pt;font-family:"Arial",sans-serif"><o:p></o:p></span></p>
</div>
<div style="margin-top:7.5pt">
<div>
<p style="margin:0cm;margin-bottom:.0001pt"><b><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#336699">Phone:
</span></b><span style="font-size:8.0pt;font-family:"Arial",sans-serif">614-2133-7927</span><span style="font-size:10.0pt;font-family:"Arial",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p style="margin:0cm;margin-bottom:.0001pt"><b><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#336699">E-mail:
</span></b><a href="mailto:abeattie@au1.ibm.com" target="_blank"><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#555555">abeattie@au1.ibm.com</span></a><span style="font-size:10.0pt;font-family:"Arial",sans-serif"><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif"> <o:p></o:p></span></p>
</div>
<div>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif"> <o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #AAAAAA 1.5pt;padding:0cm 0cm 0cm 4.0pt;margin-left:3.75pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<p style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:0cm">
<span style="font-size:10.0pt;font-family:"Arial",sans-serif">----- Original message -----<br>
From: Simon Thompson <</span><a href="mailto:S.J.Thompson@bham.ac.uk"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">S.J.Thompson@bham.ac.uk</span></a><span style="font-size:10.0pt;font-family:"Arial",sans-serif">><br>
Sent by: </span><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">gpfsug-discuss-bounces@spectrumscale.org</span></a><span style="font-size:10.0pt;font-family:"Arial",sans-serif"><br>
To: "</span><a href="mailto:gpfsug-discuss@spectrumscale.org"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">gpfsug-discuss@spectrumscale.org</span></a><span style="font-size:10.0pt;font-family:"Arial",sans-serif">" <</span><a href="mailto:gpfsug-discuss@spectrumscale.org"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">gpfsug-discuss@spectrumscale.org</span></a><span style="font-size:10.0pt;font-family:"Arial",sans-serif">><br>
Cc:<br>
Subject: [EXTERNAL] [gpfsug-discuss] GPFS and POWER9<br>
Date: Thu, Sep 19, 2019 8:42 PM<br>
<o:p></o:p></span></p>
<div>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Recently we’ve been having some issues with some of our POWER9 systems. They are occasionally handing or rebooting, in one case, we’ve found we can cause
them to do it by running some MPI IOR workload to GPFS. Every instance we’ve seen which has logged something to syslog has had mmfsd referenced, but we don’t know if that is a symptom or a cause. (sometimes they just hang and we don’t see such a message) We
see the following in the kern log:<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif"> <o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Sep 18 18:45:14 bear-pg0306u11a kernel: Hypervisor Maintenance interrupt [Recovered]<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Sep 18 18:45:14 bear-pg0306u11a kernel: Error detail: Malfunction Alert<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Sep 18 18:45:14 bear-pg0306u11a kernel: #011HMER: 8040000000000000<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Sep 18 18:45:14 bear-pg0306u11a kernel: #011Unknown Malfunction Alert of type 3<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Sep 18 18:45:14 bear-pg0306u11a kernel: Hypervisor Maintenance interrupt [Recovered]<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Sep 18 18:45:14 bear-pg0306u11a kernel: Error detail: Malfunction Alert<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Sep 18 18:45:14 bear-pg0306u11a kernel: #011HMER: 8040000000000000<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Sep 18 18:45:14 bear-pg0306u11a kernel: Severe Machine check interrupt [Not recovered]<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Sep 18 18:45:14 bear-pg0306u11a kernel: NIP: [00000000115a2478] PID: 141380 Comm: mmfsd<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Sep 18 18:45:14 bear-pg0306u11a kernel: Initiator: CPU<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Sep 18 18:45:14 bear-pg0306u11a kernel: Error type: UE [Load/Store]<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Sep 18 18:45:14 bear-pg0306u11a kernel: Effective address: 000003002a2a8400<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Sep 18 18:45:14 bear-pg0306u11a kernel: Physical address: 000003c016590000<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Sep 18 18:45:14 bear-pg0306u11a kernel: Severe Machine check interrupt [Not recovered]<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Sep 18 18:45:14 bear-pg0306u11a kernel: NIP: [000000001150b160] PID: 141380 Comm: mmfsd<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Sep 18 18:45:14 bear-pg0306u11a kernel: Initiator: CPU<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Sep 18 18:45:14 bear-pg0306u11a kernel: Error type: UE [Instruction fetch]<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Sep 18 18:45:14 bear-pg0306u11a kernel: Effective address: 000000001150b160<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Sep 18 18:45:14 bear-pg0306u11a kernel: Physical address: 000003c01fe80000<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Sep 18 18:45:14 bear-pg0306u11a kernel: Severe Machine check interrupt [Not recovered]<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Sep 18 18:45:14 bear-pg0306u11a kernel: NIP: [000000001086a7f0] PID: 25926 Comm: mmfsd<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Sep 18 18:45:14 bear-pg0306u11a kernel: Initiator: CPU<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Sep 18 18:45:14 bear-pg0306u11a kernel: Error type: UE [Instruction fetch]<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Sep 18 18:45:14 bear-pg0306u11a kernel: Effective address: 000000001086a7f0<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Sep 18 18:45:14 bear-pg0306u11a kernel: Physical address: 000003c00fe70000<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Sep 18 18:45:14 bear-pg0306u11a kernel: mmfsd[25926]: unhandled signal 7 at 000000001086a7f0 nip 000000001086a7f0 lr 000000001086a7f0 code 4<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif"> <o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">I’ve raised a hardware ticket with IBM, as traditionally a machine check exception would likely be a hardware/firmware issue. Anyone else seen this sort
of behaviour? Its multiple boxes doing this, but they do all have the same firmware/rhel/gpfs stack installed.<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif"> <o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Asking here as they always reference mmfsd PIDs … (but maybe it’s a symptom rather than cause)…<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif"> <o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Simon<o:p></o:p></span></p>
</div>
<div>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Courier New"">_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at spectrumscale.org<br>
</span><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><span style="font-size:10.0pt;font-family:"Courier New"">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></a><span style="font-size:10.0pt;font-family:"Courier New""> </span><span style="font-size:10.0pt;font-family:"Arial",sans-serif"><o:p></o:p></span></p>
</div>
</blockquote>
<div>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif"> <o:p></o:p></span></p>
</div>
</div>
<p style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:0cm">
<span style="font-size:10.0pt;font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at spectrumscale.org<br>
</span><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><span style="font-size:10.0pt;font-family:"Courier New"">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></a><span style="font-size:10.0pt;font-family:"Courier New""> </span><span style="font-size:10.0pt;font-family:"Arial",sans-serif"><o:p></o:p></span></p>
</div>
</blockquote>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif"> <o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>