<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body dir="auto">
<div>You know, I ran into this during the upgrade, didn’t have an explanation for it, and decided that I would see what happened when it was finished (problem was gone). I was going from DSS-G 3.2a to DSS-G 4.3a. It was at the stage where I had one machine
upgraded and one machine not upgraded, which makes sense. I believe I then tried it from the upgraded node, it worked fine, and I figured “well, I’m blowing that one away shortly anyway.”</div>
<div><br>
</div>
<div dir="ltr">Would one expect to only see this while nodes in the same cluster are not running the same version? I guess what I’m asking is would one expect to have that kind of traffic coming from any place else except the same cluster?</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Sent from my iPhone</div>
<div dir="ltr"><br>
<blockquote type="cite">On Mar 21, 2023, at 08:38, Luke Sudbery <L.R.Sudbery@bham.ac.uk> wrote:<br>
<br>
</blockquote>
</div>
<blockquote type="cite">
<div dir="ltr">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style>@font-face { font-family: Wingdings; }
@font-face { font-family: SimSun; }
@font-face { font-family: "MS Gothic"; }
@font-face { font-family: "Cambria Math"; }
@font-face { font-family: Calibri; }
@font-face { font-family: "Calibri Light"; }
@font-face { font-family: "@MS Gothic"; }
@font-face { font-family: "@SimSun"; }
@font-face { font-family: Menlo; }
p.MsoNormal, li.MsoNormal, div.MsoNormal { margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif; }
h2 { margin: 2pt 0cm 0cm; break-after: avoid; font-size: 13pt; font-family: "Calibri Light", sans-serif; color: rgb(47, 84, 150); font-weight: normal; }
a:link, span.MsoHyperlink { color: rgb(5, 99, 193); text-decoration: underline; }
pre { margin: 0cm 0cm 0.0001pt; font-size: 10pt; font-family: "Courier New"; }
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph { margin: 0cm 0cm 0cm 36pt; font-size: 11pt; font-family: Calibri, sans-serif; }
p.Code, li.Code, div.Code { margin: 0cm 0cm 0cm 36pt; background-color: rgb(255, 242, 204); font-size: 11pt; font-family: "Courier New"; }
span.CodeChar { font-family: "Courier New"; background-color: rgb(255, 242, 204); }
span.EmailStyle23 { font-family: Calibri, sans-serif; color: windowtext; }
span.Heading2Char { font-family: "Calibri Light", sans-serif; color: rgb(47, 84, 150); }
span.HTMLPreformattedChar { font-family: "Courier New"; }
.MsoChpDefault { font-size: 10pt; }
@page WordSection1 { size: 612pt 792pt; margin: 72pt; }
div.WordSection1 { page: WordSection1; }
ol { margin-bottom: 0cm; }
ul { margin-bottom: 0cm; }</style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal">Thank you, I’ll pass that onto Lenovo. I think I’ve also identified it as this in the fixies in 5.1.6.1 too:<o:p></o:p></p>
<p class="MsoNormal"><a href="https://www.ibm.com/support/pages/apar/IJ44607">https://www.ibm.com/support/pages/apar/IJ44607</a><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:24.0pt;mso-fareast-language:EN-GB">IJ44607: SPECTRUM SCALE V5.1.3 AND GREATER FAILS TO INTEROPERATE WITH SPECTRUMSCALE VERSIONS PRIOR TO 5.1.3.<o:p></o:p></span></b></p>
<h2>Problem summary<o:p></o:p></h2>
<pre style="margin-left:36.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo5"><!--[if !supportLists]--><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman""> </span></span></span><!--[endif]-->GNR RPCs fail when received by a GPFS daemon 5.1.3 or later<o:p></o:p></pre>
<pre style="margin-left:36.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo5"><!--[if !supportLists]--><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman""> </span></span></span><!--[endif]-->from a GPFS daemon older than version 5.1.3.<o:p></o:p></pre>
<pre style="margin-left:36.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo5"><!--[if !supportLists]--><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman""> </span></span></span><!--[endif]-->Kernel assert going off: privVfsP != NULL<o:p></o:p></pre>
<h2>Symptom:<o:p></o:p></h2>
<pre style="margin-left:36.0pt;text-indent:-18.0pt;mso-list:l1 level1 lfo6"><!--[if !supportLists]--><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman""> </span></span></span><!--[endif]-->Hang in the command<o:p></o:p></pre>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><o:p> </o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">As such, AFAICT it will affect all Lenovo customers going from DSSG 3.x (GPFS 5.1.1.0) to 4.x (GPFS 5.1.5.1 efix20) and I’m a bit annoyed that didn’t pick it up before release.<o:p></o:p></p>
<p class="MsoNormal">Many thanks,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Luke<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#1F497D;mso-fareast-language:EN-GB">--
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#1F497D;mso-fareast-language:EN-GB">Luke Sudbery<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#1F497D;mso-fareast-language:EN-GB">Principal Engineer (HPC and Storage).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#1F497D;mso-fareast-language:EN-GB">Architecture, Infrastructure and Systems<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#1F497D;mso-fareast-language:EN-GB">Advanced Research Computing, IT Services<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#1F497D;mso-fareast-language:EN-GB">Room 132, Computer Centre G5, Elms Road<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#1F497D;mso-fareast-language:EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:9.0pt;color:#1F497D;mso-fareast-language:EN-GB">Please note I don’t work on Monday.<o:p></o:p></span></b></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="mso-fareast-language:EN-GB">From:</span></b><span lang="EN-US" style="mso-fareast-language:EN-GB"> Gang Qiu <gangqiu@cn.ibm.com>
<br>
<b>Sent:</b> 21 March 2023 12:31<br>
<b>To:</b> Luke Sudbery (Advanced Research Computing) <l.r.sudbery@bham.ac.uk>; gpfsug main discussion list <gpfsug-discuss@gpfsug.org><br>
<b>Subject:</b> </span><span style="font-family:"MS Gothic";mso-fareast-language:EN-GB">答复</span><span lang="EN-US" style="mso-fareast-language:EN-GB">: mmvdisk version/communication issues?<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left" width="100%" style="width:100.0%">
<tbody>
<tr>
<td width="10%" style="width:10.0%;border:solid #9C6500 1.0pt;background:#FFEB9C;padding:2.0pt 2.0pt 2.0pt 2.0pt">
<p class="MsoNormal" style="mso-element:frame;mso-element-frame-hspace:2.25pt;mso-element-wrap:around;mso-element-anchor-vertical:paragraph;mso-element-anchor-horizontal:column;mso-height-rule:exactly">
<b><span style="color:black;mso-fareast-language:EN-GB">CAUTION:</span></b><span style="color:black;mso-fareast-language:EN-GB"> This email originated from outside the organisation. Do not click links or open attachments unless you recognise the sender and
know the content is safe.</span><span style="mso-fareast-language:EN-GB"><o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-CN"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;mso-fareast-language:ZH-CN"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;mso-fareast-language:ZH-CN"><o:p> </o:p></span></p>
<div>
<div>
<div>
<p class="MsoNormal" style="line-height:13.5pt;background:white"><span lang="EN-US" style="font-size:10.5pt;color:black;mso-fareast-language:ZH-CN">Luke,<o:p></o:p></span></p>
<p class="MsoNormal" style="line-height:13.5pt;background:white"><span lang="EN-US" style="font-size:10.5pt;color:black;mso-fareast-language:ZH-CN"><o:p> </o:p></span></p>
<p class="MsoNormal" style="line-height:13.5pt;background:white"><span lang="EN-US" style="font-size:9.0pt;font-family:Menlo;color:#0451A5;mso-fareast-language:ZH-CN">1.</span><span lang="EN-US" style="font-size:9.0pt;font-family:Menlo;color:black;mso-fareast-language:ZH-CN">
The case number is TS011014198 (Internal defect number is 1163600)<o:p></o:p></span></p>
<p class="MsoNormal" style="line-height:13.5pt;background:white"><span lang="EN-US" style="font-size:9.0pt;font-family:Menlo;color:black;mso-fareast-language:ZH-CN"> 5.1.3.1 efix47 includes this fix. (I didn't find the efix for 5.1.4 or 5.1.5)<o:p></o:p></span></p>
<p class="MsoNormal" style="line-height:13.5pt;background:white"><span lang="EN-US" style="font-size:9.0pt;font-family:Menlo;color:#0451A5;mso-fareast-language:ZH-CN">2.</span><span lang="EN-US" style="font-size:9.0pt;font-family:Menlo;color:black;mso-fareast-language:ZH-CN">
The impact is that any GNR-related command from 5.1.2 to 5.1.5 will hang.<o:p></o:p></span></p>
<p class="MsoNormal" style="line-height:13.5pt;background:white"><span lang="EN-US" style="font-size:10.5pt;color:black;mso-fareast-language:ZH-CN"><o:p> </o:p></span></p>
<p class="MsoNormal" style="line-height:13.5pt;background:white"><span lang="EN-US" style="font-size:10.5pt;color:black;mso-fareast-language:ZH-CN">Regards,<o:p></o:p></span></p>
<p class="MsoNormal" style="line-height:13.5pt;background:white"><span lang="EN-US" style="font-size:10.5pt;color:black;mso-fareast-language:ZH-CN">Gang Qiu<o:p></o:p></span></p>
<p class="MsoNormal" style="line-height:13.5pt;background:white"><span lang="EN-US" style="font-size:10.5pt;color:black;mso-fareast-language:ZH-CN">====================================================<o:p></o:p></span></p>
<p class="MsoNormal" style="line-height:13.5pt;background:white"><span lang="EN-US" style="font-size:10.5pt;color:black;mso-fareast-language:ZH-CN">Gang Qiu(</span><span lang="ZH-CN" style="font-size:10.5pt;font-family:SimSun;color:black;mso-fareast-language:ZH-CN">邱钢</span><span lang="EN-US" style="font-size:10.5pt;color:black;mso-fareast-language:ZH-CN">)<o:p></o:p></span></p>
<p class="MsoNormal" style="line-height:13.5pt;background:white"><span lang="EN-US" style="font-size:10.5pt;color:black;mso-fareast-language:ZH-CN">Spectrum Scale Development - ECE<o:p></o:p></span></p>
<p class="MsoNormal" style="line-height:13.5pt;background:white"><span lang="EN-US" style="font-size:10.5pt;color:black;mso-fareast-language:ZH-CN">IBM China Systems Lab<o:p></o:p></span></p>
<p class="MsoNormal" style="line-height:13.5pt;background:white"><span lang="EN-US" style="font-size:10.5pt;color:black;mso-fareast-language:ZH-CN">Mobile: +86-18612867902<o:p></o:p></span></p>
<p class="MsoNormal" style="line-height:13.5pt;background:white"><span lang="EN-US" style="font-size:10.5pt;color:black;mso-fareast-language:ZH-CN">====================================================<o:p></o:p></span></p>
</div>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;mso-fareast-language:ZH-CN"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;mso-fareast-language:ZH-CN"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span lang="EN-US" style="font-family:SimSun;color:black">发件人</span></b><b><span lang="EN-US" style="color:black">:</span></b><span lang="EN-US" style="color:black"> Luke Sudbery <<a href="mailto:l.r.sudbery@bham.ac.uk">l.r.sudbery@bham.ac.uk</a>><br>
</span><b><span lang="EN-US" style="font-family:SimSun;color:black">日期</span></b><b><span lang="EN-US" style="color:black">:</span></b><span lang="EN-US" style="color:black">
</span><span lang="EN-US" style="font-family:SimSun;color:black">星期二</span><span lang="EN-US" style="color:black">, 2023</span><span lang="EN-US" style="font-family:SimSun;color:black">年</span><span lang="EN-US" style="color:black">3</span><span lang="EN-US" style="font-family:SimSun;color:black">月</span><span lang="EN-US" style="color:black">21</span><span lang="EN-US" style="font-family:SimSun;color:black">日</span><span lang="EN-US" style="color:black">
17:31<br>
</span><b><span lang="EN-US" style="font-family:SimSun;color:black">收件人</span></b><b><span lang="EN-US" style="color:black">:</span></b><span lang="EN-US" style="color:black"> gpfsug main discussion list <<a href="mailto:gpfsug-discuss@gpfsug.org">gpfsug-discuss@gpfsug.org</a>>,
Gang Qiu <<a href="mailto:gangqiu@cn.ibm.com">gangqiu@cn.ibm.com</a>><br>
</span><b><span lang="EN-US" style="font-family:SimSun;color:black">主题</span></b><b><span lang="EN-US" style="color:black">:</span></b><span lang="EN-US" style="color:black"> [EXTERNAL] RE: mmvdisk version/communication issues?</span><span lang="EN-US" style="font-size:12.0pt;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span lang="EN-US" style="font-size:1.0pt;color:white">Thank you, that’s far more useful than Lenovo support have been so far! Unfortunately Lenovo only support particular versions of scale on their DSSG
and they are our only source of the required GNR packages. So a couple more questions if you
</span><span lang="EN-US" style="font-size:1.0pt;font-family:SimSun;color:white"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span lang="EN-US" style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerStart<o:p></o:p></span></p>
</div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;border-radius:4px">
<tbody>
<tr>
<td style="padding:12.0pt 0cm 12.0pt 0cm">
<table class="MsoNormalTable" border="1" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;background:#D0D8DC;border:none;border-top:solid #90A4AE 3.0pt">
<tbody>
<tr>
<td valign="top" style="border:none;padding:0cm 7.5pt 3.75pt 4.5pt">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left">
<tbody>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">This Message Is From an External Sender
<o:p></o:p></span></b></p>
</td>
</tr>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:black">This message came from outside your organization.
<o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span lang="EN-US" style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerEnd</span><span lang="EN-US" style="font-size:1.0pt;font-family:SimSun;color:white"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US">Thank you, that’s far more useful than Lenovo support have been so far!<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Unfortunately Lenovo only support particular versions of scale on their DSSG and they are our only source of the required GNR packages.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">So a couple more questions if you don’t mind!<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<ul style="margin-top:0cm" type="disc">
<li class="MsoListParagraph" style="margin-left:0cm;mso-list:l2 level1 lfo3"><span lang="EN-US">Can you provide a link the APAR or similar I can share with Lenovo?<o:p></o:p></span></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l2 level1 lfo3"><span lang="EN-US">Do you know of any workaround or other impact of this issue? (I think I’ve seen mmheath show false errors because of it). This may help us to decide whether to just
press ahead upgrading to 5.1.5 everywhere if the issue is not present 5.1.5 -> 5.1.5.<o:p></o:p></span></li></ul>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Many thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Luke<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;color:#1F497D;mso-fareast-language:EN-GB">--
</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;color:#1F497D;mso-fareast-language:EN-GB">Luke Sudbery</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;color:#1F497D;mso-fareast-language:EN-GB">Principal Engineer (HPC and Storage).</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;color:#1F497D;mso-fareast-language:EN-GB">Architecture, Infrastructure and Systems</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;color:#1F497D;mso-fareast-language:EN-GB">Advanced Research Computing, IT Services</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;color:#1F497D;mso-fareast-language:EN-GB">Room 132, Computer Centre G5, Elms Road</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;color:#1F497D;mso-fareast-language:EN-GB"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:9.0pt;color:#1F497D;mso-fareast-language:EN-GB">Please note I don’t work on Monday.</span></b><span lang="EN-US"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="mso-fareast-language:EN-GB">From:</span></b><span lang="EN-US" style="mso-fareast-language:EN-GB"> gpfsug-discuss <<a href="mailto:gpfsug-discuss-bounces@gpfsug.org">gpfsug-discuss-bounces@gpfsug.org</a>>
<b>On Behalf Of </b>Gang Qiu<br>
<b>Sent:</b> 21 March 2023 03:49<br>
<b>To:</b> gpfsug main discussion list <<a href="mailto:gpfsug-discuss@gpfsug.org">gpfsug-discuss@gpfsug.org</a>><br>
<b>Subject:</b> [gpfsug-discuss] </span><span style="font-family:"MS Gothic";mso-fareast-language:EN-GB">答复</span><span lang="EN-US" style="mso-fareast-language:EN-GB">: mmvdisk version/communication issues?</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left" width="100%" style="width:100.0%">
<tbody>
<tr>
<td width="10%" style="width:10.0%;border:solid #9C6500 1.0pt;background:#FFEB9C;padding:2.0pt 2.0pt 2.0pt 2.0pt">
<p class="MsoNormal" style="mso-element:frame;mso-element-frame-hspace:2.25pt;mso-element-wrap:around;mso-element-anchor-vertical:paragraph;mso-element-anchor-horizontal:column;mso-height-rule:exactly">
<b><span lang="EN-US" style="color:black;mso-fareast-language:EN-GB">CAUTION:</span></b><span lang="EN-US" style="color:black;mso-fareast-language:EN-GB"> This email originated from outside the organisation. Do not click links or open attachments unless you
recognise the sender and know the content is safe.</span><span lang="EN-US"><o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN"> </span><span lang="EN-US"><o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;mso-fareast-language:ZH-CN"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;mso-fareast-language:ZH-CN"> </span><span lang="EN-US"><o:p></o:p></span></p>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt">It is not mmvdisk issue. Mmvdisk hangs because ‘tslsrecgroup’ command hangs which executed by mmvdisk.</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt">I recreated it on my environment (coexist 5.1.3 or 5.1.4, 5.1.5 with 5.1.2 or previous version)</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt">After further investigation, it should be a known issue which is fixed in 5.1.6 PTF1.
</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt">To resolve this issue, we need to upgrade the version to 5.1.6.1 at least.
</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="line-height:13.5pt;background:white"><span lang="EN-US" style="font-size:10.5pt;color:black;mso-fareast-language:ZH-CN"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="line-height:13.5pt;background:white"><span lang="EN-US" style="font-size:10.5pt;color:black;mso-fareast-language:ZH-CN"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="line-height:13.5pt;background:white"><span lang="EN-US" style="font-size:10.5pt;color:black;mso-fareast-language:ZH-CN">Regards,</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="line-height:13.5pt;background:white"><span lang="EN-US" style="font-size:10.5pt;color:black;mso-fareast-language:ZH-CN">Gang Qiu</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="line-height:13.5pt;background:white"><span lang="EN-US" style="font-size:10.5pt;color:black;mso-fareast-language:ZH-CN">====================================================</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="line-height:13.5pt;background:white"><span lang="EN-US" style="font-size:10.5pt;color:black;mso-fareast-language:ZH-CN">Gang Qiu(</span><span lang="ZH-CN" style="font-size:10.5pt;font-family:SimSun;color:black;mso-fareast-language:ZH-CN">邱钢</span><span lang="EN-US" style="font-size:10.5pt;color:black;mso-fareast-language:ZH-CN">)</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="line-height:13.5pt;background:white"><span lang="EN-US" style="font-size:10.5pt;color:black;mso-fareast-language:ZH-CN">Spectrum Scale Development - ECE</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="line-height:13.5pt;background:white"><span lang="EN-US" style="font-size:10.5pt;color:black;mso-fareast-language:ZH-CN">IBM China Systems Lab</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="line-height:13.5pt;background:white"><span lang="EN-US" style="font-size:10.5pt;color:black;mso-fareast-language:ZH-CN">Mobile: +86-18612867902</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="line-height:13.5pt;background:white"><span lang="EN-US" style="font-size:10.5pt;color:black;mso-fareast-language:ZH-CN">====================================================</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;mso-fareast-language:ZH-CN"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;mso-fareast-language:ZH-CN"> </span><span lang="EN-US"><o:p></o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span lang="EN-US" style="font-family:SimSun;color:black">发件人</span></b><b><span lang="EN-US" style="color:black">:</span></b><span lang="EN-US" style="color:black"> gpfsug-discuss <<a href="mailto:gpfsug-discuss-bounces@gpfsug.org">gpfsug-discuss-bounces@gpfsug.org</a>>
</span><span lang="EN-US" style="font-family:SimSun;color:black">代表</span><span lang="EN-US" style="color:black"> Luke Sudbery <<a href="mailto:l.r.sudbery@bham.ac.uk">l.r.sudbery@bham.ac.uk</a>><br>
</span><b><span lang="EN-US" style="font-family:SimSun;color:black">日期</span></b><b><span lang="EN-US" style="color:black">:</span></b><span lang="EN-US" style="color:black">
</span><span lang="EN-US" style="font-family:SimSun;color:black">星期六</span><span lang="EN-US" style="color:black">, 2023</span><span lang="EN-US" style="font-family:SimSun;color:black">年</span><span lang="EN-US" style="color:black">3</span><span lang="EN-US" style="font-family:SimSun;color:black">月</span><span lang="EN-US" style="color:black">18</span><span lang="EN-US" style="font-family:SimSun;color:black">日</span><span lang="EN-US" style="color:black">
00:06<br>
</span><b><span lang="EN-US" style="font-family:SimSun;color:black">收件人</span></b><b><span lang="EN-US" style="color:black">:</span></b><span lang="EN-US" style="color:black"> gpfsug main discussion list <<a href="mailto:gpfsug-discuss@gpfsug.org">gpfsug-discuss@gpfsug.org</a>><br>
</span><b><span lang="EN-US" style="font-family:SimSun;color:black">主题</span></b><b><span lang="EN-US" style="color:black">:</span></b><span lang="EN-US" style="color:black"> [EXTERNAL] Re: [gpfsug-discuss] mmvdisk version/communication issues?</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span lang="EN-US" style="font-size:1.0pt;color:white">On further investigation the command does eventually complete, after 11 minutes rather than a couple of seconds. [root@ rds-pg-dssg01 ~]# time mmvdisk
pdisk list --rg rds_er_dssg02 --not-ok mmvdisk: All pdisks of recovery group 'rds_er_dssg02'
</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span lang="EN-US" style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerStart</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;border-radius:4px">
<tbody>
<tr>
<td style="padding:12.0pt 0cm 12.0pt 0cm">
<table class="MsoNormalTable" border="1" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;background:#D0D8DC;border:none;border-top:solid #90A4AE 3.0pt">
<tbody>
<tr>
<td valign="top" style="border:none;padding:0cm 7.5pt 3.75pt 4.5pt">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left">
<tbody>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black">This Message Is From an External Sender
</span></b><span lang="EN-US"><o:p></o:p></span></p>
</td>
</tr>
<tr>
<td style="padding:3.0pt 6.0pt 3.0pt 6.0pt">
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:black">This message came from outside your organization.
</span><span lang="EN-US"><o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<div>
<p class="MsoNormal" style="mso-line-height-alt:.75pt"><span lang="EN-US" style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerEnd</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US">On further investigation the command does eventually complete, after 11 minutes rather than a couple of seconds.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black;mso-fareast-language:EN-GB">[root@rds-pg-dssg01 ~]# time mmvdisk pdisk list --rg rds_er_dssg02 --not-ok</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black;mso-fareast-language:EN-GB">mmvdisk: All pdisks of recovery group 'rds_er_dssg02' are ok.</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black;mso-fareast-language:EN-GB"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black;mso-fareast-language:EN-GB">real 11m14.106s</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black;mso-fareast-language:EN-GB">user 0m1.430s</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black;mso-fareast-language:EN-GB">sys 0m0.555s</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black;mso-fareast-language:EN-GB">[root@rds-pg-dssg01 ~]#</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Looking at the process tree, the bits that hang are:<o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black">[root@rds-pg-dssg01 ~]# time tslsrecgroup rds_er_dssg02 -Y --v2 --failure-domain</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black">Failed to connect to file system daemon: Connection timed out</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black">real 5m30.181s</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black">user 0m0.001s</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black">sys 0m0.003s</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">and then<o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black">[root@rds-pg-dssg01 ~]# time tslspdisk --recovery-group rds_er_dssg02 --notOK</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black">Failed to connect to file system daemon: Connection timed out</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black">real 5m30.247s</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black">user 0m0.003s</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black">sys 0m0.002s</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black">[root@rds-pg-dssg01 ~]#</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Which adds up to the 11 minutes.... then it does something else and just works. Or maybe it <i>doesn't work</i> and just wouldn't report any failed disks is there were any….<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">While hanging, the ts commands appear to be LISTENing, not attempting to make connections:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black">[root@rds-pg-dssg01 ~]# pidof tslspdisk</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black">2156809</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black">[root@rds-pg-dssg01 ~]# netstat -apt | grep 2156809</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black">tcp 0 0 0.0.0.0:60000 0.0.0.0:* LISTEN 2156809/tslspdisk</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black">[root@rds-pg-dssg01 ~]#</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Port 60000 is the lowest of our </span><span class="CodeChar"><span lang="EN-US" style="color:black">tscCmdPortRange</span></span><span lang="EN-US">.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Don’t know if that helps anyone….<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Cheers,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Luke<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;color:#1F497D;mso-fareast-language:EN-GB">--
</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;color:#1F497D;mso-fareast-language:EN-GB">Luke Sudbery</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;color:#1F497D;mso-fareast-language:EN-GB">Principal Engineer (HPC and Storage).</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;color:#1F497D;mso-fareast-language:EN-GB">Architecture, Infrastructure and Systems</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;color:#1F497D;mso-fareast-language:EN-GB">Advanced Research Computing, IT Services</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;color:#1F497D;mso-fareast-language:EN-GB">Room 132, Computer Centre G5, Elms Road</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;color:#1F497D;mso-fareast-language:EN-GB"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:9.0pt;color:#1F497D;mso-fareast-language:EN-GB">Please note I don’t work on Monday.</span></b><span lang="EN-US"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="mso-fareast-language:EN-GB">From:</span></b><span lang="EN-US" style="mso-fareast-language:EN-GB"> gpfsug-discuss <<a href="mailto:gpfsug-discuss-bounces@gpfsug.org">gpfsug-discuss-bounces@gpfsug.org</a>>
<b>On Behalf Of </b>Luke Sudbery<br>
<b>Sent:</b> 17 March 2023 15:11<br>
<b>To:</b> <a href="mailto:gpfsug-discuss@gpfsug.org">gpfsug-discuss@gpfsug.org</a><br>
<b>Subject:</b> [gpfsug-discuss] mmvdisk version/communication issues?</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Hello,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">We 3 Lenovo DSSG “Building Blocks” as they call them – 2x GNR server pairs.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">We’ve just upgraded the 1st of them from 3.2a (GPFS 5.1.1.0) to 4.3a (5.1.5.1 efix 20).<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Now the older systems can’t communicated with the newer in certain circumstances, specifically querying recovery groups hosted on other servers.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">It works old->old, new->old and new->new but not old->new.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">If fairly sure it is not a TCP comms problem. I can ssh between the node as root and as the GPFS
</span><span class="CodeChar"><span lang="EN-US" style="color:black">sudoUser</span></span><span lang="EN-US">. Port 1191 and the
</span><span class="CodeChar"><span lang="EN-US" style="color:black">tscCmdPortRange</span></span><span lang="EN-US"> are open and accessible in both direction between the nodes. There are connections present between the nodes in netstat and in
</span><span class="CodeChar"><span lang="EN-US" style="color:black">mmfsd.latest.log</span></span><span lang="EN-US">. No pending message (to that node) in
</span><span class="CodeChar"><span lang="EN-US" style="color:black">mmdiag --network.</span></span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">In these examples </span><span class="CodeChar"><span lang="EN-US" style="color:black">rds-er-dssg01/2</span></span><span lang="EN-US"> are upgraded,
</span><span class="CodeChar"><span lang="EN-US" style="color:black">rds-pg-dssg01/2</span></span><span lang="EN-US"> are downlevel:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black">[root@rds-er-dssg01 ~]# mmvdisk pdisk list --rg rds_er_dssg02 --not-ok # New to new</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black">mmvdisk: All pdisks of recovery group 'rds_er_dssg02' are ok.</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black">[root@rds-er-dssg01 ~]# mmvdisk pdisk list --rg rds_pg_dssg02 --not-ok # New to old</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black">mmvdisk: All pdisks of recovery group 'rds_pg_dssg02' are ok.</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black">[root@rds-er-dssg01 ~]#</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black">[root@rds-pg-dssg01 ~]# mmvdisk pdisk list --rg rds_pg_dssg02 --not-ok # Old to old</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black">mmvdisk: All pdisks of recovery group 'rds_pg_dssg02' are ok.</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black">[root@rds-pg-dssg01 ~]# mmvdisk pdisk list --rg rds_er_dssg02 --not-ok # Old to new [HANGS]</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black">^Cmmvdisk: Command failed. Examine previous error messages to determine cause.</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="Code"><span lang="EN-US" style="color:black">[root@rds-pg-dssg01 ~]#</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Has anyone come across this? mmvdisk should work across slightly different versions of 5.1, right? No recovery group, cluster or filesystem versions have been changed yet.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">We will also log a ticket with snaps and more info but wondered if anyone had seen this.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">And while this particular command is not a major issue, we don’t know what else it may affect, before we proceed with the reset of the cluster.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Many thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Luke<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;color:#1F497D;mso-fareast-language:EN-GB">--
</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;color:#1F497D;mso-fareast-language:EN-GB">Luke Sudbery</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;color:#1F497D;mso-fareast-language:EN-GB">Principal Engineer (HPC and Storage).</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;color:#1F497D;mso-fareast-language:EN-GB">Architecture, Infrastructure and Systems</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;color:#1F497D;mso-fareast-language:EN-GB">Advanced Research Computing, IT Services</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;color:#1F497D;mso-fareast-language:EN-GB">Room 132, Computer Centre G5, Elms Road</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;color:#1F497D;mso-fareast-language:EN-GB"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:9.0pt;color:#1F497D;mso-fareast-language:EN-GB">Please note I don’t work on Monday.</span></b><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
</div>
<span>_______________________________________________</span><br>
<span>gpfsug-discuss mailing list</span><br>
<span>gpfsug-discuss at gpfsug.org</span><br>
<span>http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org</span><br>
</div>
</blockquote>
</body>
</html>