<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle20
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:168182435;
mso-list-type:hybrid;
mso-list-template-ids:1013343818 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l1
{mso-list-id:696009860;
mso-list-type:hybrid;
mso-list-template-ids:1614961150 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l1:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">By the way, the command to check the layoutMap option for your pools is “<b>mmlspool <fs_device> all -L</b>”. Has anyone else noticed if this option changed during your GPFS software upgrades?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Here’s how our mmlspool output looked for our lab/test environment under GPFS Version
<b>3.5.0-21</b>:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">Pool:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> name = system<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> poolID = 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> blockSize = 1024 KB<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> usage = metadataOnly<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> maxDiskSize = 16 TB<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> layoutMap =
<b>cluster</b><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> allowWriteAffinity = no<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> writeAffinityDepth = 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> blockGroupFactor = 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">Pool:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> name = writecache<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> poolID = 65537<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> blockSize = 1024 KB<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> usage = dataOnly<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> maxDiskSize = 8.0 TB<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> layoutMap =
<b>scatter</b><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> allowWriteAffinity = no<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> writeAffinityDepth = 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> blockGroupFactor = 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">Pool:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> name = data<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> poolID = 65538<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> blockSize = 1024 KB<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> usage = dataOnly<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> maxDiskSize = 8.2 TB<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> layoutMap =
<b>scatter</b><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> allowWriteAffinity = no<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> writeAffinityDepth = 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> blockGroupFactor = 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Here’s the mmlspool output immediately after the upgrade to
<b>4.1.1-17</b>:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">Pool:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> name = system<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> poolID = 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> blockSize = 1024 KB<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> usage = metadataOnly<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> maxDiskSize = 16 TB<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> layoutMap =
<b>cluster</b><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> allowWriteAffinity = no<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> writeAffinityDepth = 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> blockGroupFactor = 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">Pool:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> name = writecache<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> poolID = 65537<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> blockSize = 1024 KB<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> usage = dataOnly<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> maxDiskSize = 8.0 TB<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> layoutMap =
<b>cluster</b><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> allowWriteAffinity = no<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> writeAffinityDepth = 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> blockGroupFactor = 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D">Pool:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> name = data<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> poolID = 65538<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> blockSize = 1024 KB<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> usage = dataOnly<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> maxDiskSize = 8.2 TB<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> layoutMap =
<b>cluster</b><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> allowWriteAffinity = no<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> writeAffinityDepth = 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:#1F497D"> blockGroupFactor = 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">We also determined the following:<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo3"><![if !supportLists]><span style="font-family:Symbol;color:#1F497D"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="color:#1F497D">The layoutMap option changes back to “scatter” if we revert back to 3.5.0.21. It only changes back after the last node is downgraded.<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo3"><![if !supportLists]><span style="font-family:Symbol;color:#1F497D"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="color:#1F497D">Restarting GPFS under </span>
<span style="color:#1F497D">4.1.1-17</span><span style="color:#1F497D"> (via mmshutdown and mmstartup) has no effect on layoutMap in the lab (as expected). So, a simple restart doesn’t fix the problem.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Our production and lab deployments are using
<b>SLES 11, SP3 (3.0.101-0.47.71-default)</b>.<b><o:p></o:p></b></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Paul C.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">SIG<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> gpfsug-discuss-bounces@spectrumscale.org [mailto:gpfsug-discuss-bounces@spectrumscale.org]
<b>On Behalf Of </b>Caron, Paul<br>
<b>Sent:</b> Friday, March 23, 2018 4:10 PM<br>
<b>To:</b> gpfsug-discuss@spectrumscale.org<br>
<b>Subject:</b> [gpfsug-discuss] Pool layoutMap option changes following GPFS upgrades<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Has anyone run into a situation where the layoutMap option for a pool changes from “scatter” to “cluster” following a GPFS software upgrade? We recently upgraded a file system from 3.5.0.21, to 4.1.1.17, and finally to 4.2.3.6. We noticed
that the layoutMap option for two of our pools changed following the upgrades. We didn’t recreate the file system or any of the pools. Further lab testing has revealed that the layoutMap option change actually occurred during the first upgrade to 4.1.1.17,
and it was simply carried forward to 4.2.3.6. We have a PMR open with IBM on this problem, but they have told us that layoutMap option changes are impossible for existing pools, and that a software upgrade couldn’t do this. I sent the results of my lab testing
today, so I’m hoping to get a better response.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We would rather not have to recreate all the pools, but it is starting to look like that may be the only option to fix this. Also, it’s unclear if this could happen again during future upgrades.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Here’s some additional background.<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l1 level1 lfo2"><![if !supportLists]><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>The “-j” file system is “cluster”<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l1 level1 lfo2"><![if !supportLists]><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>We have a pretty option for the small cluster; just 13 nodes<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l1 level1 lfo2"><![if !supportLists]><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>When reproducing the problem, we noted that the layoutMap option didn’t change until the final node was upgraded<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l1 level1 lfo2"><![if !supportLists]><span style="font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>The layoutMap option changed before running the “mmchconfig release=LATEST” and “mmchfs <fs> -V full” commands, so those don’t seem to be related to the problem<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Paul C.<o:p></o:p></p>
<p class="MsoNormal">SIG<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">
<hr size="2" width="100%" align="center">
</span></div>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:gray"><br>
IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction,
disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security
or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments
is free of viruses.</span><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p></o:p></span></p>
</div>
<br>
<hr>
<font face="Arial" color="Gray" size="1"><br>
IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction,
disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security
or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments
is free of viruses.<br>
</font>
</body>
</html>