<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-GB" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">I have a question about file-systems with replicated an non replicated data.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We have a file-system where metadata is set to copies=2 and data copies=2, we then use a placement policy to selectively replicate some data only once based on file-set. We also place the non-replicated data
into a specific pool (6tnlsas) to ensure we know where it is placed.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">My understanding was that in doing this, if we took the disks with the non replicated data offline, we’d still have the FS available for users as the metadata is replicated. Sure accessing a non-replicated
data file would give an IO error, but the rest of the FS should be up.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We had a situation today where we wanted to take stg01 offline today, so tried using mmchdisk stop -d …. Once we got to about disk stg01-01_12_12, GPFS would refuse to stop any more disks and complain about
too many disks, similarly if we shutdown the NSD servers hosting the disks, the filesystem would have an SGPanic and force unmount.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">First, am I correct in thinking that a FS with non-replicated data, but replicated metadata should still be accessible (not the non-replicated data) when the LUNS hosting it are down?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">If so, any suggestions why my FS is panic-ing when we take down the one set of disks?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I thought at first we had some non-replicated metadata, tried a mmrestripefs -R –metadata-only to force it to ensure 2 replicas, but this didn’t help.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Running 5.0.0.2 on the NSD server nodes.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">(First time we went round this we didn’t have a FS descriptor disk, but you can see below that we added this)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Simon<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[root@nsd01 ~]# mmlsdisk castles -L<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">disk driver sector failure holds holds storage<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">name type size group metadata data status availability disk id pool remarks<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">------------ -------- ------ ----------- -------- ----- ------------- ------------ ------- ------------ ---------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">CASTLES_GPFS_DESCONLY01 nsd 512 310 no no ready up 1 system desc<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_3_3 nsd 4096 210 no yes ready down 4 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_4_4 nsd 4096 210 no yes ready down 5 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_5_5 nsd 4096 210 no yes ready down 6 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_6_6 nsd 4096 210 no yes ready down 7 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_7_7 nsd 4096 210 no yes ready down 8 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_8_8 nsd 4096 210 no yes ready down 9 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_9_9 nsd 4096 210 no yes ready down 10 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_10_10 nsd 4096 210 no yes ready down 11 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_11_11 nsd 4096 210 no yes ready down 12 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_12_12 nsd 4096 210 no yes ready down 13 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_13_13 nsd 4096 210 no yes ready down 14 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_14_14 nsd 4096 210 no yes ready down 15 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_15_15 nsd 4096 210 no yes ready down 16 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_16_16 nsd 4096 210 no yes ready down 17 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_17_17 nsd 4096 210 no yes ready down 18 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_18_18 nsd 4096 210 no yes ready down 19 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_19_19 nsd 4096 210 no yes ready down 20 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_20_20 nsd 4096 210 no yes ready down 21 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_21_21 nsd 4096 210 no yes ready down 22 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_ssd_54_54 nsd 4096 210 yes no ready down 23 system<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_ssd_56_56 nsd 4096 210 yes no ready down 24 system<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg02-01_0_0 nsd 4096 110 no yes ready up 25 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg02-01_1_1 nsd 4096 110 no yes ready up 26 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg02-01_2_2 nsd 4096 110 no yes ready up 27 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg02-01_3_3 nsd 4096 110 no yes ready up 28 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg02-01_4_4 nsd 4096 110 no yes ready up 29 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg02-01_5_5 nsd 4096 110 no yes ready up 30 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg02-01_6_6 nsd 4096 110 no yes ready up 31 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg02-01_7_7 nsd 4096 110 no yes ready up 32 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg02-01_8_8 nsd 4096 110 no yes ready up 33 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg02-01_9_9 nsd 4096 110 no yes ready up 34 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg02-01_10_10 nsd 4096 110 no yes ready up 35 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg02-01_11_11 nsd 4096 110 no yes ready up 36 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg02-01_12_12 nsd 4096 110 no yes ready up 37 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg02-01_13_13 nsd 4096 110 no yes ready up 38 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg02-01_14_14 nsd 4096 110 no yes ready up 39 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg02-01_15_15 nsd 4096 110 no yes ready up 40 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg02-01_16_16 nsd 4096 110 no yes ready up 41 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg02-01_17_17 nsd 4096 110 no yes ready up 42 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg02-01_18_18 nsd 4096 110 no yes ready up 43 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg02-01_19_19 nsd 4096 110 no yes ready up 44 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg02-01_20_20 nsd 4096 110 no yes ready up 45 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg02-01_21_21 nsd 4096 110 no yes ready up 46 6tnlsas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg02-01_ssd_22_22 nsd 4096 110 yes no ready up 47 system desc<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg02-01_ssd_23_23 nsd 4096 110 yes no ready up 48 system<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg02-01_ssd_24_24 nsd 4096 110 yes no ready up 49 system<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg02-01_ssd_25_25 nsd 4096 110 yes no ready up 50 system<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_22_22 nsd 4096 210 no yes ready up 51 6tnlsasnonrepl desc<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_23_23 nsd 4096 210 no yes ready up 52 6tnlsasnonrepl<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_24_24 nsd 4096 210 no yes ready up 53 6tnlsasnonrepl<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_25_25 nsd 4096 210 no yes ready up 54 6tnlsasnonrepl<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_26_26 nsd 4096 210 no yes ready up 55 6tnlsasnonrepl<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_27_27 nsd 4096 210 no yes ready up 56 6tnlsasnonrepl<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_31_31 nsd 4096 210 no yes ready up 58 6tnlsasnonrepl<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_32_32 nsd 4096 210 no yes ready up 59 6tnlsasnonrepl<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_33_33 nsd 4096 210 no yes ready up 60 6tnlsasnonrepl<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_34_34 nsd 4096 210 no yes ready up 61 6tnlsasnonrepl<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_35_35 nsd 4096 210 no yes ready up 62 6tnlsasnonrepl<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_36_36 nsd 4096 210 no yes ready up 63 6tnlsasnonrepl<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_37_37 nsd 4096 210 no yes ready up 64 6tnlsasnonrepl<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_38_38 nsd 4096 210 no yes ready up 65 6tnlsasnonrepl<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_39_39 nsd 4096 210 no yes ready up 66 6tnlsasnonrepl<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_40_40 nsd 4096 210 no yes ready up 67 6tnlsasnonrepl<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_41_41 nsd 4096 210 no yes ready up 68 6tnlsasnonrepl<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_42_42 nsd 4096 210 no yes ready up 69 6tnlsasnonrepl<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_43_43 nsd 4096 210 no yes ready up 70 6tnlsasnonrepl<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_44_44 nsd 4096 210 no yes ready up 71 6tnlsasnonrepl<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_45_45 nsd 4096 210 no yes ready up 72 6tnlsasnonrepl<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_46_46 nsd 4096 210 no yes ready up 73 6tnlsasnonrepl<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_47_47 nsd 4096 210 no yes ready up 74 6tnlsasnonrepl<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_48_48 nsd 4096 210 no yes ready up 75 6tnlsasnonrepl<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_49_49 nsd 4096 210 no yes ready up 76 6tnlsasnonrepl<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_50_50 nsd 4096 210 no yes ready up 77 6tnlsasnonrepl<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">stg01-01_51_51 nsd 4096 210 no yes ready up 78 6tnlsasnonrepl<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Number of quorum disks: 3<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Read quorum value: 2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Write quorum value: 2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
</body>
</html>