<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
tt
{mso-style-priority:99;
font-family:"Courier New";}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:325398175;
mso-list-template-ids:-1713871784;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:"Courier New";
mso-bidi-font-family:"Times New Roman";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1
{mso-list-id:913467548;
mso-list-template-ids:-469050450;}
@list l1:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:"Courier New";
mso-bidi-font-family:"Times New Roman";}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l2
{mso-list-id:1815559179;
mso-list-template-ids:1743922678;}
@list l2:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:"Courier New";
mso-bidi-font-family:"Times New Roman";}
@list l2:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l2:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l2:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l2:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l2:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l2:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l2:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=en-CH link=blue vlink=purple style='word-wrap:break-word'><div class=WordSection1><p class=MsoNormal><span lang=EN-US style='mso-fareast-language:EN-US'>Hello,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='mso-fareast-language:EN-US'>Thank you for your response. I opened a case with IBM and what we found is – as I understand:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='mso-fareast-language:EN-US'>If you change the storage pool of a file which has copy in a snapshot the inode is dublicated (copy on write) – the data pool is part of the inode and its preserved in the snapshot, the snapshot get’s its own inode version. So even if the file’s blocks actually did move to storage pool B the snapshot still shows the previous storage pool A. Once the snapshots get deleted the additional metadata space is freed. Probably backup software does save the storage pool, too. Hence the snapshot must preserve the original value.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='mso-fareast-language:EN-US'>You can easily verify with mmlsattr that the snapshot version and the plain version show different storage pools.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='mso-fareast-language:EN-US'>I saw a bout 4500 bytes extra space required for each inode when I did run the migration rule which changed the storage pool.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='mso-fareast-language:EN-US'>Kind regards,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='mso-fareast-language:EN-US'>Heiner<o:p></o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span style='font-size:12.0pt;color:black'>From: </span></b><span style='font-size:12.0pt;color:black'><gpfsug-discuss-bounces@spectrumscale.org> on behalf of IBM Spectrum Scale <scale@us.ibm.com><br><b>Reply to: </b>gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br><b>Date: </b>Wednesday, 9 June 2021 at 14:55<br><b>To: </b>gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br><b>Cc: </b>"gpfsug-discuss-bounces@spectrumscale.org" <gpfsug-discuss-bounces@spectrumscale.org><br><b>Subject: </b>Re: [gpfsug-discuss] Metadata usage almost doubled after policy run with migration rule<o:p></o:p></span></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><p><span style='font-size:10.0pt'>Hi Billich,</span><br><br><span style='font-size:10.0pt;color:blue'>></span><span style='color:blue'>Or maybe illplaced files use larger inodes? Looks like for each used inode we increased by about 4k: 400M inodes, 1.6T increase in size</span><br><br>Basically a migration policy run with -I defer would just simply mark the files as illPlaced which would not cause metadata extension for such files(e.g., inode size is fixed after file system creation). Instead, I'm just wondering about your placement rules, which are existing rules or newly installed rules? Which could set EAs to newly created files and may cause increased metadata size. Also any new EAs are inserted for files?<br><br><br><span style='font-size:10.0pt'>Regards, The Spectrum Scale (GPFS) team<br><br>------------------------------------------------------------------------------------------------------------------<br>If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at <a href="https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479">https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479</a>. <br><br>If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. <br><br>The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team.</span><br><br><img border=0 width=16 height=16 style='width:.1666in;height:.1666in' id="_x0000_i1026" src="cid:image001.gif@01D76380.17583760" alt="Inactive hide details for "Billich Heinrich Rainer (ID SD)" ---2021/06/08 05:19:32 PM--- Hello,"><span style='font-size:10.0pt;color:#424282'>"Billich Heinrich Rainer (ID SD)" ---2021/06/08 05:19:32 PM--- Hello,</span><br><br><span style='font-size:10.0pt;color:#5F5F5F'>From: </span><span style='font-size:10.0pt'>"Billich Heinrich Rainer (ID SD)" <heinrich.billich@id.ethz.ch></span><br><span style='font-size:10.0pt;color:#5F5F5F'>To: </span><span style='font-size:10.0pt'>gpfsug main discussion list <gpfsug-discuss@spectrumscale.org></span><br><span style='font-size:10.0pt;color:#5F5F5F'>Date: </span><span style='font-size:10.0pt'>2021/06/08 05:19 PM</span><br><span style='font-size:10.0pt;color:#5F5F5F'>Subject: </span><span style='font-size:10.0pt'>[EXTERNAL] [gpfsug-discuss] Metadata usage almost doubled after policy run with migration rule</span><br><span style='font-size:10.0pt;color:#5F5F5F'>Sent by: </span><span style='font-size:10.0pt'>gpfsug-discuss-bounces@spectrumscale.org</span><o:p></o:p></p><div class=MsoNormal><hr size=0 width="100%" noshade style='color:#8091A5' align=left></div><p class=MsoNormal><br><br><br><br>Hello,<br><br>A policy run with ‘-I defer’ and a placement rule did almost double the metadata usage of a filesystem. This filled the metadata disks to a critical level. I would like to understand if this is to be expected and ‘as designed’ or if I face some issue or bug. <br> I hope a subsequent run of ‘mmrstripefs -p’ will reduce the metadata usage again.<br>Thank you<br><br><br>I want to move all data to a new storage pool and did run a policy like<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>RULE 'migrate_to_Data'<br> MIGRATE<br> WEIGHT(0)<o:p></o:p></p><p class=MsoNormal> TO POOL 'Data'<br><br>for each fileset with <br><br> mmapplypolicy -I defer<br><br>Next I want to actually move the data with <br><br> mmrestripefs -p<br><br>After the policy run metadata usage increased from 2.06TiB to 3.53TiB and filled the available metadata space by >90%. This is somewhat surprising. Will the following run of ‘mmrestripefs -p’ reduce the usage again, when the files are not illplaced any more? The number of used Inodes did not change noticeably during the policy run. <br><br>Or maybe illplaced files use larger inodes? Looks like for each used inode we increased by about 4k: 400M inodes, 1.6T increase in size<br><br>Thank you,<br><br>Heiner<br><br>Some details<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'># mmlsfs fsxxxx -f -i -B -I -m -M -r -R -V<br>flag value description<br>------------------- ------------------------ -----------------------------------<br>-f 8192 Minimum fragment (subblock) size in bytes (system pool)<br> 32768 Minimum fragment (subblock) size in bytes (other pools)<br>-i 4096 Inode size in bytes<br>-B 1048576 Block size (system pool)<br> 4194304 Block size (other pools)<br>-I 32768 Indirect block size in bytes<br>-m 1 Default number of metadata replicas<br>-M 2 Maximum number of metadata replicas<br>-r 1 Default number of data replicas<br>-R 2 Maximum number of data replicas<br>-V 23.00 (5.0.5.0) Current file system version<o:p></o:p></p><p class=MsoNormal> 19.01 (5.0.1.0) Original file system version<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'>Inode Information<br>-----------------<br>Total number of used inodes in all Inode spaces: 398502837<br>Total number of free inodes in all Inode spaces: 94184267<br>Total number of allocated inodes in all Inode spaces: 492687104<o:p></o:p></p><p class=MsoNormal>Total of Maximum number of inodes in all Inode spaces: 916122880[attachment "smime.p7s" deleted by Hai Zhong HZ Zhou/China/IBM] <tt><span style='font-size:10.0pt'>_______________________________________________</span></tt><span style='font-size:10.0pt;font-family:"Courier New"'><br><tt>gpfsug-discuss mailing list</tt><br><tt>gpfsug-discuss at spectrumscale.org</tt><br><tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></tt><br></span><br><br><br><br><o:p></o:p></p></div></body></html>