<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Courier;
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style></head><body lang=en-CH link="#0563C1" vlink="#954F72" style='word-wrap:break-word'><div class=WordSection1><p class=MsoNormal><span lang=EN-US style='mso-fareast-language:EN-US'>Hello,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='mso-fareast-language:EN-US'>I just noticed:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='mso-fareast-language:EN-US'>Maybe mmrestripefs does some extra processing on snapshots? The output looks much more as expected on filesystems with no snapshots present, both number of inodes and MB data processed allow to estimate the remaining runtime. Unfortunately all our large production filesystems use snapshots …<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='mso-fareast-language:EN-US'>Cheers,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='mso-fareast-language:EN-US'>Heiner<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:9.0pt;font-family:Courier;mso-fareast-language:EN-US'> 1.23 % complete on Tue May 18 21:07:10 2021 ( 37981765 inodes with total 328617 MB data processed)<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:9.0pt;font-family:Courier;mso-fareast-language:EN-US'> 1.25 % complete on Tue May 18 21:17:44 2021 ( 39439584 inodes with total 341032 MB data processed)<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:9.0pt;font-family:Courier;mso-fareast-language:EN-US'> 100.00 % complete on Tue May 18 21:22:44 2021 ( 41312000 inodes with total 356088 MB data processed). <<<< # of inodes matches # of allocated inodes, MB processed matches used metadata disk space<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:9.0pt;font-family:Courier;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:9.0pt;font-family:Courier;mso-fareast-language:EN-US'>Scan completed successfully.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:9.0pt;font-family:Courier;mso-fareast-language:EN-US'># mmdf fsxxx -F<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:9.0pt;font-family:Courier;mso-fareast-language:EN-US'>Inode Information<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:9.0pt;font-family:Courier;mso-fareast-language:EN-US'>-----------------<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:9.0pt;font-family:Courier;mso-fareast-language:EN-US'>Total number of used inodes in all Inode spaces: 29007227<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:9.0pt;font-family:Courier;mso-fareast-language:EN-US'>Total number of free inodes in all Inode spaces: 12304773<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:9.0pt;font-family:Courier;mso-fareast-language:EN-US'>Total number of allocated inodes in all Inode spaces: 41312000 <<<<<<<<<< allocated inodes<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:9.0pt;font-family:Courier;mso-fareast-language:EN-US'>Total of Maximum number of inodes in all Inode spaces: 87323648<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:9.0pt;font-family:Courier;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:9.0pt;font-family:Courier;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:9.0pt;font-family:Courier;mso-fareast-language:EN-US'>]# mmdf fsxxx -m --block-size M<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:9.0pt;font-family:Courier;mso-fareast-language:EN-US'>disk disk size failure holds holds free in MB free in MB<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:9.0pt;font-family:Courier;mso-fareast-language:EN-US'>name in MB group metadata data in full blocks in fragments<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:9.0pt;font-family:Courier;mso-fareast-language:EN-US'>--------------- ------------- -------- -------- ----- -------------------- -------------------<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:9.0pt;font-family:Courier;mso-fareast-language:EN-US'>Disks in storage pool: system (Maximum disk size allowed is 4.13 TB)<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:9.0pt;font-family:Courier;mso-fareast-language:EN-US'> ------------- -------------------- -------------------<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:9.0pt;font-family:Courier;mso-fareast-language:EN-US'>(pool total) 2425152 2067720 ( 85%) 1342 ( 0%). <<<<<< 356190MB used<o:p></o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span style='font-size:12.0pt;color:black'>From: </span></b><span style='font-size:12.0pt;color:black'><gpfsug-discuss-bounces@spectrumscale.org> on behalf of "Billich Heinrich Rainer (ID SD)" <heinrich.billich@id.ethz.ch><br><b>Reply to: </b>gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br><b>Date: </b>Friday, 28 May 2021 at 10:25<br><b>To: </b>gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br><b>Subject: </b>[gpfsug-discuss] Mmrestripefs -R --metadat-only - how to estimate remaining execution time<o:p></o:p></span></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><p class=MsoNormal><span lang=EN-US>Hello,</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>I want to estimate how much longer a running </span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>mmrestripefs -R –metadata-only –qos maintenance</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>job will take to finish. We did switch from ‘-m 1’ to ‘-m 2 ’ and now run mmrestripefs to get the second copy of all metadata. I know that the % values in the output are useless, but now it looks like the number of inodes processed is of no use, too? The filesystem has about 1e9 allocated inodes, but the mmrestripefs reports up to 7e9 inodes processed? The number of inodes reported increases heavily since it passed the number of allocated inodes???</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> 4.41 % complete on Fri May 28 09:22:05 2021 (7219160122 inodes with total 8985067 MB data processed)</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>Should I consider the ‘MB data processed’ instead – will the job finish when all used metadata disk space is processed? </span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>I see little iops on the metadata disks and wonder if something is stuck, I’m not sure if it makes sense to wait any longer. But without an estimate on how much longer the job will take I hesitate to restart. A restart will need to scan all metadata again … Metadata is on SSD or NVMe, disks iops never were limiting. I turned qos off on the filesystem, previously I restricted the iops. But I don’t see any increase in iops.</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>Thank you. I know this is an old issue, but still I would strongly welcome any advice or comment.</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>Cheers,</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>Heiner</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>I’m aware of RFE ID:150409 mmrestripefs % complete metric is near useless. I’m not sure if it covers the metadata-only case, too.</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>Inode Information</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>-----------------</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>Total number of used inodes in all Inode spaces: 735005692</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>Total number of free inodes in all Inode spaces: 335462660</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>Total number of allocated inodes in all Inode spaces: 1070468352</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>Total of Maximum number of inodes in all Inode spaces: 1223782912</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>Mmrestripefs output:</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>START Mon May 24 20:27:25 CEST 2021</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>Scanning file system metadata, phase 1 ...</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>100 % complete on Mon May 24 21:49:27 2021</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>Scan completed successfully.</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>Scanning file system metadata, phase 2 ...</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>100 % complete on Mon May 24 21:49:28 2021</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>Scanning file system metadata for data storage pool</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> 90 % complete on Mon May 24 21:49:34 2021</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>100 % complete on Mon May 24 21:49:35 2021</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>Scanning file system metadata for Capacity storage pool</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> 49 % complete on Mon May 24 21:49:41 2021</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>100 % complete on Mon May 24 21:49:46 2021</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>Scan completed successfully.</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>Scanning file system metadata, phase 3 ...</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>Scan completed successfully.</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>Scanning file system metadata, phase 4 ...</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>100 % complete on Mon May 24 21:49:48 2021</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>Scan completed successfully.</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>Scanning file system metadata, phase 5 ...</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>100 % complete on Mon May 24 21:49:52 2021</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>Scan completed successfully.</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>Scanning user file metadata ...</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> 0.01 % complete on Mon May 24 21:50:13 2021 ( 613108 inodes with total 76693 MB data processed)</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> 0.01 % complete on Mon May 24 21:50:33 2021 ( 1090048 inodes with total 80495 MB data processed)</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> 0.01 % complete on Mon May 24 21:50:53 2021 ( 1591808 inodes with total 84576 MB data processed)</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>…</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> 3.97 % complete on Thu May 27 22:01:02 2021 (1048352497 inodes with total 8480467 MB data processed)</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> 3.99 % complete on Thu May 27 22:30:18 2021 (1050254855 inodes with total 8495969 MB data processed)</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> 4.01 % complete on Thu May 27 22:59:39 2021 (1052304294 inodes with total 8512683 MB data processed)</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> 4.03 % complete on Thu May 27 23:29:11 2021 (1055390220 inodes with total 8537615 MB data processed)</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> 4.05 % complete on Thu May 27 23:58:58 2021 (1059333989 inodes with total 8568871 MB data processed)</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> 4.07 % complete on Fri May 28 00:28:48 2021 (1064728403 inodes with total 8611605 MB data processed)</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> 4.09 % complete on Fri May 28 00:58:50 2021 (1067749260 inodes with total 8636120 MB data processed). <<< approximately number of allocated inodes</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> 4.11 % complete on Fri May 28 01:29:00 2021 (1488665433 inodes with total 8661588 MB data processed) <<< whats going??</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> 4.13 % complete on Fri May 28 01:59:23 2021 (1851124480 inodes with total 8682324 MB data processed)</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> 4.15 % complete on Fri May 28 02:29:55 2021 (1885948840 inodes with total 8700082 MB data processed)</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> 4.17 % complete on Fri May 28 03:00:38 2021 (2604503808 inodes with total 8724069 MB data processed)</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> 4.19 % complete on Fri May 28 03:31:38 2021 (2877196260 inodes with total 8746504 MB data processed)</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> 4.21 % complete on Fri May 28 04:02:38 2021 (2933166080 inodes with total 8762555 MB data processed)</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> 4.23 % complete on Fri May 28 04:33:48 2021 (2956295936 inodes with total 8782298 MB data processed)</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> 4.25 % complete on Fri May 28 05:05:09 2021 (3628799151 inodes with total 8802452 MB data processed)</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> 4.27 % complete on Fri May 28 05:36:40 2021 (3970093965 inodes with total 8823885 MB data processed)</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> 4.29 % complete on Fri May 28 06:08:20 2021 (4012553472 inodes with total 8841407 MB data processed)</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> 4.31 % complete on Fri May 28 06:40:11 2021 (4029545087 inodes with total 8858676 MB data processed)</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> 4.33 % complete on Fri May 28 07:12:11 2021 (6080613874 inodes with total 8889395 MB data processed)</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> 4.35 % complete on Fri May 28 07:44:21 2021 (6146937531 inodes with total 8907253 MB data processed)</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> 4.37 % complete on Fri May 28 08:16:45 2021 (6167408718 inodes with total 8925236 MB data processed)</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> 4.39 % complete on Fri May 28 08:49:18 2021 (7151592448 inodes with total 8968126 MB data processed)</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> 4.41 % complete on Fri May 28 09:22:05 2021 (7219160122 inodes with total 8985067 MB data processed)</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US># mmdf fsxxxx -m --block-size M</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>disk disk size failure holds holds free in MB free in MB</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>name in MB group metadata data in full blocks in fragments</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>--------------- ------------- -------- -------- ----- -------------------- -------------------</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>Disks in storage pool: system (Maximum disk size allowed is 32.20 TB)</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>fsxxxx_rg07b_1 2861295 3 yes no 1684452 ( 59%) 35886 ( 1%). << 1’140’957MB used – old nsd</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>fsxxxx_rg07a_1 2861295 3 yes no 1685002 ( 59%) 35883 ( 1%</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>fsxxxx_rg03b_1 2861295 3 yes no 1681515 ( 59%) 35627 ( 1%)</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>fsxxxx_rg03a_1 2861295 3 yes no 1680859 ( 59%) 35651 ( 1%)</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>RG003LG004VS015 2046239 12 yes no 904369 ( 44%) 114 ( 0%) << 1’141’756MB used – newly added nsd</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>RG003LG003VS015 2046239 12 yes no 904291 ( 44%) 118 ( 0%)</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>RG003LG002VS015 2046239 12 yes no 903632 ( 44%) 114 ( 0%)</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>RG003LG001VS015 2046239 12 yes no 903247 ( 44%) 115 ( 0%)</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> ------------- -------------------- -------------------</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>(pool total) 19630136 10347367 ( 53%) 143503 ( 1%)</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>We run spectrum scale 5.0.5.4/5.0.5.5 on Power. </span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>Filesystem:</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>-f 32768 Minimum fragment (subblock) size in bytes</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>-i 4096 Inode size in bytes</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>-I 32768 Indirect block size in bytes</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>-m 2 Default number of metadata replicas</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>-M 2 Maximum number of metadata replicas</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>-r 1 Default number of data replicas</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>-R 2 Maximum number of data replicas</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>-j scatter Block allocation type</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>-V 23.00 (5.0.5.0) Current file system version</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> 15.01 (4.2.0.0) Original file system version</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>-L 33554432 Logfile size</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>--subblocks-per-full-block 32 Number of subblocks per full block</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> </span><o:p></o:p></p></div></body></html>