<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:Consolas;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:61878861;
mso-list-type:hybrid;
mso-list-template-ids:853472978 894482070 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-text:"%1\.\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:.75in;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:1.25in;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:1.75in;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:2.25in;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:2.75in;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:3.25in;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:3.75in;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:4.25in;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:4.75in;
text-indent:-9.0pt;}
@list l1
{mso-list-id:571933861;
mso-list-type:hybrid;
mso-list-template-ids:987140158 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l1:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l2
{mso-list-id:1467818570;
mso-list-type:hybrid;
mso-list-template-ids:353641626 663664438 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l2:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:.75in;
text-indent:-.25in;
font-family:Consolas;
mso-fareast-font-family:Calibri;}
@list l2:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:1.25in;
text-indent:-.25in;
font-family:"Courier New";}
@list l2:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:1.75in;
text-indent:-.25in;
font-family:Wingdings;}
@list l2:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:2.25in;
text-indent:-.25in;
font-family:Symbol;}
@list l2:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:2.75in;
text-indent:-.25in;
font-family:"Courier New";}
@list l2:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:3.25in;
text-indent:-.25in;
font-family:Wingdings;}
@list l2:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:3.75in;
text-indent:-.25in;
font-family:Symbol;}
@list l2:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:4.25in;
text-indent:-.25in;
font-family:"Courier New";}
@list l2:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:4.75in;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-family:Consolas">Hello GPFS admins! I hope everybody had a great start to the new year so far.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Lately, I’ve had a few of my users get an error similar to:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> error creating file: no space left on device.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">When trying to create even simple files (using Linux `touch` command). However, if they try again in a second or two, the file is created without a problem and they go on about doing their work. I can
never tell when they are likely to get the error message about ‘no space left on device’. The filesystem creates many files in parallel (depending on the usage of the system and movement of files from other sites)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">However, let me first describe our environment a little better. We have a 3 GPFS file systems (home, project, gscratch) on RHEL 6.3 InfiniBand HPC cluster. The version of GPFS is 3.5.0-11. We utilize fileset
quotas (on block limits, not file limits) for each file system. Each user has a home fileset for storing basic configuration files, basic notes, and other small files. Each user belongs to a minimum of one project and the quota is shared between the users
of the project. The gscratch file system is similar to that of the project file system except that files are deleted after ~9 days.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">The partially good news (perhaps) is that the error mentioned above only occurs on the project file system. We’ve at least not observed the error on the home and gscratch file systems. Here’s my initial
investigation so far:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoListParagraph" style="margin-left:.75in;text-indent:-.25in;mso-list:l0 level1 lfo3">
<![if !supportLists]><span style="font-family:Consolas"><span style="mso-list:Ignore">1.)</span></span><![endif]><span style="font-family:Consolas">Checked the fileset quota on one of the experienced filesets:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">--<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"># mmlsquota -j ModMast project<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> Block Limits | File Limits<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Filesystem type KB quota limit in_doubt grace | files quota limit in_doubt grace Remarks<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">project FILESET 953382016 0 16106127360 0 none | 8666828 0 0 0 none<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">--<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">It would seem from the information that the project is indeed well under their quota for their particular project.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoListParagraph" style="margin-left:.75in;text-indent:-.25in;mso-list:l0 level1 lfo3">
<![if !supportLists]><span style="font-family:Consolas"><span style="mso-list:Ignore">2.)</span></span><![endif]><span style="font-family:Consolas">Then I checked the overall file system to see if the capacity/inode is nearly full:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">--<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"># mmdf project<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">disk disk size failure holds holds free KB free KB<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">name in KB group metadata data in full blocks in fragments<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">--------------- ------------- -------- -------- ----- -------------------- -------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Disks in storage pool: system (Maximum disk size allowed is 397 TB)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">U01_L0 15623913472 -1 Yes Yes 7404335104 ( 47%) 667820032 ( 4%)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">U01_L1 15623913472 -1 Yes Yes 7498215424 ( 48%) 642773120 ( 4%)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">U01_L2 15623913472 -1 Yes Yes 7497969664 ( 48%) 642664576 ( 4%)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">U01_L3 15623913472 -1 Yes Yes 7496232960 ( 48%) 644327936 ( 4%)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">U01_L4 15623913472 -1 Yes Yes 7499296768 ( 48%) 640117376 ( 4%)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">U01_L5 15623913472 -1 Yes Yes 7494881280 ( 48%) 644168320 ( 4%)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">U01_L6 15623913472 -1 Yes Yes 7494164480 ( 48%) 643673216 ( 4%)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">U01_L7 15623913472 -1 Yes Yes 7497433088 ( 48%) 639918976 ( 4%)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">U01_L8 15623913472 -1 Yes Yes 7494139904 ( 48%) 645130240 ( 4%)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">U01_L9 15623913472 -1 Yes Yes 7498375168 ( 48%) 639979520 ( 4%)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">U01_L10 15623913472 -1 Yes Yes 7496028160 ( 48%) 641909632 ( 4%)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">U01_L11 15623913472 -1 Yes Yes 7496093696 ( 48%) 643749504 ( 4%)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">U01_L12 15623913472 -1 Yes Yes 7496425472 ( 48%) 641556992 ( 4%)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">U01_L13 15623913472 -1 Yes Yes 7495516160 ( 48%) 643395840 ( 4%)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">U01_L14 15623913472 -1 Yes Yes 7496908800 ( 48%) 642418816 ( 4%)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">U01_L15 15623913472 -1 Yes Yes 7495823360 ( 48%) 643580416 ( 4%)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">U01_L16 15623913472 -1 Yes Yes 7499939840 ( 48%) 641538688 ( 4%)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">U01_L17 15623913472 -1 Yes Yes 7497355264 ( 48%) 642184704 ( 4%)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">U13_L0 2339553280 -1 Yes No 2322395136 ( 99%) 8190848 ( 0%)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">U13_L1 2339553280 -1 Yes No 2322411520 ( 99%) 8189312 ( 0%)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">U13_L12 15623921664 -1 Yes Yes 7799422976 ( 50%) 335150208 ( 2%)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">U13_L13 15623921664 -1 Yes Yes 8002662400 ( 51%) 126059264 ( 1%)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">U13_L14 15623921664 -1 Yes Yes 8001093632 ( 51%) 126107648 ( 1%)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">U13_L15 15623921664 -1 Yes Yes 8001732608 ( 51%) 126167168 ( 1%)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">U13_L16 15623921664 -1 Yes Yes 8000077824 ( 51%) 126240768 ( 1%)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">U13_L17 15623921664 -1 Yes Yes 8001458176 ( 51%) 126068480 ( 1%)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">U13_L18 15623921664 -1 Yes Yes 7998636032 ( 51%) 127111680 ( 1%)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">U13_L19 15623921664 -1 Yes Yes 8001892352 ( 51%) 125148928 ( 1%)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">U13_L20 15623921664 -1 Yes Yes 8001916928 ( 51%) 126187904 ( 1%)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">U13_L21 15623921664 -1 Yes Yes 8002568192 ( 51%) 126591616 ( 1%)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> ------------- -------------------- -------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">(pool total) 442148765696 219305402368 ( 50%) 13078121728 ( 3%)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> ============= ==================== ===================<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">(data) 437469659136 214660595712 ( 49%) 13061741568 ( 3%)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">(metadata) 442148765696 219305402368 ( 50%) 13078121728 ( 3%)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> ============= ==================== ===================<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">(total) 442148765696 219305402368 ( 50%) 13078121728 ( 3%)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Inode Information<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">-----------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Number of used inodes: 133031523<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Number of free inodes: 1186205<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Number of allocated inodes: 134217728<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Maximum number of inodes: 134217728<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">--<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Eureka! From here it seems that the inode capacity is teetering on its limit. I think at this point it would be best to educate our users on not writing millions of small text files as I don’t think it
is possible to adjust the GPFS block size to something lower (block size is currently 4MB). The system was originally targeted at large read/writes from traditional HPC users, but we have now diversified our user base to include other computing areas outside
traditional HPC. Documentation states that if parallel writes are to be done, that a minimum of 5% of the inodes need to be free otherwise performance will suffer. From above, we have less than 1% free which I think is the root of our problem.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Therefore, is there a method to safely increase the maximum inode count and could it be done during operation or should the system be unmounted? I’ve man paged / searched online and found a few hints suggesting
below but was curious about its safety during operation:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> mmchfs project --inode-limit <new_max_inode_count><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">The man page describes that the limit is:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> max_files = total_filesystem_space / (inode_size + subblock_size)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">and the subblock size defined from IBM’s website as 1/32 of the block size (which is 4MB). Therefore, I calculate that the maximum number of inodes I could potentially have is:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"> 3440846425<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Which is approximately 25x the current maximum, so I think there is reason that I can increase the inode count without too much worry. Are there any caveats to my logic here? I’m not saying I’ll increase
it to the maximum value right away because the inode space would take away from some usable capacity of the system.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Thanks for any comments and recommendations. I have a grand size maintenance period coming up due to datacenter power upgrades and I’ll be given ~2 weeks of down time for maintenance which I’m trying to
get all my ducks in line and if I need to do something time consuming with the file systems, I’d like to know ahead of time so I can do it during the maintenance window as I will probably not get another one window for many months after.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Again, thank you all!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Jared Baker<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">ARCC<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><o:p> </o:p></span></p>
</div>
</body>
</html>