<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">I am having problems with GPFS servers running out of memory.<o:p></o:p></p>
<p class="MsoNormal">We have an open PMR for this, however if anyone has seen this or has any ideas I would be grateful for a heads up.<o:p></o:p></p>
<p class="MsoNormal">Servers have 128 Gbytes f RAM, kernel 2.6.32-573.18.1.el6.x86_64, GPFS version 4.2.3.4<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In the latest incident the free memory went to below 1Gbyte, and we started to have processes killed, including our monitoring setup.<o:p></o:p></p>
<p class="MsoNormal">I shut down GPFS on that server and /proc/meminfo still shows:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Slab: 111192296 kB<o:p></o:p></p>
<p class="MsoNormal">SReclaimable: 29020 kB<o:p></o:p></p>
<p class="MsoNormal">SUnreclaim: 111163276 kB<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Am I barking up a wrong tree here and pointing the finger at GPFS? Something is causing the scsi_data_buffer slab memory usage (see below).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">One thing I did yesterday was to change the disk scheduler for each disk from cfq to dealine (as recommended in the tuning guide)<o:p></o:p></p>
<p class="MsoNormal">However the server was already in short memory at that point.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Slabtop shows<o:p></o:p></p>
<p class="MsoNormal">Active / Total Objects (% used) : -306803185 / -306722574 (100.0%)<o:p></o:p></p>
<p class="MsoNormal">Active / Total Slabs (% used) : 27749714 / 27749719 (100.0%)<o:p></o:p></p>
<p class="MsoNormal">Active / Total Caches (% used) : 115 / 198 (58.1%)<o:p></o:p></p>
<p class="MsoNormal">Active / Total Size (% used) : 93857848.58K / 93872319.47K (100.0%)<o:p></o:p></p>
<p class="MsoNormal">Minimum / Average / Maximum Object : 0.02K / 0.02K / 4096.00K<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME<o:p></o:p></p>
<p class="MsoNormal">3987822096 3987821817 0% 0.02K 27693209 144 110772836K scsi_data_buffer<o:p></o:p></p>
<p class="MsoNormal">91155 64448 70% 0.06K 1545 59 6180K size-64<o:p></o:p></p>
<p class="MsoNormal">36064 32035 88% 0.03K 322 112 1288K size-32<o:p></o:p></p>
<p class="MsoNormal">35505 34334 96% 0.25K 2367 15 9468K skbuff_head_cache<o:p></o:p></p>
<p class="MsoNormal">33876 33874 99% 8.00K 33876 1 271008K size-8192<o:p></o:p></p>
<p class="MsoNormal">33804 33615 99% 0.14K 1252 27 5008K sysfs_dir_cache<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
-- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated
otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your
own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be
liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt.
</body>
</html>