[gpfsug-discuss] System running out of memory - SUnreclaim is huge

John Hearns john.hearns at asml.com
Thu Jan 11 14:16:26 GMT 2018


I am having problems with GPFS servers running out of memory.
We have an open PMR for this, however if anyone has seen this or has any ideas I would be grateful for a heads up.
Servers have 128 Gbytes f RAM,  kernel 2.6.32-573.18.1.el6.x86_64,   GPFS version 4.2.3.4

In the latest incident the free memory went to below 1Gbyte, and we started to have processes killed, including our monitoring setup.
I shut down GPFS on that server and /proc/meminfo still shows:

Slab:           111192296 kB
SReclaimable:      29020 kB
SUnreclaim:     111163276 kB

Am I barking up a wrong tree here and pointing the finger at GPFS? Something is causing the scsi_data_buffer slab memory usage (see below).

One thing I did yesterday was to change the disk scheduler for each disk from cfq to dealine (as recommended in the tuning guide)
However the server was already in short memory at that point.

Slabtop shows
Active / Total Objects (% used)    : -306803185 / -306722574 (100.0%)
Active / Total Slabs (% used)      : 27749714 / 27749719 (100.0%)
Active / Total Caches (% used)     : 115 / 198 (58.1%)
Active / Total Size (% used)       : 93857848.58K / 93872319.47K (100.0%)
Minimum / Average / Maximum Object : 0.02K / 0.02K / 4096.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
3987822096 3987821817   0%    0.02K 27693209      144 110772836K scsi_data_buffer
91155  64448  70%    0.06K   1545       59      6180K size-64
36064  32035  88%    0.03K    322      112      1288K size-32
35505  34334  96%    0.25K   2367       15      9468K skbuff_head_cache
33876  33874  99%    8.00K  33876        1    271008K size-8192
33804  33615  99%    0.14K   1252       27      5008K sysfs_dir_cache

-- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20180111/48163d6f/attachment-0001.htm>


More information about the gpfsug-discuss mailing list