[gpfsug-discuss] pagepool shrink doesn't release all memory

[IBM Spectrum Scale]

AFAIK you can increase the pagepool size dynamically but you cannot shrink 
it dynamically.  To shrink it you must restart the GPFS daemon.   Also, 
could you please provide the actual pmap commands you executed?

Regards, The Spectrum Scale (GPFS) team

------------------------------------------------------------------------------------------------------------------
If you feel that your question can benefit other users of  Spectrum Scale 
(GPFS), then please post it to the public IBM developerWroks Forum at 
https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479
. 

If your query concerns a potential software error in Spectrum Scale (GPFS) 
and you have an IBM software maintenance contract please contact 
1-800-237-5511 in the United States or your local IBM Service Center in 
other countries. 

The forum is informally monitored as time permits and should not be used 
for priority messages to the Spectrum Scale (GPFS) team.



From:   Aaron Knister <aaron.s.knister at nasa.gov>
To:     <gpfsug-discuss at spectrumscale.org>
Date:   02/22/2018 10:30 PM
Subject:        Re: [gpfsug-discuss] pagepool shrink doesn't release all 
memory
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



This is also interesting (although I don't know what it really means). 
Looking at pmap run against mmfsd I can see what happens after each step:

# baseline
00007fffe4639000  59164K      0K      0K      0K      0K ---p [anon]
00007fffd837e000  61960K      0K      0K      0K      0K ---p [anon]
0000020000000000 1048576K 1048576K 1048576K 1048576K      0K rwxp [anon]
Total:           1613580K 1191020K 1189650K 1171836K      0K

# tschpool 64G
00007fffe4639000  59164K      0K      0K      0K      0K ---p [anon]
00007fffd837e000  61960K      0K      0K      0K      0K ---p [anon]
0000020000000000 67108864K 67108864K 67108864K 67108864K      0K rwxp 
[anon]
Total:           67706636K 67284108K 67282625K 67264920K      0K

# tschpool 1G
00007fffe4639000  59164K      0K      0K      0K      0K ---p [anon]
00007fffd837e000  61960K      0K      0K      0K      0K ---p [anon]
0000020001400000 139264K 139264K 139264K 139264K      0K rwxp [anon]
0000020fc9400000 897024K 897024K 897024K 897024K      0K rwxp [anon]
0000020009c00000 66052096K      0K      0K      0K      0K rwxp [anon]
Total:           67706636K 1223820K 1222451K 1204632K      0K

Even though mmfsd has that 64G chunk allocated there's none of it 
*used*. I wonder why Linux seems to be accounting it as allocated.

-Aaron

On 2/22/18 10:17 PM, Aaron Knister wrote:
> I've been exploring the idea for a while of writing a SLURM SPANK plugin 

> to allow users to dynamically change the pagepool size on a node. Every 
> now and then we have some users who would benefit significantly from a 
> much larger pagepool on compute nodes but by default keep it on the 
> smaller side to make as much physmem available as possible to batch 
work.
> 
> In testing, though, it seems as though reducing the pagepool doesn't 
> quite release all of the memory. I don't really understand it because 
> I've never before seen memory that was previously resident become 
> un-resident but still maintain the virtual memory allocation.
> 
> Here's what I mean. Let's take a node with 128G and a 1G pagepool.
> 
> If I do the following to simulate what might happen as various jobs 
> tweak the pagepool:
> 
> - tschpool 64G
> - tschpool 1G
> - tschpool 32G
> - tschpool 1G
> - tschpool 32G
> 
> I end up with this:
> 
> mmfsd thinks there's 32G resident but 64G virt
> # ps -o vsz,rss,comm -p 24397
>     VSZ   RSS COMMAND
> 67589400 33723236 mmfsd
> 
> however, linux thinks there's ~100G used
> 
> # free -g
>               total       used       free     shared    buffers     
cached
> Mem:           125        100         25          0          0          
0
> -/+ buffers/cache:         98         26
> Swap:            7          0          7
> 
> I can jump back and forth between 1G and 32G *after* allocating 64G 
> pagepool and the overall amount of memory in use doesn't balloon but I 
> can't seem to shed that original 64G.
> 
> I don't understand what's going on... :) Any ideas? This is with Scale 
> 4.2.3.6.
> 
> -Aaron
> 

-- 
Aaron Knister
NASA Center for Climate Simulation (Code 606.2)
Goddard Space Flight Center
(301) 286-2776
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=OrZQeEmI6chBdguG-h4YPHsxXZ4gTU3CtIuN4e3ijdY&s=hvVIRG5kB1zom2Iql2_TOagchsgl99juKiZfJt5S1tM&e=






-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20180223/e3859615/attachment-0002.htm>