<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Hi Brian,
<div class=""><br class="">
</div>
<div class="">It would be helpful to know what scheduling software, if any, you use.</div>
<div class=""><br class="">
</div>
<div class="">We were a PBS / Moab shop for a number of years but switched to SLURM two years ago. With both you can configure the maximum amount of memory available to all jobs on a node. So we just simply “reserve” however much we need for GPFS and other
“system” processes.</div>
<div class=""><br class="">
</div>
<div class="">I can tell you that SLURM is *much* more efficient at killing processes as soon as they exceed the amount of memory they’ve requested than PBS / Moab ever dreamed of being.</div>
<div class=""><br class="">
</div>
<div class="">Kevin</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Dec 20, 2016, at 10:27 AM, Skylar Thompson <<a href="mailto:skylar2@u.washington.edu" class="">skylar2@u.washington.edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">We're a Grid Engine shop, and use cgroups (m_mem_free) to control user process memory<br class="">
usage. In the GE exec host configuration, we reserve 4GB for the OS<br class="">
(including GPFS) so jobs are not able to consume all the physical memory on<br class="">
the system.<br class="">
<br class="">
On Tue, Dec 20, 2016 at 11:25:04AM -0500, Brian Marshall wrote:<br class="">
<blockquote type="cite" class="">All,<br class="">
<br class="">
What is your favorite method for stopping a user process from eating up all<br class="">
the system memory and saving 1 GB (or more) for the GPFS / system<br class="">
processes? We have always kicked around the idea of cgroups but never<br class="">
moved on it.<br class="">
<br class="">
The problem: A user launches a job which uses all the memory on a node,<br class="">
which causes the node to be expelled, which causes brief filesystem<br class="">
slowness everywhere.<br class="">
<br class="">
I bet this problem has already been solved and I am just googling the wrong<br class="">
search terms.<br class="">
<br class="">
<br class="">
Thanks,<br class="">
Brian<br class="">
</blockquote>
<br class="">
<blockquote type="cite" class="">_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a><br class="">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="">
</blockquote>
<br class="">
<br class="">
-- <br class="">
-- Skylar Thompson (<a href="mailto:skylar2@u.washington.edu" class="">skylar2@u.washington.edu</a>)<br class="">
-- Genome Sciences Department, System Administrator<br class="">
-- Foege Building S046, (206)-685-7354<br class="">
-- University of Washington School of Medicine<br class="">
_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a><br class="">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
<br class="">
<br class="">
<div class="">
<div class="">—</div>
<div class="">Kevin Buterbaugh - Senior System Administrator</div>
<div class="">Vanderbilt University - Advanced Computing Center for Research and Education</div>
<div class=""><a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" class="">Kevin.Buterbaugh@vanderbilt.edu</a> - (615)875-9633</div>
<div class=""><br class="">
</div>
<br class="Apple-interchange-newline">
</div>
<br class="">
</body>
</html>