<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Hi Brian,
<div class=""><br class="">
</div>
<div class="">I don’t *think* you can entirely solve this problem with Moab … as I mentioned, it’s not nearly as efficient as SLURM is at killing jobs when they exceed requested memory. We had situations where a user would be able to run a node out of memory
before Moab would kill it. Hasn’t happened once with SLURM, AFAIK.</div>
<div class=""><br class="">
</div>
<div class="">But with either Moab or SLURM what we’ve done is taken the amount of physical RAM in the box and subtracted from that the amount of memory we want to “reserve” for the system (OS, GPFS, etc.) and then told Moab / SLURM that this is how much RAM
the box has. That way they at least won’t schedule jobs on the node that would exceed available memory.</div>
<div class=""><br class="">
</div>
<div class="">HTH…</div>
<div class=""><br class="">
</div>
<div class="">Kevin</div>
<div class=""><br class="">
</div>
<div class="">
<div>
<blockquote type="cite" class="">
<div class="">On Dec 20, 2016, at 11:07 AM, Brian Marshall <<a href="mailto:mimarsh2@vt.edu" class="">mimarsh2@vt.edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="auto" class="">We use adaptive - Moab torque right now but are thinking about going to Skyrim
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class="">Brian</div>
</div>
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On Dec 20, 2016 11:38 AM, "Buterbaugh, Kevin L" <<a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" class="">Kevin.Buterbaugh@vanderbilt.edu</a>> wrote:<br type="attribution" class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word" class="">Hi Brian,
<div class=""><br class="">
</div>
<div class="">It would be helpful to know what scheduling software, if any, you use.</div>
<div class=""><br class="">
</div>
<div class="">We were a PBS / Moab shop for a number of years but switched to SLURM two years ago. With both you can configure the maximum amount of memory available to all jobs on a node. So we just simply “reserve” however much we need for GPFS and other
“system” processes.</div>
<div class=""><br class="">
</div>
<div class="">I can tell you that SLURM is *much* more efficient at killing processes as soon as they exceed the amount of memory they’ve requested than PBS / Moab ever dreamed of being.</div>
<div class=""><br class="">
</div>
<div class="">Kevin</div>
<div class=""><br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On Dec 20, 2016, at 10:27 AM, Skylar Thompson <<a href="mailto:skylar2@u.washington.edu" target="_blank" class="">skylar2@u.washington.edu</a>> wrote:</div>
<br class="m_2891201518838307593Apple-interchange-newline">
<div class="">
<div class="">We're a Grid Engine shop, and use cgroups (m_mem_free) to control user process memory<br class="">
usage. In the GE exec host configuration, we reserve 4GB for the OS<br class="">
(including GPFS) so jobs are not able to consume all the physical memory on<br class="">
the system.<br class="">
<br class="">
On Tue, Dec 20, 2016 at 11:25:04AM -0500, Brian Marshall wrote:<br class="">
<blockquote type="cite" class="">All,<br class="">
<br class="">
What is your favorite method for stopping a user process from eating up all<br class="">
the system memory and saving 1 GB (or more) for the GPFS / system<br class="">
processes? We have always kicked around the idea of cgroups but never<br class="">
moved on it.<br class="">
<br class="">
The problem: A user launches a job which uses all the memory on a node,<br class="">
which causes the node to be expelled, which causes brief filesystem<br class="">
slowness everywhere.<br class="">
<br class="">
I bet this problem has already been solved and I am just googling the wrong<br class="">
search terms.<br class="">
<br class="">
<br class="">
Thanks,<br class="">
Brian<br class="">
</blockquote>
<br class="">
<blockquote type="cite" class="">______________________________<wbr class="">_________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="http://spectrumscale.org/" target="_blank" class="">spectrumscale.org</a><br class="">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank" class="">http://gpfsug.org/mailman/<wbr class="">listinfo/gpfsug-discuss</a><br class="">
</blockquote>
<br class="">
<br class="">
-- <br class="">
-- Skylar Thompson (<a href="mailto:skylar2@u.washington.edu" target="_blank" class="">skylar2@u.washington.edu</a>)<br class="">
-- Genome Sciences Department, System Administrator<br class="">
-- Foege Building S046, <a href="tel:(206)%20685-7354" value="+12066857354" target="_blank" class="">
(206)-685-7354</a><br class="">
-- University of Washington School of Medicine<br class="">
______________________________<wbr class="">_________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="http://spectrumscale.org/" target="_blank" class="">spectrumscale.org</a><br class="">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank" class="">http://gpfsug.org/mailman/<wbr class="">listinfo/gpfsug-discuss</a><br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
<br class="">
<br class="">
<div class="">
<div class="">—</div>
<div class="">Kevin Buterbaugh - Senior System Administrator</div>
<div class="">Vanderbilt University - Advanced Computing Center for Research and Education</div>
<div class=""><a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" target="_blank" class="">Kevin.Buterbaugh@vanderbilt.<wbr class="">edu</a> -
<a href="tel:(615)%20875-9633" value="+16158759633" target="_blank" class="">(615)875-9633</a></div>
<div class=""><br class="">
</div>
<br class="m_2891201518838307593Apple-interchange-newline">
</div>
<br class="">
</div>
<br class="">
______________________________<wbr class="">_________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="http://spectrumscale.org/" rel="noreferrer" target="_blank" class="">
spectrumscale.org</a><br class="">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank" class="">http://gpfsug.org/mailman/<wbr class="">listinfo/gpfsug-discuss</a><br class="">
<br class="">
</blockquote>
</div>
</div>
_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a><br class="">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
<br class="">
<br class="">
<div class="">
<div class="">—</div>
<div class="">Kevin Buterbaugh - Senior System Administrator</div>
<div class="">Vanderbilt University - Advanced Computing Center for Research and Education</div>
<div class=""><a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" class="">Kevin.Buterbaugh@vanderbilt.edu</a> - (615)875-9633</div>
<div class=""><br class="">
</div>
<br class="Apple-interchange-newline">
</div>
<br class="">
</body>
</html>