<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
Curious, if you could say something about how you ended up with some page pool values on your client side that are that high. For what use cases does 64GB, for example, make a difference?
<div><br id="lineBreakAtBeginningOfMessage">
<div>
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
--<br>
#BlackLivesMatter</div>
<div style="color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
____<br>
|| \\UTGERS, |---------------------------*O*---------------------------<br>
||_// the State<span class="Apple-tab-span" style="white-space: pre;"> </span> | Ryan Novosielski - novosirj@rutgers.edu<br>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus<br>
|| \\ of NJ<span class="Apple-tab-span" style="white-space: pre;"> </span> | Office of Advanced Research Computing - MSB A555B, Newark<br>
`'</div>
</div>
</div>
</div>
</div>
</div>
<div><br>
<blockquote type="cite">
<div>On Mar 8, 2024, at 11:32, Wahl, Edward <ewahl@osc.edu> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Yikes! Those must be some mighty large memory compute nodes! That is an OK setting for a large memory ESS/DSS server but NOT the compute nodes at
my site, as that is in bytes.<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">(so ~324 GB) Even on our 1TB+ memory machines we do not tune it that high.<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><o:p> </o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">You can set pagepool for nodeclass machines such as all your compute, but pagepool is one of those settings where you will have to restart the clients
for it to take effect. (such as most all the rdma settings, etc)<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">You should look into creating a “nodeclass” for each of your “node types” if you have not already, so you can avoid OOM issues from just the pagepool,
and tune other settings per node-type (rdma/network settings, etc)<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">I would address this here, rather than on the Slurm side. Then you can address (total memory minus the pagepool) for the overall addressability to
Slurm for user jobs. Leave some spare memory for the system itself or you will see more memory issues and whatnot when users get close to OOM, even in their cgroup.<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><o:p> </o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Example from a cross mounted compute-side cluster. Default is 1GB:<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">[root@nostorage-manager1 ~]# mmlsconfig pagepool<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">pagepool 1024M<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">pagepool 4G [k8,pitzer]<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">pagepool 64G [ascend]<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">pagepool 16G [ib-spire-login,owenslogin,pitzerlogin]<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">pagepool 48G [dm]<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">pagepool 4G [cardinal]<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">pagepool 64G [cardinal_quadport]<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><o:p> </o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">example from the ESS/DSS server side. Later ESS versions set things by mmvdisk groups, rather than server type.<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"># mmlsconfig pagepool<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">pagepool 32G<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">pagepool 358G [gss_ppc64]<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">pagepool 16384M [ibmems11-hs,ems]<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">pagepool 324383477760 [ess3200_mmvdisk_ibmessio13_hs_ibmessio14_hs,ess3200_mmvdisk_ibmessio15_hs_ibmessio16_hs,ess3200_mmvdisk_ibmessio17_hs_ibmessio18_hs]<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">pagepool 64G [sp]<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">pagepool 384399572992 [ibmgssio1_hsibmgssio2_hs,ibmgssio3_hsibmgssio4_hs,ibmgssio5_hsibmgssio6_hs]<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">pagepool 573475966156 [ess5k_mmvdisk_ibmessio11_hs_ibmessio12_hs]<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">pagepool 96G [ces]<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><o:p> </o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">example of nodeclasses used to address other settings, such as what Infiniband port(s) to use.<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"># mmlsconfig verbsports<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">verbsPorts mlx5_0<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">verbsPorts mlx5_0 mlx5_2 [pitzer_dualport]<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">verbsPorts mlx4_1/1 mlx4_1/2 [dm]<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">verbsPorts mlx5_0 mlx5_2 [k8_dualport]<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">verbsPorts mlx5_0 mlx5_1 mlx5_2 mlx5_3 [cardinal_quadport]<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;"><o:p> </o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Ed Wahl<o:p></o:p></span></div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 11pt;">Ohio Supercomputer Center<o:p></o:p></span></div>
<div style="border-width: 1pt medium medium; border-style: solid none none; border-color: rgb(225, 225, 225) currentcolor currentcolor; border-image: none; padding: 3pt 0in 0in;">
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><b><span style="font-size: 11pt;">From:</span></b><span style="font-size: 11pt;"><span class="Apple-converted-space"> </span>gpfsug-discuss <<a href="mailto:gpfsug-discuss-bounces@gpfsug.org" style="color: blue; text-decoration: underline;">gpfsug-discuss-bounces@gpfsug.org</a>><span class="Apple-converted-space"> </span><b>On
Behalf Of<span class="Apple-converted-space"> </span></b>Iban Cabrillo<br>
<b>Sent:</b><span class="Apple-converted-space"> </span>Friday, March 8, 2024 9:40 AM<br>
<b>To:</b><span class="Apple-converted-space"> </span>gpfsug-discuss <<a href="mailto:gpfsug-discuss@spectrumscale.org" style="color: blue; text-decoration: underline;">gpfsug-discuss@spectrumscale.org</a>><br>
<b>Subject:</b><span class="Apple-converted-space"> </span>[gpfsug-discuss] pagepool<o:p></o:p></span></div>
</div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div>
<div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 1pt; color: white;">Good afternoon, We are new to the DSS system configurations. Reviewing the configuration I have seen that the default pagepool is set to
this value: pagepool 323908133683 But not only in the DSS servers, but also in the rest of the HPC nodes<span class="Apple-converted-space"> </span><o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 1pt; color: white;"><o:p></o:p></span></div>
</div>
<div>
<div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 12pt; font-family: Arial, sans-serif;">Good afternoon,<o:p></o:p></span></div>
</div>
<div>
<div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 12pt; font-family: Arial, sans-serif;"> We are new to the DSS system configurations. Reviewing the configuration I have seen that the default pagepool is
set to this value:<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 12pt; font-family: Arial, sans-serif;"><o:p> </o:p></span></div>
</div>
<div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><strong><span style="font-size: 12pt; font-family: Arial, sans-serif;"> pagepool 323908133683</span></strong><span style="font-size: 12pt; font-family: Arial, sans-serif;"><o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 12pt; font-family: Arial, sans-serif;"><o:p> </o:p></span></div>
</div>
<div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 12pt; font-family: Arial, sans-serif;">But not only in the DSS servers, but also in the rest of the HPC nodes and I don't know if it is an excessive value.
We are noticing that some jobs are dying by "Memory cgroup out of memory: Killed process XXX", and my doubt is if this pagepool is reserving too much memory for the mmfs process in decripento of the execution of jobs.<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 12pt; font-family: Arial, sans-serif;"><o:p> </o:p></span></div>
</div>
<div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 12pt; font-family: Arial, sans-serif;">Any advice is welcomed,<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 12pt; font-family: Arial, sans-serif;"><o:p> </o:p></span></div>
</div>
<div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 12pt; font-family: Arial, sans-serif;">Regards, I<o:p></o:p></span></div>
</div>
</div>
<div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 12pt; font-family: Arial, sans-serif;">--<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in; font-size: 10pt; font-family: Calibri, sans-serif;"><span style="font-size: 12pt; font-family: Arial, sans-serif;"><br>
================================================================<br>
Ibán Cabrillo Bartolomé<br>
Instituto de Física de Cantabria (IFCA-CSIC)<br>
Santander, Spain<br>
Tel: +34942200969/+34669930421<br>
Responsible for advanced computing service (RSC)<br>
=========================================================================================<br>
=========================================================================================<br>
All our suppliers must know and accept IFCA policy available at:<br>
<br>
<a href="https://urldefense.com/v3/__https:/confluence.ifca.es/display/IC/Information*Security*Policy*for*External*Suppliers__;KysrKys!!KGKeukY!3o_dGRsvxDtOG6Z646nJEb9ehb_ondS1kL3gecKjKN7mvMULc6h9iKST-ihDjnWz04X-lcNATjPzLDB2eW7P$" style="color: blue; text-decoration: underline;">https://confluence.ifca.es/display/IC/Information+Security+Policy+for+External+Suppliers</a><br>
==========================================================================================<o:p></o:p></span></div>
</div>
</div>
<p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 10pt; font-family: Calibri, sans-serif;">
<span style="font-size: 11pt;"><o:p> </o:p></span></p>
</div>
<span style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">_______________________________________________</span><br style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<span style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">gpfsug-discuss
mailing list</span><br style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<span style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">gpfsug-discuss
at<span class="Apple-converted-space"> </span></span><a href="http://gpfsug.org/" style="color: blue; text-decoration: underline; font-family: Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">gpfsug.org</a><br style="caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org" style="color: blue; text-decoration: underline; font-family: Menlo-Regular; font-size: 11px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org</a></div>
</blockquote>
</div>
<br>
</div>
</body>
</html>