<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Aptos;
panose-1:2 11 0 4 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:12.0pt;
font-family:"Aptos",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Aptos",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Yeah; you’ll want to bump that up on the home fileset, something like:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">mmchfileset cluster home --inode-limit 25000000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">(that’d give you a buffer of ~4.9 million inodes)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">For the ones that show 0, those are dependent filesets (not independent) the inode allocations are tracked in the parent independent inode fileset.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Best,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black">J.D. Maloney</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">Lead</span><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"> HPC Storage Engineer | Storage Enabling Technologies Group</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black">National Center for Supercomputing Applications (NCSA)</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div id="mail-editor-reference-message-container">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="color:black">From:
</span></b><span style="color:black">gpfsug-discuss <gpfsug-discuss-bounces@gpfsug.org> on behalf of Rob Kudyba <rk3199@columbia.edu><br>
<b>Date: </b>Thursday, June 6, 2024 at 5:51</span><span style="font-family:"Arial",sans-serif;color:black"> </span><span style="color:black">PM<br>
<b>To: </b>gpfsug main discussion list <gpfsug-discuss@gpfsug.org><br>
<b>Subject: </b>Re: [gpfsug-discuss] No space left on device, but plenty of quota space for inodes and blocks<o:p></o:p></span></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal">I guess I have my answer:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><span style="font-family:"Courier New"">/usr/lpp/mmfs/bin/mmlsfileset cluster home -L<br>
Filesets in file system 'cluster':<br>
Name Id RootInode ParentId Created InodeSpace MaxInodes AllocInodes Comment<br>
home 1 1048579 0 Thu Nov 29 15:21:52 2018 1 20971520 20971520 </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">However on some of the other filesets the AllocInodes is 0?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">/usr/lpp/mmfs/bin/mmlsfileset cluster groupa -L -i<br>
Collecting fileset usage information ...<br>
Filesets in file system 'moto':<br>
Name Id RootInode ParentId Created InodeSpace MaxInodes AllocInodes UsedInodes Comment<br>
stats 8 181207 0 Fri Nov 30 12:27:25 2018 0 0 0 7628733 </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Yes we realize it's old and it'll be retired at the end of 2024.<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Thu, Jun 6, 2024 at 6:15<span style="font-family:"Arial",sans-serif"> </span>PM Fred Stock <<a href="mailto:stockf@us.ibm.com">stockf@us.ibm.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:14.0pt;font-family:"Arial",sans-serif">You should check the inode counts for each of the filesets using the mmlsfileset command. You should check the local
disk space on all the nodes.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:14.0pt;font-family:"Arial",sans-serif"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:14.0pt;font-family:"Arial",sans-serif">I presume you are aware that Scale 4.2.3 has been out of support for 4 years.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:14.0pt;font-family:"Arial",sans-serif"> </span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Arial",sans-serif">Fred</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Arial",sans-serif"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Verdana",sans-serif;color:#121212;background:white">Fred Stock, Spectrum Scale Development Advocacy</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="font-size:11.0pt;font-family:"Verdana",sans-serif;color:#121212;background:white"><a href="mailto:stockf@us.ibm.com" target="_blank"><span style="color:#0563C1">stockf@us.ibm.com</span></a>
| 720-430-8821</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:14.0pt;font-family:"Arial",sans-serif"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:14.0pt;font-family:"Arial",sans-serif"> </span><o:p></o:p></p>
<div id="m_1864070822950688999mail-editor-reference-message-container">
<div>
<div style="border:none;border-top:solid windowtext 1.0pt;padding:3.0pt 0in 0in 0in;border-color:currentcolor currentcolor">
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><b><span style="color:black">From:
</span></b><span style="color:black">gpfsug-discuss <<a href="mailto:gpfsug-discuss-bounces@gpfsug.org" target="_blank">gpfsug-discuss-bounces@gpfsug.org</a>> on behalf of Rob Kudyba <<a href="mailto:rk3199@columbia.edu" target="_blank">rk3199@columbia.edu</a>><br>
<b>Date: </b>Thursday, June 6, 2024 at 5:39</span><span style="font-family:"Arial",sans-serif;color:black"> </span><span style="color:black">PM<br>
<b>To: </b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@gpfsug.org" target="_blank">gpfsug-discuss@gpfsug.org</a>><br>
<b>Subject: </b>[EXTERNAL] Re: [gpfsug-discuss] No space left on device, but plenty of quota space for inodes and blocks</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:1.0pt;color:white">Are you seeing the issues across the whole file system or in certain areas? Only with accounts in GPFS, local accounts and root do not
gt this. That sounds like inode exhaustion to me (and based on it not being block exhaustion as you’ve demonstrated).</span><span style="font-size:1.0pt;font-family:"Arial",sans-serif;color:white"> </span><span style="font-size:1.0pt;color:white">
</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerStart</span><o:p></o:p></p>
</div>
<div id="m_1864070822950688999pfptBanner8op7xry">
<div id="m_1864070822950688999pfptBanner8op7xry">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:13.5pt;background:#D0D8DC">
<b><span lang="EN" style="font-family:"Arial",sans-serif;color:black">This Message Is From an Untrusted Sender
</span></b><o:p></o:p></p>
</div>
<div id="m_1864070822950688999pfptBanner8op7xry">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:13.5pt;background:#D0D8DC">
<span lang="EN" style="font-family:"Arial",sans-serif;color:black">You have not previously corresponded with this sender.
</span><o:p></o:p></p>
</div>
</div>
<div id="m_1864070822950688999pfptBanner8op7xry">
<div style="border:solid #666666 1.0pt;padding:12.0pt 12.0pt 12.0pt 12.0pt;margin-left:12.0pt;margin-top:12.0pt;margin-right:12.0pt;margin-bottom:12.0pt;display:inline-block;opacity:1;border-radius:2px;width:fit-content">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:#D0D8DC">
<span lang="EN" style="font-size:10.5pt;font-family:"Arial",sans-serif;color:black"><a href="https://us-phishalarm-ewt.proofpoint.com/EWT/v1/AdhS1Rd-!-XFVHHiymkdb2PX5Ys9u3xcIH6Vd3Ap1CobKrLSv4AEKLaxWTmX-SIXo5pwXtsG8GuxP6yYyms8BE2p0j0YYMsauSua4xvEzG7v8C4nNZ8q-8rr50pPoh5DWHA$" target="_blank"><span style="color:black;text-decoration:none">Report Suspicious</span></a></span><span style="font-size:10.5pt;color:black"><o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:#D0D8DC">
<span lang="EN" style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:#D0D8DC">
<span lang="EN" style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:1.0pt;color:white">ZjQcmQRYFpfptBannerEnd</span><o:p></o:p></p>
</div>
<div>
<div>
<blockquote style="border:none;border-left:solid windowtext 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Are you seeing the issues across the whole file system or in certain areas?
</span><o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Only with accounts in GPFS, local accounts and root do not gt this.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid windowtext 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">That sounds like inode exhaustion to me (and based on it not being block exhaustion as you’ve demonstrated).
</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">What does a “df -i /cluster” show you?
</span><o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">We bumped it up a few weeks ago:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New"">df -i /cluster<br>
Filesystem Inodes IUsed IFree IUse% Mounted on<br>
cluster 276971520 154807697 122163823 56% /cluster</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid windowtext 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Or if this is only in a certain area you can “cd” into that directory and run a “df -i .”</span><o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">As root on a login node;<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New"">df -i<br>
Filesystem Inodes IUsed IFree IUse% Mounted on<br>
/dev/sda2 20971520 169536 20801984 1% /<br>
devtmpfs 12169978 528 12169450 1% /dev<br>
tmpfs 12174353 1832 12172521 1% /run<br>
tmpfs 12174353 77 12174276 1% /dev/shm<br>
tmpfs 12174353 15 12174338 1% /sys/fs/cgroup<br>
/dev/sda1 0 0 0 - /boot/efi<br>
/dev/sda3 52428800 2887 52425913 1% /var<br>
/dev/sda7 277368832 35913 277332919 1% /local<br>
/dev/sda5 104857600 398 104857202 1% /tmp<br>
tmpfs 12174353 1 12174352 1% /run/user/551336<br>
tmpfs 12174353 1 12174352 1% /run/user/0<br>
moto 276971520 154807697 122163823 56% /cluster<br>
tmpfs 12174353 3 12174350 1% /run/user/441245<br>
tmpfs 12174353 12 12174341 1% /run/user/553562<br>
tmpfs 12174353 1 12174352 1% /run/user/525583<br>
tmpfs 12174353 1 12174352 1% /run/user/476374<br>
tmpfs 12174353 1 12174352 1% /run/user/468934<br>
tmpfs 12174353 5 12174348 1% /run/user/551200<br>
tmpfs 12174353 1 12174352 1% /run/user/539143<br>
tmpfs 12174353 1 12174352 1% /run/user/488676<br>
tmpfs 12174353 1 12174352 1% /run/user/493713<br>
tmpfs 12174353 1 12174352 1% /run/user/507831<br>
tmpfs 12174353 1 12174352 1% /run/user/549822<br>
tmpfs 12174353 1 12174352 1% /run/user/500569<br>
tmpfs 12174353 1 12174352 1% /run/user/443748<br>
tmpfs 12174353 1 12174352 1% /run/user/543676<br>
tmpfs 12174353 1 12174352 1% /run/user/451446<br>
tmpfs 12174353 1 12174352 1% /run/user/497945<br>
tmpfs 12174353 6 12174347 1% /run/user/554672<br>
tmpfs 12174353 32 12174321 1% /run/user/554653<br>
tmpfs 12174353 1 12174352 1% /run/user/30094<br>
tmpfs 12174353 1 12174352 1% /run/user/470790<br>
tmpfs 12174353 59 12174294 1% /run/user/553037<br>
tmpfs 12174353 1 12174352 1% /run/user/554670<br>
tmpfs 12174353 1 12174352 1% /run/user/548236<br>
tmpfs 12174353 1 12174352 1% /run/user/547288</span><o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New"">tmpfs 12174353 1 12174352 1% /run/user/547289 </span><o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid windowtext 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">You may need to allocate more inodes to an independent inode fileset somewhere. Especially with something as old as 4.2.3 you won’t have auto-inode
expansion for the filesets.</span><o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Do we have to restart any service after upping the inode count?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid windowtext 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt">Best,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black">J.D. Maloney</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">Lead</span><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"> HPC Storage Engineer
| Storage Enabling Technologies Group</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black">National Center for Supercomputing Applications (NCSA)</span><o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Ho JD I took an intermediate LCI workshop with you at Univ of Cincinnati!<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid windowtext 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div id="m_1864070822950688999m_2465296025784317369mail-editor-reference-message-container">
<div>
<div style="border:none;border-top:solid windowtext 1.0pt;padding:3.0pt 0in 0in 0in;border-color:currentcolor">
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><b><span style="color:black">From:
</span></b><span style="color:black">gpfsug-discuss <<a href="mailto:gpfsug-discuss-bounces@gpfsug.org" target="_blank">gpfsug-discuss-bounces@gpfsug.org</a>> on behalf of Rob Kudyba <<a href="mailto:rk3199@columbia.edu" target="_blank">rk3199@columbia.edu</a>><br>
<b>Date: </b>Thursday, June 6, 2024 at 3:50</span><span style="font-family:"Arial",sans-serif;color:black"> </span><span style="color:black">PM<br>
<b>To: </b><a href="mailto:gpfsug-discuss@gpfsug.org" target="_blank">gpfsug-discuss@gpfsug.org</a> <<a href="mailto:gpfsug-discuss@gpfsug.org" target="_blank">gpfsug-discuss@gpfsug.org</a>><br>
<b>Subject: </b>[gpfsug-discuss] No space left on device, but plenty of quota space for inodes and blocks</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Running GPFS 4.2.3 on a DDN GridScaler and users are getting the <span style="font-family:"Courier New"">No space left on device</span> message when trying to write to a file. In <span style="font-family:"Courier New"">/var/adm/ras/mmfs.log
</span>the only recent errors are this:<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New"">2024-06-06_15:51:22.311-0400: mmcommon getContactNodes cluster failed. Return code -1.<br>
2024-06-06_15:51:22.311-0400: The previous error was detected on node x.x.x.x (headnode).<br>
2024-06-06_15:53:25.088-0400: mmcommon getContactNodes cluster failed. Return code -1.<br>
2024-06-06_15:53:25.088-0400: The previous error was detected on node x.x.x.x (headnode).</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">according to
<a href="https://urldefense.com/v3/__https:/www.ibm.com/docs/en/storage-scale/5.1.9?topic=messages-6027-615__;!!DZ3fjg!6G98Oo44gOmToQaQbotCPR6nxP_ox0xaRc9U6-ZjcWg8XMrwGvtt72VimFO6hSNnNkySuxawIVoA3_uEDKIgjW0$" target="_blank">
https://www.ibm.com/docs/en/storage-scale/5.1.9?topic=messages-6027-615</a> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid windowtext 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Check the preceding messages, and consult the earlier chapters of this document. A frequent cause for such errors is lack of space in
<span style="font-family:"Courier New"">/var</span>.<o:p></o:p></p>
</blockquote>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">We have plenty of space left.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New""> /usr/lpp/mmfs/bin/mmlsdisk cluster<br>
disk driver sector failure holds holds storage<br>
name type size group metadata data status availability pool<br>
------------ -------- ------ ----------- -------- ----- ------------- ------------ ------------<br>
S01_MDT200_1 nsd 4096 200 Yes No ready up system
<br>
S01_MDT201_1 nsd 4096 201 Yes No ready up system
<br>
S01_DAT0001_1 nsd 4096 100 No Yes ready up data1 <br>
S01_DAT0002_1 nsd 4096 101 No Yes ready up data1 <br>
S01_DAT0003_1 nsd 4096 100 No Yes ready up data1 <br>
S01_DAT0004_1 nsd 4096 101 No Yes ready up data1 <br>
S01_DAT0005_1 nsd 4096 100 No Yes ready up data1 <br>
S01_DAT0006_1 nsd 4096 101 No Yes ready up data1 <br>
S01_DAT0007_1 nsd 4096 100 No Yes ready up data1 </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New""> /usr/lpp/mmfs/bin/mmdf headnode <br>
disk disk size failure holds holds free KB free KB<br>
name in KB group metadata data in full blocks in fragments<br>
--------------- ------------- -------- -------- ----- -------------------- -------------------<br>
Disks in storage pool: system (Maximum disk size allowed is 14 TB)<br>
S01_MDT200_1 1862270976 200 Yes No 969134848 ( 52%) 2948720 ( 0%)
<br>
S01_MDT201_1 1862270976 201 Yes No 969126144 ( 52%) 2957424 ( 0%)
<br>
------------- -------------------- -------------------<br>
(pool total) 3724541952 1938260992 ( 52%) 5906144 ( 0%)<br>
<br>
Disks in storage pool: data1 (Maximum disk size allowed is 578 TB)<br>
S01_DAT0007_1 77510737920 100 No Yes 21080752128 ( 27%) 897723392 ( 1%)
<br>
S01_DAT0005_1 77510737920 100 No Yes 14507212800 ( 19%) 949412160 ( 1%)
<br>
S01_DAT0001_1 77510737920 100 No Yes 14503620608 ( 19%) 951327680 ( 1%)
<br>
S01_DAT0003_1 77510737920 100 No Yes 14509205504 ( 19%) 949340544 ( 1%)
<br>
S01_DAT0002_1 77510737920 101 No Yes 14504585216 ( 19%) 948377536 ( 1%)
<br>
S01_DAT0004_1 77510737920 101 No Yes 14503647232 ( 19%) 952892480 ( 1%)
<br>
S01_DAT0006_1 77510737920 101 No Yes 14504486912 ( 19%) 949072512 ( 1%)
<br>
------------- -------------------- -------------------<br>
(pool total) 542575165440 108113510400 ( 20%) 6598146304 ( 1%)<br>
<br>
============= ==================== ===================<br>
(data) 542575165440 108113510400 ( 20%) 6598146304 ( 1%)<br>
(metadata) 3724541952 1938260992 ( 52%) 5906144 ( 0%)<br>
============= ==================== ===================<br>
(total) 546299707392 110051771392 ( 22%) 6604052448 ( 1%)<br>
<br>
Inode Information<br>
-----------------<br>
Total number of used inodes in all Inode spaces: 154807668<br>
Total number of free inodes in all Inode spaces: 12964492<br>
Total number of allocated inodes in all Inode spaces: 167772160<br>
Total of Maximum number of inodes in all Inode spaces: 276971520</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Arial",sans-serif">On the head node:</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New"">df -h<br>
Filesystem Size Used Avail Use% Mounted on<br>
/dev/sda4 430G 216G 215G 51% /<br>
devtmpfs 47G 0 47G 0% /dev<br>
tmpfs 47G 0 47G 0% /dev/shm<br>
tmpfs 47G 4.1G 43G 9% /run<br>
tmpfs 47G 0 47G 0% /sys/fs/cgroup<br>
/dev/sda1 504M 114M 365M 24% /boot<br>
/dev/sda2 100M 9.9M 90M 10% /boot/efi<br>
x.x.x.:/nfs-share 430G 326G 105G 76% /nfs-share<br>
cluster 506T 405T 101T 81% /cluster<br>
tmpfs 9.3G 0 9.3G 0% /run/user/443748<br>
tmpfs 9.3G 0 9.3G 0% /run/user/547288<br>
tmpfs 9.3G 0 9.3G 0% /run/user/551336<br>
tmpfs 9.3G 0 9.3G 0% /run/user/547289</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Arial",sans-serif">The login nodes have plenty of space in
</span><span style="font-family:"Courier New"">/var:</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Courier New"">/dev/sda3 50G 8.7G 42G 18% /var</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Arial",sans-serif">What else should we check? We are just at 81% on the GPFS mounted file system but that should be enough for more space without these
errors. Any recommended service(s) that we can restart?</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="https://urldefense.com/v3/__http:/gpfsug.org__;!!DZ3fjg!6G98Oo44gOmToQaQbotCPR6nxP_ox0xaRc9U6-ZjcWg8XMrwGvtt72VimFO6hSNnNkySuxawIVoA3_uEpMAUCh8$" target="_blank">
gpfsug.org</a><br>
<a href="https://urldefense.com/v3/__http:/gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org__;!!DZ3fjg!6G98Oo44gOmToQaQbotCPR6nxP_ox0xaRc9U6-ZjcWg8XMrwGvtt72VimFO6hSNnNkySuxawIVoA3_uEFX20ez8$" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org</a><o:p></o:p></p>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal">_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="https://urldefense.com/v3/__http:/gpfsug.org__;!!DZ3fjg!6G98Oo44gOmToQaQbotCPR6nxP_ox0xaRc9U6-ZjcWg8XMrwGvtt72VimFO6hSNnNkySuxawIVoA3_uEpMAUCh8$" target="_blank">
gpfsug.org</a><br>
<a href="https://urldefense.com/v3/__http:/gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org__;!!DZ3fjg!6G98Oo44gOmToQaQbotCPR6nxP_ox0xaRc9U6-ZjcWg8XMrwGvtt72VimFO6hSNnNkySuxawIVoA3_uEFX20ez8$" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org</a><o:p></o:p></p>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</body>
</html>