<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;
panose-1:2 11 0 4 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:12.0pt;
font-family:"Aptos",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Aptos",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Are you seeing the issues across the whole file system or in certain areas? That sounds like inode exhaustion to me (and based on it not being block exhaustion as you’ve demonstrated).
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">What does a “df -i /cluster” show you? Or if this is only in a certain area you can “cd” into that directory and run a “df -i .”<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">You may need to allocate more inodes to an independent inode fileset somewhere. Especially with something as old as 4.2.3 you won’t have auto-inode expansion for the filesets.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Best,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black">J.D. Maloney</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">Lead</span><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black"> HPC Storage Engineer | Storage Enabling Technologies Group</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:black">National Center for Supercomputing Applications (NCSA)</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div id="mail-editor-reference-message-container">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="color:black">From:
</span></b><span style="color:black">gpfsug-discuss <gpfsug-discuss-bounces@gpfsug.org> on behalf of Rob Kudyba <rk3199@columbia.edu><br>
<b>Date: </b>Thursday, June 6, 2024 at 3:50</span><span style="font-family:"Arial",sans-serif;color:black"> </span><span style="color:black">PM<br>
<b>To: </b>gpfsug-discuss@gpfsug.org <gpfsug-discuss@gpfsug.org><br>
<b>Subject: </b>[gpfsug-discuss] No space left on device, but plenty of quota space for inodes and blocks<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">Running GPFS 4.2.3 on a DDN GridScaler and users are getting the <span style="font-family:"Courier New"">No space left on device</span> message when trying to write to a file. In <span style="font-family:"Courier New"">/var/adm/ras/mmfs.log
</span>the only recent errors are this:<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">2024-06-06_15:51:22.311-0400: mmcommon getContactNodes cluster failed. Return code -1.<br>
2024-06-06_15:51:22.311-0400: The previous error was detected on node x.x.x.x (headnode).<br>
2024-06-06_15:53:25.088-0400: mmcommon getContactNodes cluster failed. Return code -1.<br>
2024-06-06_15:53:25.088-0400: The previous error was detected on node x.x.x.x (headnode).</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">according to <a href="https://urldefense.com/v3/__https:/www.ibm.com/docs/en/storage-scale/5.1.9?topic=messages-6027-615__;!!DZ3fjg!4ZyUNmTiGNp6C3Yls1wqW-RdRGa8n-ZmfZ0y0i-y6pce_ZIFSaefpOWvKIYIXspKjfREPtf3BRuO5VqAS6Y9UXQ$">
https://www.ibm.com/docs/en/storage-scale/5.1.9?topic=messages-6027-615</a> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal">Check the preceding messages, and consult the earlier chapters of this document. A frequent cause for such errors is lack of space in
<span style="font-family:"Courier New"">/var</span>.<o:p></o:p></p>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We have plenty of space left.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> /usr/lpp/mmfs/bin/mmlsdisk cluster<br>
disk driver sector failure holds holds storage<br>
name type size group metadata data status availability pool<br>
------------ -------- ------ ----------- -------- ----- ------------- ------------ ------------<br>
S01_MDT200_1 nsd 4096 200 Yes No ready up system
<br>
S01_MDT201_1 nsd 4096 201 Yes No ready up system
<br>
S01_DAT0001_1 nsd 4096 100 No Yes ready up data1 <br>
S01_DAT0002_1 nsd 4096 101 No Yes ready up data1 <br>
S01_DAT0003_1 nsd 4096 100 No Yes ready up data1 <br>
S01_DAT0004_1 nsd 4096 101 No Yes ready up data1 <br>
S01_DAT0005_1 nsd 4096 100 No Yes ready up data1 <br>
S01_DAT0006_1 nsd 4096 101 No Yes ready up data1 <br>
S01_DAT0007_1 nsd 4096 100 No Yes ready up data1 </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> /usr/lpp/mmfs/bin/mmdf headnode <br>
disk disk size failure holds holds free KB free KB<br>
name in KB group metadata data in full blocks in fragments<br>
--------------- ------------- -------- -------- ----- -------------------- -------------------<br>
Disks in storage pool: system (Maximum disk size allowed is 14 TB)<br>
S01_MDT200_1 1862270976 200 Yes No 969134848 ( 52%) 2948720 ( 0%)
<br>
S01_MDT201_1 1862270976 201 Yes No 969126144 ( 52%) 2957424 ( 0%)
<br>
------------- -------------------- -------------------<br>
(pool total) 3724541952 1938260992 ( 52%) 5906144 ( 0%)<br>
<br>
Disks in storage pool: data1 (Maximum disk size allowed is 578 TB)<br>
S01_DAT0007_1 77510737920 100 No Yes 21080752128 ( 27%) 897723392 ( 1%)
<br>
S01_DAT0005_1 77510737920 100 No Yes 14507212800 ( 19%) 949412160 ( 1%)
<br>
S01_DAT0001_1 77510737920 100 No Yes 14503620608 ( 19%) 951327680 ( 1%)
<br>
S01_DAT0003_1 77510737920 100 No Yes 14509205504 ( 19%) 949340544 ( 1%)
<br>
S01_DAT0002_1 77510737920 101 No Yes 14504585216 ( 19%) 948377536 ( 1%)
<br>
S01_DAT0004_1 77510737920 101 No Yes 14503647232 ( 19%) 952892480 ( 1%)
<br>
S01_DAT0006_1 77510737920 101 No Yes 14504486912 ( 19%) 949072512 ( 1%)
<br>
------------- -------------------- -------------------<br>
(pool total) 542575165440 108113510400 ( 20%) 6598146304 ( 1%)<br>
<br>
============= ==================== ===================<br>
(data) 542575165440 108113510400 ( 20%) 6598146304 ( 1%)<br>
(metadata) 3724541952 1938260992 ( 52%) 5906144 ( 0%)<br>
============= ==================== ===================<br>
(total) 546299707392 110051771392 ( 22%) 6604052448 ( 1%)<br>
<br>
Inode Information<br>
-----------------<br>
Total number of used inodes in all Inode spaces: 154807668<br>
Total number of free inodes in all Inode spaces: 12964492<br>
Total number of allocated inodes in all Inode spaces: 167772160<br>
Total of Maximum number of inodes in all Inode spaces: 276971520</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">On the head node:</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">df -h<br>
Filesystem Size Used Avail Use% Mounted on<br>
/dev/sda4 430G 216G 215G 51% /<br>
devtmpfs 47G 0 47G 0% /dev<br>
tmpfs 47G 0 47G 0% /dev/shm<br>
tmpfs 47G 4.1G 43G 9% /run<br>
tmpfs 47G 0 47G 0% /sys/fs/cgroup<br>
/dev/sda1 504M 114M 365M 24% /boot<br>
/dev/sda2 100M 9.9M 90M 10% /boot/efi<br>
x.x.x.:/nfs-share 430G 326G 105G 76% /nfs-share<br>
cluster 506T 405T 101T 81% /cluster<br>
tmpfs 9.3G 0 9.3G 0% /run/user/443748<br>
tmpfs 9.3G 0 9.3G 0% /run/user/547288<br>
tmpfs 9.3G 0 9.3G 0% /run/user/551336<br>
tmpfs 9.3G 0 9.3G 0% /run/user/547289</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">The login nodes have plenty of space in
</span><span style="font-family:"Courier New"">/var:</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">/dev/sda3 50G 8.7G 42G 18% /var</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">What else should we check? We are just at 81% on the GPFS mounted file system but that should be enough for more space without these errors. Any recommended service(s) that we can restart?</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
</div>
</body>
</html>