<font size=2 face="sans-serif">You are correct about mmchfs, you
can increase the inode maximum but once an inode is allocated it cannot
be de-allocated in the sense that the space can be recovered. You
can of course decreased the inode maximum to a value equal to the used
and allocated inodes but that would not help you here. Providing
more metadata space via additional NSDs seems your most expedient
option to address the issue.</font><br><br><font size=3 face="sans-serif">Fred<br>__________________________________________________<br>Fred Stock | IBM Pittsburgh Lab | 720-430-8821<br>stockf@us.ibm.com</font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">"Buterbaugh, Kevin
L" <Kevin.Buterbaugh@Vanderbilt.Edu></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug main discussion
list <gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">01/23/2018 01:10 PM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
Metadata only system pool</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><font size=3>Hi All, </font><br><br><font size=3>I do have metadata replication set to two, so Alex, does
that make more sense?</font><br><br><font size=3>And I had forgotten about indirect blocks for large files,
which actually makes sense with the user in question … my apologies for
that … due to a very gravely ill pet and a recovering at home from pneumonia
family member I’m way more sleep deprived right now than I’d like. :-(</font><br><br><font size=3>Fred - I think you’ve already answered this … but mmchfs
can only create / allocate more inodes … it cannot be used to shrink the
number of inodes? That would make sense, and if that’s the case
then I can allocate more NSDs to the system pool.</font><br><br><font size=3>Thanks…</font><br><br><font size=3>Kevin</font><br><br><font size=3>On Jan 23, 2018, at 11:27 AM, Alex Chekholko <</font><a href=mailto:alex@calicolabs.com><font size=3 color=blue><u>alex@calicolabs.com</u></font></a><font size=3>>
wrote:</font><br><br><font size=3>2.8TB seems quite high for only 350M inodes. Are
you sure you only have metadata in there?</font><br><br><font size=3>On Tue, Jan 23, 2018 at 9:25 AM, Frederick Stock <</font><a href=mailto:stockf@us.ibm.com target=_blank><font size=3 color=blue><u>stockf@us.ibm.com</u></font></a><font size=3>>
wrote:</font><br><font size=2 face="sans-serif">One possibility is the creation/expansion
of directories or allocation of indirect blocks for large files.</font><font size=3><br></font><font size=2 face="sans-serif"><br>Not sure if this is the issue here but at one time inode allocation was
considered slow and so folks may have pre-allocated inodes to avoid that
overhead during file creation. To my understanding inode creation
time is not so slow that users need to pre-allocate inodes. Yes,
there are likely some applications where pre-allocating may be necessary
but I expect they would be the exception. I mention this because
you have a lot of free inodes and of course once they are allocated they
cannot be de-allocated. </font><font size=3><br></font><font size=3 face="sans-serif"><br>Fred<br>__________________________________________________<br>Fred Stock | IBM Pittsburgh Lab | </font><a href="tel:(720)%20430-8821" target=_blank><font size=3 color=blue face="sans-serif"><u>720-430-8821</u></font></a><font size=3 color=blue face="sans-serif"><u><br></u></font><a href=mailto:stockf@us.ibm.com target=_blank><font size=3 color=blue face="sans-serif"><u>stockf@us.ibm.com</u></font></a><font size=3><br><br><br></font><font size=1 color=#5f5f5f face="sans-serif"><br>From: </font><font size=1 face="sans-serif">"Buterbaugh,
Kevin L" <Kevin.Buterbaugh@Vanderbilt.Edu></font><font size=1 color=#5f5f5f face="sans-serif"><br>To: </font><font size=1 face="sans-serif">gpfsug
main discussion list <</font><a href="mailto:gpfsug-discuss@spectrumscale.org" target=_blank><font size=1 color=blue face="sans-serif"><u>gpfsug-discuss@spectrumscale.org</u></font></a><font size=1 face="sans-serif">></font><font size=1 color=#5f5f5f face="sans-serif"><br>Date: </font><font size=1 face="sans-serif">01/23/2018
12:17 PM</font><font size=1 color=#5f5f5f face="sans-serif"><br>Subject: </font><font size=1 face="sans-serif">[gpfsug-discuss]
Metadata only system pool</font><font size=1 color=#5f5f5f face="sans-serif"><br>Sent by: </font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target=_blank><font size=1 color=blue face="sans-serif"><u>gpfsug-discuss-bounces@spectrumscale.org</u></font></a><font size=3><br></font><hr noshade><br><font size=3><br><br><br>Hi All, <br><br>I was under the (possibly false) impression that if you have a filesystem
where the system pool contains metadata only then the only thing that would
cause the amount of free space in that pool to change is the creation of
more inodes … is that correct? In other words, given that I have
a filesystem with 130 million free (but allocated) inodes:<br><br>Inode Information<br>-----------------<br>Number of used inodes: 218635454<br>Number of free inodes: 131364674<br>Number of allocated inodes: 350000128<br>Maximum number of inodes: 350000128<br><br>I would not expect that a user creating a few hundred or thousands of files
could cause a “no space left on device” error (which I’ve got one user
getting). There’s plenty of free data space, BTW.<br><br>Now my system pool is almost “full”:<br><br>(pool total) 2.878T
34M ( 0%) 140.9M
( 0%)<br><br>But again, what - outside of me creating more inodes - would cause that
to change??<br><br>Thanks…<br><br>Kevin<br><br>—<br>Kevin Buterbaugh - Senior System Administrator<br>Vanderbilt University - Advanced Computing Center for Research and Education</font><font size=3 color=blue><u><br></u></font><a href=mailto:Kevin.Buterbaugh@vanderbilt.edu target=_blank><font size=3 color=blue><u>Kevin.Buterbaugh@vanderbilt.edu</u></font></a><font size=3>-
</font><a href="tel:(615)%20875-9633" target=_blank><font size=3 color=blue><u>(615)875-9633</u></font></a><font size=3><br><br></font><br><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font></tt><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253A-252F-252Fspectrumscale.org-26data-3D02-257C01-257CKevin.Buterbaugh-2540vanderbilt.edu-257C1607a3fe872e4241587b08d56286a746-257Cba5a7f39e3be4ab3b45067fa80faecad-257C0-257C1-257C636523252830007825-26sdata-3Dh-252BCEzaXFYl-252By89m3IVPo960AeN2CL7idGpgCLZgOip8-253D-26reserved-3D0&d=DwMGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=fiiMOociXV9hsufScVc2JiRsOMFP-VQALqdlqN9U0HU&s=lEzvkGBlhPKzNOSp-F7bbQlnY1oFRUBoNjE1AL4aOoc&e=" target=_blank><tt><font size=2 color=blue><u>spectrumscale.org</u></font></tt></a><font size=3 color=blue><u><br></u></font><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Furldefense.proofpoint.com-252Fv2-252Furl-253Fu-253Dhttp-2D3A-5F-5Fgpfsug.org-5Fmailman-5Flistinfo-5Fgpfsug-2D2Ddiscuss-2526d-253DDwICAg-2526c-253Djf-5FiaSHvJObTbx-2DsiA1ZOg-2526r-253Dp-5F1XEUyoJ7-2DVJxF-5Fw8h9gJh8-5FWj0Pey73LCLLoxodpw-2526m-253Dgou0xYZwz8M-2D5i8mT6Tthafi8JW2aMrzQGMK1hUEUls-2526s-253DjcHOB-5FvmJjE8PnrpfHqzMkm1nk6QWwkn2npTEP6kcKs-2526e-253D-26data-3D02-257C01-257CKevin.Buterbaugh-2540vanderbilt.edu-257C1607a3fe872e4241587b08d56286a746-257Cba5a7f39e3be4ab3b45067fa80faecad-257C0-257C0-257C636523252830007825-26sdata-3D-252B87gWCFaIiPJUDwTWW5KxsR11rJJB6o6EfrIgIOGPKE-253D-26reserved-3D0&d=DwMGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=fiiMOociXV9hsufScVc2JiRsOMFP-VQALqdlqN9U0HU&s=Jp253kPz84WcXW-bNxgfjIgH2TUMuLzxgQCU-FAefRE&e=" target=_blank><tt><font size=2 color=blue><u>https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=gou0xYZwz8M-5i8mT6Tthafi8JW2aMrzQGMK1hUEUls&s=jcHOB_vmJjE8PnrpfHqzMkm1nk6QWwkn2npTEP6kcKs&e=</u></font></tt></a><font size=3><br><br><br><br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253A-252F-252Fspectrumscale.org-26data-3D02-257C01-257CKevin.Buterbaugh-2540vanderbilt.edu-257C1607a3fe872e4241587b08d56286a746-257Cba5a7f39e3be4ab3b45067fa80faecad-257C0-257C1-257C636523252830007825-26sdata-3Dh-252BCEzaXFYl-252By89m3IVPo960AeN2CL7idGpgCLZgOip8-253D-26reserved-3D0&d=DwMGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=fiiMOociXV9hsufScVc2JiRsOMFP-VQALqdlqN9U0HU&s=lEzvkGBlhPKzNOSp-F7bbQlnY1oFRUBoNjE1AL4aOoc&e=" target=_blank><font size=3 color=blue><u>spectrumscale.org</u></font></a><font size=3 color=blue><u><br></u></font><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253A-252F-252Fgpfsug.org-252Fmailman-252Flistinfo-252Fgpfsug-2Ddiscuss-26data-3D02-257C01-257CKevin.Buterbaugh-2540vanderbilt.edu-257C1607a3fe872e4241587b08d56286a746-257Cba5a7f39e3be4ab3b45067fa80faecad-257C0-257C1-257C636523252830007825-26sdata-3DrIFx3lzbAIH5SZtFxJsVqWMMSo-252F0LssNc4K4tZH3uQc-253D-26reserved-3D0&d=DwMGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=fiiMOociXV9hsufScVc2JiRsOMFP-VQALqdlqN9U0HU&s=DD5ADwMibpZ8VIayWssXqo2JGroNVYSlvyHIvpQ7u3w&e=" target=_blank><font size=3 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></a><font size=3><br></font><br><br><font size=3>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=fiiMOociXV9hsufScVc2JiRsOMFP-VQALqdlqN9U0HU&s=gh9x62oyxcdXWk7gd7RyLn4vZiFeEcwB1wdOwH95vRc&e="><font size=3 color=blue><u>spectrumscale.org</u></font></a><font size=3 color=blue><u><br></u></font><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttp-253A-252F-252Fgpfsug.org-252Fmailman-252Flistinfo-252Fgpfsug-2Ddiscuss-26data-3D02-257C01-257CKevin.Buterbaugh-2540vanderbilt.edu-257C1607a3fe872e4241587b08d56286a746-257Cba5a7f39e3be4ab3b45067fa80faecad-257C0-257C1-257C636523252830007825-26sdata-3DrIFx3lzbAIH5SZtFxJsVqWMMSo-252F0LssNc4K4tZH3uQc-253D-26reserved-3D0&d=DwMGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=fiiMOociXV9hsufScVc2JiRsOMFP-VQALqdlqN9U0HU&s=DD5ADwMibpZ8VIayWssXqo2JGroNVYSlvyHIvpQ7u3w&e="><font size=3 color=blue><u>https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C1607a3fe872e4241587b08d56286a746%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C1%7C636523252830007825&sdata=rIFx3lzbAIH5SZtFxJsVqWMMSo%2F0LssNc4K4tZH3uQc%3D&reserved=0</u></font></a><br><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=fiiMOociXV9hsufScVc2JiRsOMFP-VQALqdlqN9U0HU&s=LN14zBEOYVrP2YRk3of_f08Ok5f256m7SYf1xL0qDvU&e="><tt><font size=2>https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=fiiMOociXV9hsufScVc2JiRsOMFP-VQALqdlqN9U0HU&s=LN14zBEOYVrP2YRk3of_f08Ok5f256m7SYf1xL0qDvU&e=</font></tt></a><tt><font size=2><br></font></tt><br><br><BR>