<div dir="auto">That part is true however the FEK (File Encryption Key) goes into the inode and they can be very large, and you can have up to 8 of them.. so may be good having one or 2 FEKs but if you go to rotate FEK's, and need an extra 2 to handle the change you could run out of room.<div dir="auto"><br></div><div dir="auto">In our FPO we don't encrypt .ksh, .sh, and other source type files. We have another policy in place that reencrypts unencrypted files that are larger 1mb by creating them under a different file name extension and then move them back over. That's how we get around this limitation. We explain that if a file is less than 1mb it shouldn't have data we are worried about encryption.</div><div dir="auto"><br></div><div dir="auto">Alec</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Aug 2, 2023, 9:34 AM Wahl, Edward <<a href="mailto:ewahl@osc.edu">ewahl@osc.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="m_3380433451914484841WordSection1">
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">>Someone mentioned encryption will bypass this feature, but it's actually encryption that perhaps requires larger inode sizes to store all the key meta info (you can have up to 8 keys per inode I believe).<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I believe that is incorrect. If encryption is used, the size of the inode makes no difference. This is due to the fact that Only data, NOT metadata is encrypted on the file system. So storing blocks in MD spaces is out.
<br>
See the Scale documentation, and older GPFS documentation, for more information. (such as
<a href="https://www.ibm.com/docs/en/storage-scale/5.1.3?topic=administering-encryption" target="_blank" rel="noreferrer">
Encryption - IBM Documentation</a> ) Until such time as they start encrypting the metadata, it’s pointless to size MD for small files.
<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Ed Wahl<u></u><u></u></p>
<p class="MsoNormal">Ohio Supercomputer Center<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> gpfsug-discuss <<a href="mailto:gpfsug-discuss-bounces@gpfsug.org" target="_blank" rel="noreferrer">gpfsug-discuss-bounces@gpfsug.org</a>>
<b>On Behalf Of </b>Alec<br>
<b>Sent:</b> Wednesday, August 2, 2023 12:07 PM<br>
<b>To:</b> gpfsug main discussion list <<a href="mailto:gpfsug-discuss@gpfsug.org" target="_blank" rel="noreferrer">gpfsug-discuss@gpfsug.org</a>><br>
<b>Subject:</b> Re: [gpfsug-discuss] Inode size, and system pool subblock<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal"><span style="font-size:1.0pt;color:white">I think things are conflated here. . . The inode size is really just a call on how much functionality you need in an inode. I wouldn't even think about disk block
size when setting this. Essentially the smaller the inode the less space I <u></u>
<u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:1.0pt;color:white"><u></u><u></u></span></p>
</div>
<div>
<div>
<p class="MsoNormal">I think things are conflated here...<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">The inode size is really just a call on how much functionality you need in an inode. I wouldn't even think about disk block size when setting this. Essentially the smaller the inode the less space I need for metadata but also the less
capacity I have in my inode.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal">The default is 4k and if you don't change it then GPFS will put up to a 3.8k file in the inode itself vs going to an indirect disk allocation. Someone mentioned encryption will bypass this feature, but it's actually encryption that perhaps
requires larger inode sizes to store all the key meta info (you can have up to 8 keys per inode I believe).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">So essentially it you've got a smaller inode size your directories max size will max out sooner, your ACLs could be constrained, large file names can exhaust, you may not have enough space for Encryption details. But the upshot is you
need to dedicate less space to metadata and can handle more file entries. So if you've got billions of files and are managing replicas then you should consider fine tuning inode size down.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">You can go from 3.5% of space going to inodes to 1% if you went from 4k to 512 bytes.. but there is a reason GPFS defaults to 4k... And doesn't expand on it too much. If you've guessed wrong you're kind of hosed.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">None of this has to do with hardware block sizes, subblock allocation and fragment sizes. And further compounded by 4k native block sizes vs emulated 512 block size some disk hardware does.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">For GPFS you generally will have a very large block size 256kb or 1MB and GPFS will divide those blocks into 32 fragments. So you may have your smallest unit being a 8kb or 32kb fragment. If you have a dedicated MD pool (highly recommended)
you'd definitely specify a smaller block size than 1MB (128kb = 4kb fragments).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">The balance you're trying to strike here is the least amount of commands to retrieve your data efficiently. Think about the roundtrip on the bus being the same for a 4kb read vs a 1mb read so try to maximize this.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Generally the goal of the file system is to ensure that the excess data that is read when trying to pull fragments is as useless as possible.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I may also be confused but I wouldn't worry so much about inode size to block size.. just worry about getting large blocks working well for regular storage pool if your data is huge and using a smaller block size in MD if dedicate pool
which is almost always recommended.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Be very careful of specifying a small inode size because it's not just max filenames and max file counts in a directory.. it is much more.. and if you have a lot of small files don't underestimate the advantage of those files being stored
directly in the inode. A 512 byte inode could only store about a 380byte file vs a 4k file storing 3800 byte file. These files tend to be shell scripts and config files which you really don't want to be waiting around for and occupying a huge 1mb read for
and waisting a potentially larger 64kb fragment allocation on.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Alec<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Wed, Aug 2, 2023, 4:47 AM Olaf Weiser <<a href="mailto:olaf.weiser@de.ibm.com" target="_blank" rel="noreferrer">olaf.weiser@de.ibm.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black">Hallo Peter, <u></u>
<u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black;background:white">[1]
<i>[...] having a smaller inode size than the subblock size means</i></span><i><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"> there's a big wastage on disk usage, with no performance benefit to doing so[...]
</span></i><span style="font-size:12.0pt;color:black"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">in short - yes
</span><span style="font-size:10.0pt;font-family:"Segoe UI Emoji",sans-serif;color:black">😉</span><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">
</span><span style="font-size:12.0pt;color:black"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">[2]
<i>[...] I believe I'm correct in saying that inodes are not the only things to live on the metadata pool, so I assume that some other metadata might benefit from the larger block/subblock size. But looking at the number of inodes, the inode size, and the
space consumed in the system pool, it really looks like the majority of space consumed is by inodes.[...]
</i></span><span style="font-size:12.0pt;color:black"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">you may need to consider snapshots and directories , which all contributes to MD space</span><span style="font-size:12.0pt;color:black"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"> </span><span style="font-size:12.0pt;color:black"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">predicting the space requirements for MD for directories is always hard, because the size of a directory is depending on the file's name length, the users will create...
</span><span style="font-size:12.0pt;color:black"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">further more, using a less than 4k inode size makes also not much sense, when taking into account, that NVMEs and other modern block storage devices comes with
a hardware block size of 4k (even though GPFS still can deal with 512 Bytes per sector)
</span><span style="font-size:12.0pt;color:black"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">hope this helps ..
</span><span style="font-size:12.0pt;color:black"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><u></u> <u></u></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;color:black"><u></u> <u></u></span></p>
</div>
<div id="m_3380433451914484841m_-7736005608686638624m_735502529790437196Signature">
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><u></u> <u></u></p>
</div>
</div>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="98%" align="center">
</div>
<div id="m_3380433451914484841m_-7736005608686638624m_735502529790437196divRplyFwdMsg">
<p class="MsoNormal"><b><span style="color:black">Von:</span></b><span style="color:black"> gpfsug-discuss <<a href="mailto:gpfsug-discuss-bounces@gpfsug.org" target="_blank" rel="noreferrer">gpfsug-discuss-bounces@gpfsug.org</a>> im Auftrag von Peter Chase <<a href="mailto:peter.chase@metoffice.gov.uk" target="_blank" rel="noreferrer">peter.chase@metoffice.gov.uk</a>><br>
<b>Gesendet:</b> Mittwoch, 2. August 2023 11:09<br>
<b>An:</b> <a href="mailto:gpfsug-discuss@gpfsug.org" target="_blank" rel="noreferrer">gpfsug-discuss@gpfsug.org</a> <<a href="mailto:gpfsug-discuss@gpfsug.org" target="_blank" rel="noreferrer">gpfsug-discuss@gpfsug.org</a>><br>
<b>Betreff:</b> [EXTERNAL] [gpfsug-discuss] Inode size, and system pool subblock</span>
<u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:1.0pt;color:white">Good Morning, I have a question about inode size vs subblock size. Can anyone think of a reason that the chosen inode size of a scale filesystem should be smaller
than the subblock size for the metadata pool? I'm looking at an existing filesystem,
<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:1.0pt;color:white"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">Good Morning,<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">I have a question about inode size vs subblock size. Can anyone think of a reason that the chosen inode size of a scale filesystem should be smaller than the subblock
size for the metadata pool?<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">I'm looking at an existing filesystem, the inode size is 2KiB, and the subblock is 4KiB.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">It feels like I'm missing something. If I've understood the docs on blocks and subblocks correctly, it sounds like the subblock is the smallest atomic access size.
Meaning with a 4K subblock, and a 2K inode, reading the inode would return its contents and 2K of empty subblock every time. So, in my head (and maybe only there), <span style="background:white">having a smaller inode size than the subblock size means</span> there's
a big wastage on disk usage, with no performance benefit to doing so.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">I believe I'm correct in saying that inodes are not the only things to live on the metadata pool, so I assume that some other metadata might benefit from the larger
block/subblock size. But looking at the number of inodes, the inode size, and the space consumed in the system pool, it really looks like the majority of space consumed is by inodes.<u></u><u></u></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">As I said, I feel like I'm missing something, so if anyone can tell me where I'm wrong it would be greatly appreciated!<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">Sincerely,<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"><u></u> <u></u></span></p>
</div>
<div id="m_3380433451914484841m_-7736005608686638624m_735502529790437196x_Signature">
<div>
<div id="m_3380433451914484841m_-7736005608686638624m_735502529790437196x_divtagdefaultwrapper">
<p style="margin:0in"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">Pete Chase</span><span style="font-size:12.0pt;color:black"><u></u><u></u></span></p>
<p style="margin:0in"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">UKMO</span><span style="font-size:12.0pt;color:black"><u></u><u></u></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal">_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="https://urldefense.com/v3/__http:/gpfsug.org__;!!KGKeukY!wKer_px73AVXSgqasA-xymOOL3Y-Ln5AOyO_hz3e81yY2Y3Bx_IhmuPN87Q8-uneGQK5yacvKmWa$" target="_blank" rel="noreferrer">
gpfsug.org</a><br>
<a href="https://urldefense.com/v3/__http:/gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org__;!!KGKeukY!wKer_px73AVXSgqasA-xymOOL3Y-Ln5AOyO_hz3e81yY2Y3Bx_IhmuPN87Q8-uneGQK5yRmAg67I$" target="_blank" rel="noreferrer">http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org</a><u></u><u></u></p>
</blockquote>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://gpfsug.org" rel="noreferrer noreferrer" target="_blank">gpfsug.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org" rel="noreferrer noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org</a><br>
</blockquote></div>