<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
tt
{mso-style-priority:99;
font-family:"Courier New";}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Hello Kums,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Thank you; I could verify that the data of small files is in the inode. In the table below you see the filesize and the result of the tsdbfs query. Up to 3k all data is in the inode.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">The result of stat calls on small files is very puzzling. When data is in the inode stat reports one block of 512 bytes used, even for 3k of data. I don’t see how this would affect any of our applications,
so it’s just something to note.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">I append some output and the script to generate it, just for completeness.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Cheers,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Heiner<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"># stat -c "inode %i" $f | tsdbfs fs1301 | grep indirectionLevel<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">0 indirectionLevel=DIRECT status=USERFILE. <empty file is special><o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">1 indirectionLevel=INODE status=USERFILE<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">16 indirectionLevel=INODE status=USERFILE<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">512 indirectionLevel=INODE status=USERFILE<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">1k indirectionLevel=INODE status=USERFILE<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">2k indirectionLevel=INODE status=USERFILE<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">3k indirectionLevel=INODE status=USERFILE. < up to 3k data is in inode ><o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">4k indirectionLevel=DIRECT status=USERFILE<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">16k indirectionLevel=DIRECT status=USERFILE<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">64k indirectionLevel=DIRECT status=USERFILE<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">1M indirectionLevel=DIRECT status=USERFILE<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">2M indirectionLevel=DIRECT status=USERFILE<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Stat output<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"># stat -c “%n size: %s allocated: %b*%B”<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"># stat -c %n size: %s allocated: %b*%B<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">0 size: 0 allocated: 0*512<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">1 size: 1 allocated: 1*512<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">16 size: 16 allocated: 1*512<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">512 size: 512 allocated: 1*512<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">1k size: 1024 allocated: 1*512<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">2k size: 2048 allocated: 1*512<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">3k size: 3072 allocated: 1*512 < 3k file and all data in inode: stat reports 1*512 allocated/used)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">4k size: 4096 allocated: 64*512<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">16k size: 16384 allocated: 64*512. < as expected, 32k subblock size ><o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">64k size: 65536 allocated: 128*512<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">1M size: 1048576 allocated: 2048*512<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">2M size: 2097152 allocated: 4096*512<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">The script:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"># test-data-in-inode.sh<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">sizes="0 1 16 512 1k 2k 3k 4k 16k 64k 1M 2M"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">echo create files<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">for s in $sizes<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">do<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"> head -c $s /dev/zero > $s<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">done<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">echo sleep 20. # give gpfs some time to update metadata<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">sleep 20<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">echo<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">echo "# ls -ls $sizes"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">ls -ls $sizes<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">echo<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">echo "# stat -c %n size: %s allocated: %b*%B"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">stat -c "%n size: %s allocated: %b*%B" $sizes<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">echo<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">echo '# stat -c "inode %i" $f | tsdbfs fs1301 | grep indirectionLevel'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">for f in $sizes<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">do<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">echo -n $f<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">stat -c "inode %i" $f | tsdbfs fs1301 | grep indirectionLevel<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">done<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black"><gpfsug-discuss-bounces@spectrumscale.org> on behalf of Kumaran Rajaram <kums@us.ibm.com><br>
<b>Reply to: </b>gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
<b>Date: </b>Wednesday, 17 July 2019 at 14:38<br>
<b>To: </b>gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
<b>Subject: </b>Re: [gpfsug-discuss] How to prove that data is in inode<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Hi,</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif">>> How can I prove that data of a small file is stored in the inode (and not on a data nsd)?<br>
</span><br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif">You may use echo "inode file_inode_number" | tsdbfs fs_device | grep indirectionLevel and if it points to INODE, then the file is stored in the inodes</span><br>
<br>
<i><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:blue"># 4K Inode Size</span></i><br>
<i><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:blue"># mmlsfs gpfs3a | grep 'Inode size'</span></i><br>
<i><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:blue">-i 4096
<b>Inode size</b> in bytes</span></i><br>
<br>
<i><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:blue"># Small file
</span></i><br>
<i><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:blue"># ls -l /mnt/gpfs3a/hello.txt
</span></i><br>
<i><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:blue">-rw-r--r-- 1 root root 6 Jul 17 08:32 /mnt/gpfs3a/hello.txt</span></i><br>
<br>
<i><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:blue"># ls -i /mnt/gpfs3a/hello.txt
</span></i><br>
<i><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:blue">91649 /mnt/gpfs3a/hello.txt</span></i><br>
<br>
<i><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:blue">#File is inlined within Inode</span></i><br>
<i><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:blue"># echo "inode 91649" | tsdbfs gpfs3a | grep indirectionLevel</span></i><br>
<b><i><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:blue">indirectionLevel=INODE</span></i></b><i><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:blue"> status=USERFILE</span></i><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif">Regards,</span><br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif">-Kums</span><br>
<br>
<br>
<br>
<img width="16" height="16" style="width:.1666in;height:.1666in" id="_x0000_i1026" src="cid:image001.gif@01D53D83.F9076840" alt="Inactive hide details for "Billich Heinrich Rainer (ID SD)" ---07/17/2019 07:49:56 AM---Hello, How can I prove that data of a "><span style="font-size:10.0pt;color:#424282">"Billich
Heinrich Rainer (ID SD)" ---07/17/2019 07:49:56 AM---Hello, How can I prove that data of a small file is stored in the inode (and not on a data nsd)?</span><br>
<br>
<span style="font-size:10.0pt;color:#5F5F5F">From: </span><span style="font-size:10.0pt">"Billich Heinrich Rainer (ID SD)" <heinrich.billich@id.ethz.ch></span><br>
<span style="font-size:10.0pt;color:#5F5F5F">To: </span><span style="font-size:10.0pt">gpfsug main discussion list <gpfsug-discuss@spectrumscale.org></span><br>
<span style="font-size:10.0pt;color:#5F5F5F">Date: </span><span style="font-size:10.0pt">07/17/2019 07:49 AM</span><br>
<span style="font-size:10.0pt;color:#5F5F5F">Subject: </span><span style="font-size:10.0pt">[EXTERNAL] [gpfsug-discuss] How to prove that data is in inode</span><br>
<span style="font-size:10.0pt;color:#5F5F5F">Sent by: </span><span style="font-size:10.0pt">gpfsug-discuss-bounces@spectrumscale.org</span><o:p></o:p></p>
<div class="MsoNormal">
<hr size="2" width="100%" noshade="" style="color:#8091A5" align="left">
</div>
<p class="MsoNormal"><br>
<br>
<br>
<tt><span style="font-size:10.0pt">Hello,</span></tt><span style="font-size:10.0pt;font-family:"Courier New""><br>
<br>
<tt>How can I prove that data of a small file is stored in the inode (and not on a data nsd)?</tt><br>
<br>
<tt>We have a filesystem with 4k inodes on Scale 5.0.2 , but it seems there is no file data in the inodes?</tt><br>
<br>
<tt>I would expect that 'stat' reports 'Blocks: 0' for a small file, but I see 'Blocks:1'.</tt><br>
<br>
<tt>Cheers,</tt><br>
<br>
<tt>Heiner</tt><br>
<br>
<tt>I tried</tt><br>
<br>
<tt>[]# rm -f test; echo hello > test</tt><br>
<tt>[]# ls -ls test</tt><br>
<tt>1 -rw-r--r-- 1 root root 6 Jul 17 13:11 test</tt><br>
<tt>[root@testnas13ems01 test]# stat test</tt><br>
<tt> File: ‘test’</tt><br>
<tt> Size: 6 Blocks: 1 IO Block: 1048576 regular file</tt><br>
<tt>Device: 2dh/45d Inode: 353314 Links: 1</tt><br>
<tt>Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)</tt><br>
<tt>Access: 2019-07-17 13:11:03.037049000 +0200</tt><br>
<tt>Modify: 2019-07-17 13:11:03.037331000 +0200</tt><br>
<tt>Change: 2019-07-17 13:11:03.037259319 +0200</tt><br>
<tt>Birth: -</tt><br>
<tt>[root@testnas13ems01 test]# du test</tt><br>
<tt>1 test</tt><br>
<tt>[root@testnas13ems01 test]# du -b test</tt><br>
<tt>6 test</tt><br>
<tt>[root@testnas13ems01 test]#</tt><br>
<br>
<tt>Filesystem</tt><br>
<br>
<tt># mmlsfs f****</tt><br>
<tt>flag value description</tt><br>
<tt>------------------- ------------------------ -----------------------------------</tt><br>
<tt>-f 32768 Minimum fragment (subblock) size in bytes</tt><br>
<tt>-i 4096 Inode size in bytes</tt><br>
<tt>-I 32768 Indirect block size in bytes</tt><br>
<tt>-m 1 Default number of metadata replicas</tt><br>
<tt>-M 2 Maximum number of metadata replicas</tt><br>
<tt>-r 1 Default number of data replicas</tt><br>
<tt>-R 2 Maximum number of data replicas</tt><br>
<tt>-j cluster Block allocation type</tt><br>
<tt>-D nfs4 File locking semantics in effect</tt><br>
<tt>-k nfs4 ACL semantics in effect</tt><br>
<tt>-n 32 Estimated number of nodes that will mount file system</tt><br>
<tt>-B 1048576 Block size</tt><br>
<tt>-Q user;group;fileset Quotas accounting enabled</tt><br>
<tt> user;group;fileset Quotas enforced</tt><br>
<tt> user;group;fileset Default quotas enabled</tt><br>
<tt>--perfileset-quota Yes Per-fileset quota enforcement</tt><br>
<tt>--filesetdf Yes Fileset df enabled?</tt><br>
<tt>-V 20.01 (5.0.2.0) Current file system version</tt><br>
<tt> 15.01 (4.2.0.0) Original file system version</tt><br>
<tt>--create-time ***** 2017 File system creation time</tt><br>
<tt>-z No Is DMAPI enabled?</tt><br>
<tt>-L 33554432 Logfile size</tt><br>
<tt>-E Yes Exact mtime mount option</tt><br>
<tt>-S relatime Suppress atime mount option</tt><br>
<tt>-K whenpossible Strict replica allocation option</tt><br>
<tt>--fastea Yes Fast external attributes enabled?</tt><br>
<tt>--encryption No Encryption enabled?</tt><br>
<tt>--inode-limit 1294592 Maximum number of inodes in all inode spaces</tt><br>
<tt>--log-replicas 0 Number of log replicas</tt><br>
<tt>--is4KAligned Yes is4KAligned?</tt><br>
<tt>--rapid-repair Yes rapidRepair enabled?</tt><br>
<tt>--write-cache-threshold 0 HAWC Threshold (max 65536)</tt><br>
<tt>--subblocks-per-full-block 32 Number of subblocks per full block</tt><br>
<tt>-P system;data Disk storage pools in file system</tt><br>
<tt>--file-audit-log No File Audit Logging enabled?</tt><br>
<tt>--maintenance-mode No Maintenance Mode enabled?</tt><br>
<tt>-d ******</tt><br>
<tt>-A yes Automatic mount option</tt><br>
<tt>-o nfssync,nodev Additional mount options</tt><br>
<tt>-T /**** Default mount point</tt><br>
<tt>--mount-priority 0 Mount priority</tt><br>
<br>
<tt>-- </tt><br>
<tt>=======================</tt><br>
<tt>Heinrich Billich</tt><br>
<tt>ETH Zürich</tt><br>
<tt>Informatikdienste</tt><br>
<tt>Tel.: +41 44 632 72 56</tt><br>
<tt>heinrich.billich@id.ethz.ch</tt><br>
<tt>========================</tt><br>
<br>
<br>
<br>
<tt>_______________________________________________</tt><br>
<tt>gpfsug-discuss mailing list</tt><br>
<tt>gpfsug-discuss at spectrumscale.org</tt><br>
<tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> </tt><br>
<br>
</span><br>
<br>
<br>
<br>
<o:p></o:p></p>
</div>
</body>
</html>