<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
tt
{mso-style-priority:99;
font-family:"Courier New";}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:297076013;
mso-list-template-ids:-1618345678;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:"Courier New";
mso-bidi-font-family:"Times New Roman";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="DE-CH" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Hello Venkat,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">thank you, this seems to match our issue. did trace tspcachescan and do see a long series of open()/read()/close() to the dirtyDirs file. The dirtyDirs file holds 130’000 lines, which
don’t seem to be so many. But dirtyDirDirents holds about 80M entries. Can we estimate how long it will take to finish processing?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">tspcachescan does the following again and again for different directories<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">11:11:36.837032 stat("/fs3101/XXXXX/.snapshots/XXXXX.afm.75872/yyyyy/yyyy", {st_mode=S_IFDIR|0770, st_size=4096, ...}) = 0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">11:11:36.837092 open("/var/mmfs/afm/fs3101-43/recovery/policylist.data.list.dirtyDirs", O_RDONLY) = 8<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">11:11:36.837127 fstat(8, {st_mode=S_IFREG|0600, st_size=32564140, ...}) = 0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">11:11:36.837160 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x3fff96930000<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">11:11:36.837192 read(8, "539492355 65537 2795 553648131 "..., 8192) = 8192<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">11:11:36.837317 read(8, "Caches/com.apple.helpd/Generated"..., 8192) = 8192<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">11:11:36.837439 read(8, "ish\n539848852 1509237202 2795 5"..., 8192) = 8192<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Many more reads<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">11:11:36.864104 close(8) = 0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">11:11:36.864135 munmap(0x3fff96930000, 8192) = 0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">A single iteration takes about 27ms. Doing this 130’000 times would be o.k., but if tspcachescan does it 80M times we wait 600hours. Is there a way to estimate how many iteration tspcachescan
will do? The cache fileset holds 140M inodes.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">At the moment all we can do is to wait? We run version 5.0.2.3. Would version 5.0.3 or 5.0.4 show a different behavior? Is this fixed/improved in a later release?
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">There probably is no way to flush the pending queue entries while recovery is ongoing?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">I did open a case with IBM TS003219893 and will continue there.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Kind regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Heiner<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:12.0pt;color:black">From:
</span></b><span lang="EN-US" style="font-size:12.0pt;color:black"><gpfsug-discuss-bounces@spectrumscale.org> on behalf of Venkateswara R Puvvada <vpuvvada@in.ibm.com><br>
<b>Reply to: </b>gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
<b>Date: </b>Monday, 13 January 2020 at 08:40<br>
<b>To: </b>gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
<b>Subject: </b>Re: [gpfsug-discuss] AFM Recovery of SW cache does a full scan of home - is this to be expected?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">AFM maintains in-memory queue at the gateway node to keep track of changes happening on the fileset. If the in-memory queue is lost (memory pressure, daemon shutdown etc..),
AFM runs recovery process which involves creating the snapshot, running the policy scan and finally queueing the recovered operations. Due to message (operations) dependency, any changes to the AFM fileset during the recovery won't get replicated until the
recovery the completion. AFM does the home directory scan for only dirty directories to get the names of the deleted and renamed files because old name for renamed file and deleted file name are not available at the cache on disk. Directories are made dirty
when there is a rename or unlink operation is performed inside it. In your case it may be that all the directories became dirty due to the rename/unlink operations. AFM recovery process is single threaded.</span><br>
<br>
<span style="font-size:10.0pt">>Is this to be expected and normal behavior? What to do about it?</span><br>
<span style="font-size:10.0pt">>Will every reboot of a gateway node trigger a recovery of all afm filesets and a full scan of home? This would make normal rolling updates very unpractical, or is there some better way?</span><br>
<br>
<span style="font-size:10.0pt"> Only for the dirty directories, see above.</span><br>
<span style="font-size:10.0pt"> </span><br>
<span style="font-size:10.0pt">>Home is a gpfs cluster, hence we easily could produce the needed filelist on home with a policyscan in a few minutes.</span><br>
<span style="font-size:10.0pt"> </span><br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif">There is some work going on to preserve the file names of the unlinked/renamed files in the cache until they get replicated to home so that home directory scan can be avoided.</span><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif">These are some issues fixed in this regard. What is the scale version ?</span><br>
<br>
<a href="https://www-01.ibm.com/support/docview.wss?uid=isg1IJ15436"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">https://www-01.ibm.com/support/docview.wss?uid=isg1IJ15436</span></a><br>
<br>
<span style="font-size:10.0pt;font-family:"Arial",sans-serif">~Venkat (vpuvvada@in.ibm.com)</span><br>
<br>
<br>
<br>
<span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">From: </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif">"Billich Heinrich Rainer (ID SD)" <heinrich.billich@id.ethz.ch></span><br>
<span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">To: </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif">gpfsug main discussion list <gpfsug-discuss@spectrumscale.org></span><br>
<span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">Date: </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif">01/08/2020 10:32 PM</span><br>
<span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">Subject: </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif">[EXTERNAL] [gpfsug-discuss] AFM Recovery of SW cache does a full scan of home - is this to be expected?</span><br>
<span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#5F5F5F">Sent by: </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif">gpfsug-discuss-bounces@spectrumscale.org</span><o:p></o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" noshade="" style="color:#A0A0A0" align="center">
</div>
<p class="MsoNormal"><br>
<br>
<br>
<span style="font-size:10.0pt">Hello,</span><br>
<span style="font-size:10.0pt"> </span><br>
<span style="font-size:10.0pt"> </span><br>
<span style="font-size:10.0pt">still new to AFM, so some basic question on how Recovery works for a SW cache:</span><br>
<span style="font-size:10.0pt"> </span><br>
<span style="font-size:10.0pt">we have an AFM SW cache in recovery mode – recovery first did run policies on the cache cluster, but now I see a ‘tcpcachescan’ process on cache slowly scanning home via nfs. Single host, single process, no parallelism as far
as I can see, but I may be wrong. This scan of home on a cache afmgateway takes very long while further updates on cache queue up. Home has about 100M files. After 8hours I see about 70M entries in the file /var/mmfs/afm/…/recovery/homelist, i.e. we get about
2500 lines/s. (We may have very many changes on cache due to some recursive ACL operations, but I’m not sure.)</span><br>
<span style="font-size:10.0pt"> </span><br>
<span style="font-size:10.0pt">So I expect that 12hours pass to buildup filelists before recovery starts to update home. I see some risk: In this time new changes pile up on cache. Memory may become an issue? Cache may fill up and we can’t evict?</span><br>
<span style="font-size:10.0pt"> </span><br>
<span style="font-size:10.0pt">I wonder</span> <o:p></o:p></p>
<ul type="disc">
<li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
<span style="font-size:10.0pt">Is this to be expected and normal behavior? What to do about it?</span><o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
<span style="font-size:10.0pt">Will every reboot of a gateway node trigger a recovery of all afm filesets and a full scan of home? This would make normal rolling updates very unpractical, or is there some better way?</span><o:p></o:p></li></ul>
<p class="MsoNormal"><span style="font-size:10.0pt"> </span><br>
<span style="font-size:10.0pt">Home is a gpfs cluster, hence we easily could produce the needed filelist on home with a policyscan in a few minutes.</span><br>
<span style="font-size:10.0pt"> </span><br>
<span style="font-size:10.0pt">Thank you, I will welcome and clarification, advice or comments.</span><br>
<span style="font-size:10.0pt"> </span><br>
<span style="font-size:10.0pt">Kind regards,</span><br>
<span style="font-size:10.0pt"> </span><br>
<span style="font-size:10.0pt">Heiner</span><br>
<span style="font-size:10.0pt">.</span><br>
<span style="font-size:10.0pt"> </span><br>
<span style="font-size:10.0pt">-- </span><br>
<span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#104160">=======================</span><br>
<span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#104160">Heinrich Billich</span><br>
<span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#104160">ETH Zürich</span><br>
<span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#104160">Informatikdienste</span><br>
<span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#104160">Tel.: +41 44 632 72 56</span><br>
<span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#104160">heinrich.billich@id.ethz.ch</span><br>
<span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#104160">========================</span><br>
<span style="font-size:10.0pt"> </span><br>
<span style="font-size:10.0pt"> </span><br>
<span style="font-size:10.0pt"> </span><tt><span style="font-size:10.0pt">_______________________________________________</span></tt><span style="font-size:10.0pt;font-family:"Courier New""><br>
<tt>gpfsug-discuss mailing list</tt><br>
<tt>gpfsug-discuss at spectrumscale.org</tt><br>
</span><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><span style="font-size:10.0pt">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></tt></a><span style="font-size:10.0pt;font-family:"Courier New""><br>
</span><br>
<br>
<br>
<br>
<o:p></o:p></p>
</div>
</body>
</html>