<font size=2 face="Calibri">Hi,</font><br><br><font size=2 face="Calibri">>The dirtyDirs file holds 130’000 lines,
which don’t seem to be so many. But dirtyDirDirents holds about 80M entries.
Can we estimate how long it will take to finish processing?</font><br><br><font size=2 face="sans-serif">Yes, this is the major problem fixed
as mentioned in the APAR below. The </font><font size=2 face="Calibri">dirtyDirs
</font><font size=2 face="sans-serif">file is opened for the each entry
in the </font><font size=2 face="Calibri">dirtyDirDirents </font><font size=2 face="sans-serif">file,
and this causes the performance overhead.</font><br><br><font size=2 face="Calibri">>At the moment all we can do is to wait?
We run version 5.0.2.3. Would version 5.0.3 or 5.0.4 show a different behavior?
Is this fixed/improved in a later release? </font><br><font size=2 face="Calibri">>There probably is no way to flush the
pending queue entries while recovery is ongoing?</font><br><br><font size=2 face="sans-serif">Later versions have the fix mentioned
in that APAR, and I believe it should fix the your current performance
issue. Flushing the pending queue entries is not avaible as of today (5.0.4),
we are currently working on this feature.</font><br><br><font size=2 face="sans-serif">~Venkat (vpuvvada@in.ibm.com)</font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">"Billich Heinrich
Rainer (ID SD)" <heinrich.billich@id.ethz.ch></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug main discussion
list <gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">01/13/2020 05:29 PM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">[EXTERNAL] Re:
[gpfsug-discuss] AFM Recovery of SW cache does a full scan of home - is
this to be expected?</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><font size=2 face="Calibri">Hello Venkat,</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">thank you, this seems to match our
issue. did trace tspcachescan and do see a long series of open()/read()/close()
to the dirtyDirs file. The dirtyDirs file holds 130’000 lines, which don’t
seem to be so many. But dirtyDirDirents holds about 80M entries. Can
we estimate how long it will take to finish processing?</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">tspcachescan does the following again and
again for different directories</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">11:11:36.837032 stat("/fs3101/XXXXX/.snapshots/XXXXX.afm.75872/yyyyy/yyyy",
{st_mode=S_IFDIR|0770, st_size=4096, ...}) = 0</font><br><font size=2 face="Calibri">11:11:36.837092 open("/var/mmfs/afm/fs3101-43/recovery/policylist.data.list.dirtyDirs",
O_RDONLY) = 8</font><br><font size=2 face="Calibri">11:11:36.837127 fstat(8, {st_mode=S_IFREG|0600,
st_size=32564140, ...}) = 0</font><br><font size=2 face="Calibri">11:11:36.837160 mmap(NULL, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x3fff96930000</font><br><font size=2 face="Calibri">11:11:36.837192 read(8, "539492355
65537 2795 553648131 "..., 8192) = 8192</font><br><font size=2 face="Calibri">11:11:36.837317 read(8, "Caches/com.apple.helpd/Generated"...,
8192) = 8192</font><br><font size=2 face="Calibri">11:11:36.837439 read(8, "ish\n539848852
1509237202 2795 5"..., 8192) = 8192</font><br><font size=2 face="Calibri">Many more reads</font><br><font size=2 face="Calibri">11:11:36.864104 close(8)
= 0</font><br><font size=2 face="Calibri">11:11:36.864135 munmap(0x3fff96930000,
8192) = 0</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">A single iteration takes about 27ms. Doing
this 130’000 times would be o.k., but if tspcachescan does it 80M times
we wait 600hours. Is there a way to estimate how many iteration tspcachescan
will do? The cache fileset holds 140M inodes.</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">At the moment all we can do is to wait?
We run version 5.0.2.3. Would version 5.0.3 or 5.0.4 show a different behavior?
Is this fixed/improved in a later release? </font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">There probably is no way to flush the pending
queue entries while recovery is ongoing?</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">I did open a case with IBM TS003219893
and will continue there.</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">Kind regards,</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">Heiner</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri"> </font><br><font size=3 face="Calibri"><b>From: </b><gpfsug-discuss-bounces@spectrumscale.org>
on behalf of Venkateswara R Puvvada <vpuvvada@in.ibm.com><b><br>Reply to: </b>gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><b><br>Date: </b>Monday, 13 January 2020 at 08:40<b><br>To: </b>gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><b><br>Subject: </b>Re: [gpfsug-discuss] AFM Recovery of SW cache does a full
scan of home - is this to be expected?</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Arial">AFM maintains in-memory queue at the gateway
node to keep track of changes happening on the fileset. If the in-memory
queue is lost (memory pressure, daemon shutdown etc..), AFM runs recovery
process which involves creating the snapshot, running the policy scan and
finally queueing the recovered operations. Due to message (operations)
dependency, any changes to the AFM fileset during the recovery won't get
replicated until the recovery the completion. AFM does the home directory
scan for only dirty directories to get the names of the deleted and
renamed files because old name for renamed file and deleted file name are
not available at the cache on disk. Directories are made dirty when there
is a rename or unlink operation is performed inside it. In your case
it may be that all the directories became dirty due to the rename/unlink
operations. AFM recovery process is single threaded.</font><font size=2 face="Calibri"><br></font><font size=2 face="Calibri"><br>>Is this to be expected and normal behavior? What to do about
it?<br>>Will every reboot of a gateway node trigger a recovery of all afm filesets
and a full scan of home? This would make normal rolling updates very
unpractical, or is there some better way?</font><font size=2 face="Calibri"><br></font><font size=2 face="Calibri"><br> Only for the dirty directories, see above.<br> <br>>Home is a gpfs cluster, hence we easily could produce the needed filelist
on home with a policyscan in a few minutes.<br> </font><font size=2 face="Arial"><br>There is some work going on to preserve the file names of the
unlinked/renamed files in the cache until they get replicated to
home so that home directory scan can be avoided.</font><font size=2 face="Calibri"><br></font><font size=2 face="Arial"><br>These are some issues fixed in this regard. What is the scale version ?</font><font size=2 face="Calibri"><br></font><font size=2 color=blue face="Calibri"><u><br></u></font><a href="https://www-01.ibm.com/support/docview.wss?uid=isg1IJ15436"><font size=2 color=blue face="Arial"><u>https://www-01.ibm.com/support/docview.wss?uid=isg1IJ15436</u></font></a><font size=2 face="Calibri"><br></font><font size=2 face="Arial"><br>~Venkat (vpuvvada@in.ibm.com)</font><font size=2 face="Calibri"><br><br><br></font><font size=1 color=#5f5f5f face="Arial"><br>From: </font><font size=1 face="Arial">"Billich
Heinrich Rainer (ID SD)" <heinrich.billich@id.ethz.ch></font><font size=1 color=#5f5f5f face="Arial"><br>To: </font><font size=1 face="Arial">gpfsug
main discussion list <gpfsug-discuss@spectrumscale.org></font><font size=1 color=#5f5f5f face="Arial"><br>Date: </font><font size=1 face="Arial">01/08/2020
10:32 PM</font><font size=1 color=#5f5f5f face="Arial"><br>Subject: </font><font size=1 face="Arial">[EXTERNAL]
[gpfsug-discuss] AFM Recovery of SW cache does a full scan of home - is
this to be expected?</font><font size=1 color=#5f5f5f face="Arial"><br>Sent by: </font><font size=1 face="Arial">gpfsug-discuss-bounces@spectrumscale.org</font><div align=center><hr noshade></div><br><font size=2 face="Calibri"><br><br></font><font size=2 face="Calibri"><br>Hello,<br> <br> <br>still new to AFM, so some basic question on how Recovery works for a SW
cache:<br> <br>we have an AFM SW cache in recovery mode – recovery first did run policies
on the cache cluster, but now I see a ‘tcpcachescan’ process on cache
slowly scanning home via nfs. Single host, single process, no parallelism
as far as I can see, but I may be wrong. This scan of home on a cache afmgateway
takes very long while further updates on cache queue up. Home has about
100M files. After 8hours I see about 70M entries in the file /var/mmfs/afm/…/recovery/homelist,
i.e. we get about 2500 lines/s. (We may have very many changes on
cache due to some recursive ACL operations, but I’m not sure.)<br> <br>So I expect that 12hours pass to buildup filelists before recovery starts
to update home. I see some risk: In this time new changes pile up on cache.
Memory may become an issue? Cache may fill up and we can’t evict?<br> <br>I wonder</font><font size=2 face="Calibri"> </font><ul><li><font size=2 face="Calibri">Is this to be expected and normal behavior?
What to do about it?</font><li><font size=2 face="Calibri">Will every reboot of a gateway node trigger
a recovery of all afm filesets and a full scan of home? This would make
normal rolling updates very unpractical, or is there some better
way?</font></ul><font size=2 face="Calibri"> <br>Home is a gpfs cluster, hence we easily could produce the needed filelist
on home with a policyscan in a few minutes.<br> <br>Thank you, I will welcome and clarification, advice or comments.<br> <br>Kind regards,<br> <br>Heiner<br>.<br> <br>-- </font><font size=1 color=#104160 face="Arial"><br>=======================<br>Heinrich Billich<br>ETH Zürich<br>Informatikdienste<br>Tel.: +41 44 632 72 56<br>heinrich.billich@id.ethz.ch<br>========================</font><font size=2 face="Calibri"><br> <br> <br> </font><font size=2 face="Courier New">_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org</font><font size=2 color=blue face="Calibri"><u><br></u></font><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><font size=2 color=blue face="Courier New"><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></a><font size=2 face="Calibri"><br><br><br><br></font><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br></font></tt><br><br><BR>