<font size=2 face="sans-serif">AFM maintains in-memory queue at the gateway
node to keep track of changes happening on the fileset. If the in-memory
queue is lost (memory pressure, daemon shutdown etc..), AFM runs recovery
process which involves creating the snapshot, running the policy scan and
finally queueing the recovered operations. Due to message (operations)
dependency, any changes to the AFM fileset during the recovery won't get
replicated until the recovery the completion. AFM does the home directory
scan for only dirty directories to get the names of the deleted and
renamed files because old name for renamed file and deleted file name are
not available at the cache on disk. Directories are made dirty when there
is a rename or unlink operation is performed inside it. In your case
it may be that all the directories became dirty due to the rename/unlink
operations. AFM recovery process is single threaded.</font><br><br><font size=2 face="Calibri">>Is this to be expected and normal behavior?
What to do about it?</font><br><font size=2 face="Calibri">>Will every reboot of a gateway node
trigger a recovery of all afm filesets and a full scan of home? This would
make normal rolling updates very unpractical, or is there some better
way?</font><br><br><font size=2 face="Calibri"> Only for the dirty directories,
see above.</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">>Home is a gpfs cluster, hence we easily
could produce the needed filelist on home with a policyscan in a few minutes.</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="sans-serif">There is some work going on to preserve
the file names of the unlinked/renamed files in the cache
until they get replicated to home so that home directory scan can be avoided.</font><br><br><font size=2 face="sans-serif">These are some issues fixed in this
regard. What is the scale version ?</font><br><br><a href="https://www-01.ibm.com/support/docview.wss?uid=isg1IJ15436"><font size=2 color=blue face="sans-serif">https://www-01.ibm.com/support/docview.wss?uid=isg1IJ15436</font></a><br><br><font size=2 face="sans-serif">~Venkat (vpuvvada@in.ibm.com)</font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">"Billich Heinrich
Rainer (ID SD)" <heinrich.billich@id.ethz.ch></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug main discussion
list <gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">01/08/2020 10:32 PM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">[EXTERNAL] [gpfsug-discuss]
AFM Recovery of SW cache does a full scan of home - is this to be expected?</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><font size=2 face="Calibri">Hello,</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">still new to AFM, so some basic question
on how Recovery works for a SW cache:</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">we have an AFM SW cache in recovery mode
– recovery first did run policies on the cache cluster, but now I see
a ‘tcpcachescan’ process on cache slowly scanning home via nfs. Single
host, single process, no parallelism as far as I can see, but I may be
wrong. This scan of home on a cache afmgateway takes very long while further
updates on cache queue up. Home has about 100M files. After 8hours I see
about 70M entries in the file /var/mmfs/afm/…/recovery/homelist, i.e.
we get about 2500 lines/s. (We may have very many changes on cache
due to some recursive ACL operations, but I’m not sure.)</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">So I expect that 12hours pass to buildup
filelists before recovery starts to update home. I see some risk: In this
time new changes pile up on cache. Memory may become an issue? Cache may
fill up and we can’t evict?</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">I wonder</font><ul><li><font size=2 face="Calibri">Is this to be expected and normal behavior?
What to do about it?</font><li><font size=2 face="Calibri">Will every reboot of a gateway node trigger
a recovery of all afm filesets and a full scan of home? This would make
normal rolling updates very unpractical, or is there some better
way?</font></ul><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">Home is a gpfs cluster, hence we easily
could produce the needed filelist on home with a policyscan in a few minutes.</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">Thank you, I will welcome and clarification,
advice or comments.</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">Kind regards,</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">Heiner</font><br><font size=2 face="Calibri">.</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri">-- </font><br><font size=1 color=#104160 face="Arial">=======================</font><br><font size=1 color=#104160 face="Arial">Heinrich Billich</font><br><font size=1 color=#104160 face="Arial">ETH Zürich</font><br><font size=1 color=#104160 face="Arial">Informatikdienste</font><br><font size=1 color=#104160 face="Arial">Tel.: +41 44 632 72 56</font><br><font size=1 color=#104160 face="Arial">heinrich.billich@id.ethz.ch</font><br><font size=1 color=#104160 face="Arial">========================</font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri"> </font><br><font size=2 face="Calibri"> </font><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br></font></tt><br><br><BR>