<font size=2 face="sans-serif">1. What is the version of GPFS ? Eviction
should not be blocking the applications. Was partial file caching enabled
? Eviction cannot evict partially cached files in recent releases. Eviction
does not use space inside .afm directory, and its logs are stored under
/var/mmfs/tmp by default.</font><br><br><font size=2 face="sans-serif">2. I did not understand this requirement.</font><br><font size=2 face="sans-serif"> a. When IO to home fails with
quota exceeded or no space error , the messages are requeued at gateway
node and will be retried later (usually 15 minutes). Cache cannot
read home quotas today, and in most of the cases this is not valid. </font><br><font size=2 face="sans-serif"> b. When soft quota is exceeded
on AFM fileset, auto eviction clears data blocks on files based on
LRU policy to bring quota below soft limit. These evicted files are uncached
and there is no real migration of data to home during eviction. Eviction
should get triggered before fileset usage nearing hard quota and applications
getting errors.</font><br><br><font size=2 face="sans-serif">~Venkat (vpuvvada@in.ibm.com)</font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Jake Carroll <jake.carroll@uq.edu.au></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">"gpfsug-discuss@spectrumscale.org"
<gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">03/05/2017 02:05 AM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">[gpfsug-discuss]
Quota issues, eviction, AFM won't stop throwing data to a full location
- probably a rookie AFM mistake?</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><font size=3 face="Calibri">Hi all,</font><br><font size=3 face="Calibri"> </font><br><font size=3 face="Calibri">I think I need some help with GPFS quotas
and hard limits vs soft limits + eviction in AFM scenarios. We’ve got
a couple of issues:</font><br><font size=3 face="Calibri"> </font><br><font size=3 face="Calibri"><b>One:</b></font><br><font size=3 face="Calibri">-------</font><br><font size=3 face="Calibri">We’ve come across a scenario where if
a user hits the hard quota while ingesting into cache in an AFM “home
to cache” relationship whilst an eviction loop is being triggered, things
seem to go wrong – and the filesystem runs off into locking up territory.
The report I have on the last incident is that a file-set got stuck
at 100% (capacity utilisation), the eviction loop either failed or blocked
and the IO requests blocked and/or failed (this one I'm a little fuzzy
on). </font><br><font size=3 face="Calibri"> </font><br><font size=3 face="Calibri">Maybe it isn’t a bug and our guess is
that someone on here will probably tell us that the likely “fix” is to
right-size our high and low water marks appropriately. We considered a
potential bug mechanism or race condition if the eviction loop uses space
in the file-set in the .afm directory – but I then thought better of it
and though “<i>Nah, surely IBM would have thought of that!</i>”.</font><br><font size=3 face="Calibri"> </font><br><font size=3 face="Calibri"><b>Two:</b></font><br><font size=3 face="Calibri">-------</font><br><font size=3 face="Calibri">We witness a scenario where AFM doesn't
back off if it gets a filesystem full error code when trying to make the
cache clean in migrating data to “home”. If this takes a couple of seconds
to raise the error each attempt, gpfs/mmfsd will deplete NFS daemons causing
a DoS against the NFS server that is powering the cache/home relationship
for the AFM transport.<br></font><br><font size=3 face="Calibri">We had a mental model that AFM cache wouldn’t
or shouldn’t overload hard and soft quota as the high and low watermarks
for cache eviction policies. I guess in our heads, we’d like caches to
also enforce and respect <i>quotas</i> based on requests received from
home. There are probably lots of reasons this doesn’t make sense programmatically,
or to the rest of scale – but it would (seem to us) that it would clean
up this problem or at least some of it.</font><br><font size=3 face="Calibri"> </font><br><font size=3 face="Calibri">Happy to chat through his further and explain
it more if anyone is interested. If there are any AFM users out there,
we’d love to hear about how you deal with quotas, hwm/lwm and eviction
over-flow scenarios, if they exist in your environment.</font><br><font size=3 face="Calibri"> </font><br><font size=3 face="Calibri">Thank you as always, list.</font><br><font size=3 face="Calibri"> </font><br><font size=3 face="Calibri">-jc</font><br><font size=3 face="Calibri"> </font><br><font size=3 face="Calibri"> </font><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br></font></tt><br><br><BR>