<font size=2>As Simon already mentioned, set the similar quotas at both
cache and home clusters to avoid the queue stuck problem due to quotas
being exceeds home.</font><br><br><br><font size=2>>At home we had replication of two so it wasn't straight
forward to set the same quotas on cache, we could just about fudge it for
user home directories but not for most of our project storage as we use
dependent fileaet >quotas.</font><br><br><br><font size=2>AFM will support dependent filesets from 5.0.4. Dependent
filesets can be created at the cache in the independent fileset and set
the same quotas from the home</font><br><br><font size=2>>We also saw issues with data in inode at home as this
doesn't work at AFM cache so it goes into a block. I've forgotten the exact
issues around that now.</font><br><br><font size=2 face="sans-serif">AFM uses some inode space to store the
remote file attributes like file handle, file times etc.. as part of the
EAs. If the file does not have hard links, maximum inode space used by
the AFM is around 200 bytes. AFM cache can store the file's data in the
inode if it have 200 bytes of more free space in the inode, otherwise file's
data will be stored in subblock rather than using the full block. For example
if the inode size is 4K at both cache and home, if the home file size is
3k and inode is using 300 bytes to store the file metadata, then free space
in the inode at the home will be 724 bytes(4096 - (3072 + 300)). When this
file is cached by the AFM , AFM adds internal EAs for 200 bytes, then the
free space in the inode at the cache will be 524 bytes(4096 - (3072 + 300
+ 200)). If the filesize is 3600 bytes at the home, AFM cannot store the
data in the inode at the cache. So AFM stores the file data in the block
only if it does not have enough space to store the internal EAs.</font><br><br><font size=2 face="sans-serif">~Venkat (vpuvvada@in.ibm.com)</font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Simon Thompson <S.J.Thompson@bham.ac.uk></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug main discussion
list <gpfsug-discuss@spectrumscale.org></font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">10/12/2019 01:52 AM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">[EXTERNAL] Re:
[gpfsug-discuss] Quotas and AFM</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><font size=2>Oh and I forgot. This only works if you precache th data
from home. Otherwise the disk usage at cache is only what you cached, as
you don't know what size it is from home.<br></font><br><font size=2>Unless something has changed recently at any rate.<br></font><br><font size=2>Simon</font><br><hr><br><font size=2 face="Calibri"><b>From:</b> gpfsug-discuss-bounces@spectrumscale.org
<gpfsug-discuss-bounces@spectrumscale.org> on behalf of Simon Thompson
<S.J.Thompson@bham.ac.uk><b><br>Sent:</b> Friday, October 11, 2019 9:10:20 PM<b><br>To:</b> gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><b><br>Subject:</b> Re: [gpfsug-discuss] Quotas and AFM</font><font size=3> </font><br><font size=3> </font><br><font size=2>Yes just set the quotas the same on both. Or a default
quota and have exceptions if that works in your case. But this was where
I think the inode in file is an issue if you have a lot of small files
as in the inode at home they don't consume quota I think but as they are
in a data block at cache they do. So it might now be quite so straightforward.<br></font><br><font size=2>And yes writes at home just get out of space, it's the
AFM cache that fails on the write back to home but then its in the queue
and can block it.<br></font><br><font size=2>Simon</font><br><hr><br><font size=2 face="Calibri"><b>From:</b> gpfsug-discuss-bounces@spectrumscale.org
<gpfsug-discuss-bounces@spectrumscale.org> on behalf of Ryan Novosielski
<novosirj@rutgers.edu><b><br>Sent:</b> Friday, October 11, 2019 9:05:15 PM<b><br>To:</b> gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><b><br>Subject:</b> Re: [gpfsug-discuss] Quotas and AFM</font><font size=3> </font><br><font size=3> </font><br><font size=3>Do you know is there anything that prevents me from just
setting the quotas the same on the IW cache, if there’s no way to inherit?
For the case of the home directories, it’s simple, as they are all 100G
with some exceptions, so a default user quota takes care of almost all
of it. Luckily, that’s right now where our problem is, but we have the
potential with other filesets later. </font><br><br><font size=3>I’m also wondering if you can confirm that I should /not/
need to be looking at people who are writing to the at home fileset, where
the quotas are set, as a problem syncing TO the cache, e.g. they don’t
add to the queue. I assume GPFS sees the over quota and just denies the
write, yes? I originally thought the problem was in that direction and
was totally perplexed about how it could be so stupid. 😅<br></font><br><font size=3>--<br>____<br>|| \\UTGERS, |---------------------------*O*---------------------------<br>||_// the State | Ryan Novosielski
- </font><a href="mailto:novosirj@rutgers.edu"><font size=3 color=blue><u>novosirj@rutgers.edu</u></font></a><font size=3><br>|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus<br>|| \\ of NJ | Office of Advanced Research
Computing - MSB C630, Newark<br> `'</font><br><br><font size=3>On Oct 11, 2019, at 15:56, Simon Thompson <S.J.Thompson@bham.ac.uk>
wrote:<br></font><br><font size=3> </font><br><font size=2>Yes.</font><br><br><font size=2>When we ran AFM, we had exactly this issue. What would
happen is that a user/fileset quota would be hit and a compute job would
continue writing. This would eventually fill the AFM queue. If you were
lucky you could stop and restart the queue and it would process other files
from other users but inevitably we'd get back to the same state. The solution
was to increase the quota at home to clear the queue, kill user workload
and then reduce their quota again.</font><br><br><font size=2>At home we had replication of two so it wasn't straight
forward to set the same quotas on cache, we could just about fudge it for
user home directories but not for most of our project storage as we use
dependent fileaet quotas.</font><br><br><font size=2>We also saw issues with data in inode at home as this
doesn't work at AFM cache so it goes into a block. I've forgotten the exact
issues around that now.</font><br><br><font size=3>So our experience was much like you describe.</font><br><br><font size=3>Simon</font><br><br><hr><br><font size=3><b>From:</b> <gpfsug-discuss-bounces@spectrumscale.org>
on behalf of Ryan Novosielski <novosirj@rutgers.edu><b><br>Sent:</b> Friday, 11 October 2019, 18:43<b><br>To:</b> gpfsug main discussion list<b><br>Subject:</b> [gpfsug-discuss] Quotas and AFM</font><br><font size=3><br>Does anyone have any good resources or experience with quotas and AFM caches?
Our scenario is that we have an AFM home one one site, an AFM cache on
another site, and then a client cluster on that remote site that mounts
the cache. The AFM filesets are IW. One of them contains our home directories,
which have a quota set on the home side. Quotas were disabled entirely
on the cache side (I enabled them recently, but did not set them to anything).
What I believe we’re running into is scary long AFM queues that are caused
by people writing an amount that is over the home quota to the cache, but
the cache is accepting it and then failing to sync back to the home because
the user is at their hard limit. I believe we’re also seeing delays on
unaffected users who are not over their quota, but that’s harder to tell.
We have the AFM gateways poorly/not tuned, so that is likely interacting.
Is there any way to make the quotas apparent to the cache cluster too,
beyond setting a quota there as well, or do I just fundamentally misunderstand
this in some other way? We really just want the quotas on the home cluster
to be enforced everywhere, more or less. Thanks! -- ____ || \\UTGERS, |---------------------------*O*---------------------------
||_// the State | Ryan Novosielski - novosirj@rutgers.edu || \\ University
| Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus || \\ of NJ
| Office of Advanced Research Computing - MSB C630, Newark `' _______________________________________________
gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org </font><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><font size=3>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></a><font size=3></font><br><font size=3>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><font size=3>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></a><tt><font size=2>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br></font></tt><br><br><BR>