<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body dir="auto">
Do you know is there anything that prevents me from just setting the quotas the same on the IW cache, if there’s no way to inherit? For the case of the home directories, it’s simple, as they are all 100G with some exceptions, so a default user quota takes care
of almost all of it. Luckily, that’s right now where our problem is, but we have the potential with other filesets later.
<div><br>
</div>
<div>I’m also wondering if you can confirm that I should /not/ need to be looking at people who are writing to the at home fileset, where the quotas are set, as a problem syncing TO the cache, e.g. they don’t add to the queue. I assume GPFS sees the over quota
and just denies the write, yes? I originally thought the problem was in that direction and was totally perplexed about how it could be so stupid. 😅<br>
<br>
<div dir="ltr"><span style="background-color: rgba(255, 255, 255, 0);">--<br>
____<br>
|| \\UTGERS, |---------------------------*O*---------------------------<br>
||_// the State | Ryan Novosielski - <a href="mailto:novosirj@rutgers.edu" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="link" x-apple-data-detectors-result="1">novosirj@rutgers.edu</a><br>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus<br>
|| \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark<br>
`'</span></div>
<div dir="ltr"><br>
<blockquote type="cite">On Oct 11, 2019, at 15:56, Simon Thompson <S.J.Thompson@bham.ac.uk> wrote:<br>
<br>
</blockquote>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr" text-align="left">
<div dir="auto" style="direction: ltr; margin: 0px; padding: 0px; font-family: sans-serif; font-size: 11pt; color: black;">
Yes.<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0px; padding: 0px; font-family: sans-serif; font-size: 11pt; color: black;">
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0px; padding: 0px; font-family: sans-serif; font-size: 11pt; color: black;">
When we ran AFM, we had exactly this issue. What would happen is that a user/fileset quota would be hit and a compute job would continue writing. This would eventually fill the AFM queue. If you were lucky you could stop and restart the queue and it would process
other files from other users but inevitably we'd get back to the same state. The solution was to increase the quota at home to clear the queue, kill user workload and then reduce their quota again.</div>
<div dir="ltr" style="direction: ltr; margin: 0px; padding: 0px; font-family: sans-serif; font-size: 11pt; color: black; text-align: left;">
<br>
</div>
<div dir="ltr" style="direction: ltr; margin: 0px; padding: 0px; font-family: sans-serif; font-size: 11pt; color: black; text-align: left;">
At home we had replication of two so it wasn't straight forward to set the same quotas on cache, we could just about fudge it for user home directories but not for most of our project storage as we use dependent fileaet quotas.</div>
<div dir="ltr" style="direction: ltr; margin: 0px; padding: 0px; font-family: sans-serif; font-size: 11pt; color: black; text-align: left;">
<br>
</div>
<div dir="ltr" style="direction: ltr; margin: 0px; padding: 0px; font-family: sans-serif; font-size: 11pt; color: black; text-align: left;">
We also saw issues with data in inode at home as this doesn't work at AFM cache so it goes into a block. I've forgotten the exact issues around that now.</div>
<div id="ms-outlook-mobile-signature" dir="ltr" style="text-align: left;">
<div><br>
</div>
<div dir="ltr" style="text-align: left;">So our experience was much like you describe.</div>
<div dir="ltr" style="text-align: left;"><br>
</div>
<div dir="ltr" style="text-align: left;">Simon</div>
</div>
<div id="id-3625899c-7e84-432e-b82a-92d1bdb743e8" class="ms-outlook-mobile-reference-message">
<div style="font-family: sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg"><strong>From:</strong> <gpfsug-discuss-bounces@spectrumscale.org> on behalf of Ryan Novosielski <novosirj@rutgers.edu><br>
<strong>Sent:</strong> Friday, 11 October 2019, 18:43<br>
<strong>To:</strong> gpfsug main discussion list<br>
<strong>Subject:</strong> [gpfsug-discuss] Quotas and AFM<br>
</div>
<br>
Does anyone have any good resources or experience with quotas and AFM caches? Our scenario is that we have an AFM home one one site, an AFM cache on another site, and then a client cluster on that remote site that mounts the cache. The AFM filesets are IW.
One of them contains our home directories, which have a quota set on the home side. Quotas were disabled entirely on the cache side (I enabled them recently, but did not set them to anything). What I believe we’re running into is scary long AFM queues that
are caused by people writing an amount that is over the home quota to the cache, but the cache is accepting it and then failing to sync back to the home because the user is at their hard limit. I believe we’re also seeing delays on unaffected users who are
not over their quota, but that’s harder to tell. We have the AFM gateways poorly/not tuned, so that is likely interacting. Is there any way to make the quotas apparent to the cache cluster too, beyond setting a quota there as well, or do I just fundamentally
misunderstand this in some other way? We really just want the quotas on the home cluster to be enforced everywhere, more or less. Thanks! -- ____ || \\UTGERS, |---------------------------*O*--------------------------- ||_// the State | Ryan Novosielski - novosirj@rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus || \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark `' _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss <br>
</div>
</div>
<span>_______________________________________________</span><br>
<span>gpfsug-discuss mailing list</span><br>
<span>gpfsug-discuss at spectrumscale.org</span><br>
<span>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span><br>
</div>
</blockquote>
</div>
</body>
</html>