<font size=2 face="sans-serif">Hi,</font><br><br><font size=2 face="sans-serif">The major change around 4.X in quotas
was the introduction of dynamic shares. In the past, every client share
request was for constant number of blocks ( 20 blocks by default). For
high performing system, it wasn't enough sometime ( imagine 320M for nodes
are writing at 20GB/s). So, dynamic shares means that a client node can
request 10000 blocks etc. etc. ( it doesn't mean that the server will provide
those...).</font><br><font size=2 face="sans-serif">OTOH, node failure will leave more "stale
in doubt" capacity since the server don't know how much of the share
was actually used.</font><br><br><font size=2 face="sans-serif">Imagine a client node getting 1024 blocks
( 16G), using 20M and crashing. From the server perspective, there are
16G "unknown", now multiple that by multiple nodes...</font><br><font size=2 face="sans-serif">The only way to solve it is indeed to
execute mmcheckquota - but as you probably know, its not cheap.</font><br><br><font size=2 face="sans-serif">So, do you experience large number of
node expels/crashes etc. that might be related to that ( otherwise, it
might be some other bug that needs to be fixed...). </font><br><font size=2 face="sans-serif"><br>Regards,<br><br>Tomer Perry<br>Scalable I/O Development (Spectrum Scale)<br>email: tomp@il.ibm.com<br>1 Azrieli Center, Tel Aviv 67021, Israel<br>Global Tel: +1 720 3422758<br>Israel Tel: +972 3 9188625<br>Mobile: +972 52 2554625<br></font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Jaime Pinto <pinto@scinet.utoronto.ca></font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug-discuss@spectrumscale.org</font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">07/10/2019 17:40</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">[EXTERNAL] Re:
[gpfsug-discuss] Large in doubt on fileset</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font><br><hr noshade><br><br><br><tt><font size=2>We run DSS as well, also 4.2.x versions, and large
indoubt entries are common on our file systems, much larger than what you
are seeing, for USR, GRP <br>and FILESET.<br><br>It didn't use to be so bad on versions 3.4|3.5 in other IBM appliances
(GSS, ESS), even DDN's or Cray G200. Under 4.x series the internal automatic
<br>mechanism to reconcile accounting seems very laggy by default, and I couldn't
find (yet) a config parameter to adjust this. I stopped trying to <br>understand why this happens.<br><br>Our users are all subject to quotas, and can't wait indefinitely for this
reconciliation. I just run mmcheckquota every 6 hours via a crontab.<br><br>I hope version 5 is better. Will know in a couple of months.<br>Jaime<br><br><br><br>On 2019-10-07 10:07 a.m., Jonathan Buzzard wrote:<br>> <br>> I have a DSS-G system running 4.2.3-7, and on Friday afternoon became<br>> aware that there is a very large (at least I have never seen anything<br>> on this scale before) in doubt on a fileset. It has persisted over
the<br>> weekend and is sitting at 17.5TB, with the fileset having a 150TB
quota<br>> and only 82TB in use.<br>> <br>> There is a relatively large 26,500 files in doubt, though there is
no<br>> quotas on file numbers for the fileset. This has come down from some<br>> 47,500 on Friday when the in doubt was a shade over 18TB.<br>> <br>> The largest in doubt I have seen in the past was in the order of a
few<br>> hundred GB under very heavy write that went away very quickly after
the<br>> writing stopped.<br>> <br>> There is no evidence of heavy writing going on in the file system
so I<br>> am perplexed as to why the in doubt is remaining so high.<br>> <br>> Any thoughts as to what might be going on?<br>> <br>> <br>> JAB.<br>> <br><br><br><br> ************************************<br> TELL US ABOUT YOUR SUCCESS STORIES<br> </font></tt><a href="http://www.scinethpc.ca/testimonials"><tt><font size=2>http://www.scinethpc.ca/testimonials</font></tt></a><tt><font size=2><br> ************************************<br>---<br>Jaime Pinto - Storage Analyst<br>SciNet HPC Consortium - Compute/Calcul Canada<br></font></tt><a href="www.scinet.utoronto.ca"><tt><font size=2>www.scinet.utoronto.ca</font></tt></a><tt><font size=2>- </font></tt><a href="www.computecanada.ca"><tt><font size=2>www.computecanada.ca</font></tt></a><tt><font size=2><br>University of Toronto<br>661 University Ave. (MaRS), Suite 1140<br>Toronto, ON, M5G1M1<br>P: 416-978-2755<br>C: 416-505-1477<br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br><br></font></tt><br><br><BR>