[gpfsug-discuss] gpfsug-discuss Digest, Vol 108, Issue 18

[Alec]

In our environment managing the GPFS policies (or anything GPFS) are a bit
of an Art form and really only a few people out of a sea of engineers
can/will do it.  So I like to keep the GPFS policies simple and durable.

If you really wanted to you could generate a list of every US (or whatever
country) holiday you wanted and use Python, Awk, whatever to produce an
exact number of business days versus not;

You can also use directory structure, GPFS attributes in parent directory,
or a config file to maintain the requirements, which may be more manageable
by other team members.

Anyhow my point being don't just look at the technical of how to implement,
but what is the most maintainable, .documentable, and recoverable solution.

Some things I would consider in my design?
* Supportability - What is the skill level of the engineer required to
maintain the system.
* Serviceability - What component would change most frequently, what's the
complexity and impact of a change.
* Fail Safe - If a component fails, what will the negative impact be.
* Trackability - Are all the activities logged.
* Sanity Checking - Can you define safety mechanisms to prevent a failure.
For instance if your purge script detects that there are 10k+ files to
purge, can you interrupt the process?

For GPFS we try to keep the service components simple, and the logic in
shell scripting where there is a plethora of knowledgeable engineers.  So
my attack would be like this:
  * Policy runs, purges any file with purgeafter beyond the current date
(maybe validate that the purge.date and modify/access date agree still).
<-- is simple
    * I would have it run the logic twice, and bomb if a safety threshhold
is exceeded.  Should log the results.
  * Config in /gpfs/configs/purge_configuration
       /some/path 10bd
       /some/scratchspace 30d
       /some/reallybigfiles 1d
   * mmfind $(awk '{print $1}' < gpfs/configs/purge_configuration) -type f
--exec /gpfs/scripts/setpurgedates
      * Could also be a regular mmapplypolicy command as well...
      * /gpfs/scripts/setpurgedates can rescan
/gpfs/configs/purge_configuration and read in the date, check against the
path passed, and then perform the necessary date logic, and exec the
mmchattr, preserve the file's modify/access time to a purge.date time..
         * Could log its decisions to a log file.

Happy Hunting, I've already learned quite a few things from monitoring this
list in a very short period of time.

* If a candidate ever tells me they know everything about Unix... I pass.
We're all very experienced noobies in this ever progressing
field/platform.  *

Alec

On Mon, Feb 1, 2021 at 1:58 PM Jonathan Buzzard <
jonathan.buzzard at strath.ac.uk> wrote:

> On 01/02/2021 21:09, Owen Morgan wrote:
> > CAUTION: This email originated outside the University. Check before
> > clicking links or attachments.
> > Jonathan,
> >
> > If I have a single policy file with all the related department rules and
> > each time they want to add additional rules with different working day
> > thresholds maybe using this -M method is easier. Its clear that the
> > 'maths' and date/timestamp manipulation is easier in shell (my preferred
> > is bash) than in the SQL of the policy (your example is succinct but
> > needs to be repeated everytime a new rule is added with a different
> > working day threshold, which is what I'm trying (if possiblr) to avoid.
> >
>
> I actually think there is a much better way to achieve it than the case
> statement, but that would have required me to do some more thinking and
> testing to make sure it worked ;-)
>
> Anyway how many different "working" days old do you need?
>
>
> JAB.
>
> --
> Jonathan A. Buzzard                         Tel: +44141-5483420
> HPC System Administrator, ARCHIE-WeSt.
> University of Strathclyde, John Anderson Building, Glasgow. G4 0NG
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210201/24dc3b68/attachment-0002.htm>