[gpfsug-discuss] mmbackup feature request

Wed Sep 6 15:45:26 BST 2023

We use independent filesets so we can snapshot individually, and then a
separate TSM node for each fileset. Finally we have a "canary" directory
that's local to the node running the schedule that dsmc can backup as a
virtual mount point that we use to track backup success/failure, since
mmbackup-driven backups won't update the last backup time of the GPFS
filesystem.

All of the configuration is Puppet-managed, which sets up a separate dsmcad
service for each fileset that connects to the TSM server using the
fileset-specific node, which is associated with a schedule that runs the
backup. Relevant details from one of our schedules:

Node Name: GPFS-GS2-GS-VOL5
    Schedule Name: DAILY_GPFS-GS2-GS-VOL5
   Schedule Style: Classic
           Action: Incremental
          Options: -presched='/usr/local/sbin/backup-gpfs -s gpfs-gs2-gs-vol5' -dom=/mnt/canary.nobackup/gpfs-gs2-gs-vol5
          Objects: 
         Priority: 5
   Next Execution: 13 Hours
         Duration: 5 Minutes 
           Period: 1 Day  
      Day of Week: Any

The backup-gpfs script sets up the snapshot, backs up some metadata about
the filesets so we can recover configuration easily (size, inode limit,
junction, etc.), runs mmbackup, and will exit with success/failure
depending on what the underlying commands return. It also logs mmbackup
output to Splunk which makes log trawling a lot easier.

If it runs into a problem, a trap removes the snapshot so we can run the
backup again without manual intervention. If the
/mnt/canary.nobackup/gpfs-gs2-gs-vol5 filespace associated with the
GPFS-GS2-GS-VOL5 node gets more than a couple days behind, we'll notice
that with our monitoring of the filespace last backup time (same as our
non-GPFS filespaces) and we can go in and take a look.

We use this methodology to backup five entire GPFS filesystems for our
larger labs, along with a sixth that's divided into 76 filesets as a
storage condo for our smaller labs. There's some quirks but scales better
than trying to run mmbackup independently for everything.

Hope that helps!

On Wed, Sep 06, 2023 at 11:44:56AM +0200, Christian Petersson wrote:
> Just a follow up question, how do you backup multiple filesets?
> We have a 50 filesets to backup, at the moment do we have a text file that
> contains all of them and we run a for loop. But that is not at all
> scalable.
> 
> Is it any other ways that are much better?
> 
> /Christian
> 
> ons 6 sep. 2023 kl. 11:35 skrev Marcus Koenig <marcus at koenighome.de>:
> 
> > I'm using this one liner to get the progress
> >
> > grep 'mmbackup:Backup job finished'|cut -d ":" -f 6|awk '{print $1}'|awk
> > '{s+=$1}END{print s}'
> >
> > That can be compared to the files identified during the scan.
> >
> > On Wed, 6 Sept 2023, 21:29 Stephan Graf, <st.graf at fz-juelich.de> wrote:
> >
> >> Hi
> >>
> >> I think it should be possible because mmbackup know, how many files are
> >> to be backed up, which have been already processed and how many are
> >> still to go.
> >>
> >> BTW it would also be nice to have an option in mmbackup to generate
> >> machine readable log file like JSON or CSV.
> >>
> >> But the right way to ask for a new feature or to look if there is
> >> already a request open is the IBM IDEA portal:
> >>
> >> https://urldefense.com/v3/__https://ideas.ibm.com__;!!K-Hz7m0Vt54!mV9vXvf6GYeaY4hHi834eHy2L_41MW2qtO-ZhGwdc1U5YqN7WAiEI6GB6IH2aXbcUw_gfkBdDK9jsSAfPU3tTuUNUQmn$ 
> >>
> >> Stephan
> >>
> >> On 9/6/23 11:02, Jonathan Buzzard wrote:
> >> >
> >> > Would it be possible to have the mmbackup output display the percentage
> >> > output progress when backing up files?
> >> >
> >> > So at the top we you see something like this
> >> >
> >> > Tue Sep  5 23:13:35 2023 mmbackup:changed=747204, expired=427702,
> >> > unsupported=0 for server [XXXX]
> >> >
> >> > Then after it does the expiration you see during the backup lines like
> >> >
> >> > Wed Sep  6 02:43:53 2023 mmbackup:Backing up files: 527024 backed up,
> >> > 426018 expired, 4408 failed. (Backup job exit with 4)
> >> >
> >> > It would IMHO be helpful if it looked like
> >> >
> >> > Wed Sep  6 02:43:53 2023 mmbackup:Backing up files: 527024 (70.5%)
> >> > backed up, 426018 (100%) expired, 4408 failed. (Backup job exit with 4)
> >> >
> >> > Just based on the number of files. Though as I look at it now I am
> >> > curious about the discrepancy in the number of files expired, given
> >> that
> >> > the expiration stage allegedly concluded with no errors?
> >> >
> >> > Tue Sep  5 23:21:49 2023 mmbackup:Completed policy expiry run with 0
> >> > policy errors, 0 files failed, 0 severe errors, returning rc=0.
> >> > Tue Sep  5 23:21:49 2023 mmbackup:Policy for expiry returned 0 Highest
> >> > TSM error 0
> >> >
> >> >
> >> >
> >> > JAB.
> >> >
> >>
> >> --
> >> Stephan Graf
> >> Juelich Supercomputing Centre
> >>
> >> Phone:  +49-2461-61-6578
> >> Fax:    +49-2461-61-6656
> >> E-mail: st.graf at fz-juelich.de
> >> WWW:    https://urldefense.com/v3/__http://www.fz-juelich.de/jsc/__;!!K-Hz7m0Vt54!mV9vXvf6GYeaY4hHi834eHy2L_41MW2qtO-ZhGwdc1U5YqN7WAiEI6GB6IH2aXbcUw_gfkBdDK9jsSAfPU3tTkQ48VFb$ 
> >>
> >> ---------------------------------------------------------------------------------------------
> >>
> >> ---------------------------------------------------------------------------------------------
> >> Forschungszentrum Juelich GmbH
> >> 52425 Juelich
> >> Sitz der Gesellschaft: Juelich
> >> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> >> Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
> >> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
> >> Karsten Beneke (stellv. Vorsitzender), Dr. Astrid Lambrecht,
> >> Prof. Dr. Frauke Melchior
> >>
> >> ---------------------------------------------------------------------------------------------
> >>
> >> ---------------------------------------------------------------------------------------------
> >> _______________________________________________
> >> gpfsug-discuss mailing list
> >> gpfsug-discuss at gpfsug.org
> >> https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org__;!!K-Hz7m0Vt54!mV9vXvf6GYeaY4hHi834eHy2L_41MW2qtO-ZhGwdc1U5YqN7WAiEI6GB6IH2aXbcUw_gfkBdDK9jsSAfPU3tTgsSVVvv$ 
> >>
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at gpfsug.org
> > https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org__;!!K-Hz7m0Vt54!mV9vXvf6GYeaY4hHi834eHy2L_41MW2qtO-ZhGwdc1U5YqN7WAiEI6GB6IH2aXbcUw_gfkBdDK9jsSAfPU3tTgsSVVvv$ 
> >

> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org__;!!K-Hz7m0Vt54!mV9vXvf6GYeaY4hHi834eHy2L_41MW2qtO-ZhGwdc1U5YqN7WAiEI6GB6IH2aXbcUw_gfkBdDK9jsSAfPU3tTgsSVVvv$ 

-- 
-- Skylar Thompson (skylar2 at u.washington.edu)
-- Genome Sciences Department (UW Medicine), System Administrator
-- Foege Building S046, (206)-685-7354
-- Pronouns: He/Him/His