[gpfsug-discuss] gpfsug-discuss Digest, Vol 51, Issue 9
Dominic Mueller-Wicke01
dominic.mueller at de.ibm.com
Mon Apr 11 16:26:45 BST 2016
Spectrum Protect backup (under the hood of mmbackup) and Spectrum Protect
for Space Management (HSM) can be combined on the same data. There are some
valuable integration topics between the products that can reduce the
overall network traffic if using backup and HSM on the same files. With the
combination of the products you have the ability to free file system space
from cold data and migrate them out to tape and to have several versions of
frequently used files in backup in the same file system.
Greetings, Dominic.
______________________________________________________________________________________________________________
Dominic Mueller-Wicke | IBM Spectrum Protect Development | Technical Lead |
+49 7034 64 32794 | dominic.mueller at de.ibm.com
Vorsitzende des Aufsichtsrats: Martina Koederitz; Geschäftsführung: Dirk
Wittkopp
Sitz der Gesellschaft: Böblingen; Registergericht: Amtsgericht Stuttgart,
HRB 243294
From: gpfsug-discuss-request at spectrumscale.org
To: gpfsug-discuss at spectrumscale.org
Date: 11.04.2016 17:11
Subject: gpfsug-discuss Digest, Vol 51, Issue 9
Sent by: gpfsug-discuss-bounces at spectrumscale.org
Send gpfsug-discuss mailing list submissions to
gpfsug-discuss at spectrumscale.org
To subscribe or unsubscribe via the World Wide Web, visit
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
or, via email, send a message with subject or body 'help' to
gpfsug-discuss-request at spectrumscale.org
You can reach the person managing the list at
gpfsug-discuss-owner at spectrumscale.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of gpfsug-discuss digest..."
Today's Topics:
1. backup and disaster recovery solutions (Damir Krstic)
2. Re: backup and disaster recovery solutions (Jaime Pinto)
3. Re: backup and disaster recovery solutions (Jonathan Buzzard)
4. Re: backup and disaster recovery solutions (Marc A Kaplan)
----- Message from Damir Krstic <damir.krstic at gmail.com> on Mon, 11 Apr
2016 13:15:30 +0000 -----
To: gpfsug main discussion list
<gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] backup and disaster recovery
solutions
We have implemented 1.5PB ESS solution recently in our HPC environment.
Today we are kicking of backup and disaster recovery discussions so I was
wondering what everyone else is using for their backup?
In our old storage environment we simply rsync-ed home and software
directories and projects were not backed up.
With ESS we are looking for more of a GPFS based backup solution -
something to tape possibly and also something that will have life cycle
feature - so if the file is not touched for number of days, it's moved to a
tape (something like LTFS).
Thanks in advance.
DAmir
----- Message from Jaime Pinto <pinto at scinet.utoronto.ca> on Mon, 11 Apr
2016 10:34:54 -0400 -----
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>, Damir Krstic
<damir.krstic at gmail.com>
Subject: Re: [gpfsug-discuss] backup and disaster recovery solutions
Do you want backups or periodic frozen snapshots of the file system?
Backups can entail some level of version control, so that you or
end-users can get files back on certain points in time, in case of
accidental deletions. Besides 1.5PB is a lot of material, so you may
not want to take full snapshots that often. In that case, a
combination of daily incremental backups using TSM with GPFS's
mmbackup can be a good option. TSM also does a very good job at
controlling how material is distributed across multiple tapes, and
that is something that requires a lot of micro-management if you want
a home grown solution of rsync+LTFS.
On the other hand, you could use gpfs built-in tools such a
mmapplypolicy to identify candidates for incremental backup, and send
them to LTFS. Just more micro management, and you may have to come up
with your own tool to let end-users restore their stuff, or you'll
have to act on their behalf.
Jaime
Quoting Damir Krstic <damir.krstic at gmail.com>:
> We have implemented 1.5PB ESS solution recently in our HPC environment.
> Today we are kicking of backup and disaster recovery discussions so I was
> wondering what everyone else is using for their backup?
>
> In our old storage environment we simply rsync-ed home and software
> directories and projects were not backed up.
>
> With ESS we are looking for more of a GPFS based backup solution -
> something to tape possibly and also something that will have life cycle
> feature - so if the file is not touched for number of days, it's moved to
a
> tape (something like LTFS).
>
> Thanks in advance.
>
> DAmir
>
************************************
TELL US ABOUT YOUR SUCCESS STORIES
http://www.scinethpc.ca/testimonials
************************************
---
Jaime Pinto
SciNet HPC Consortium - Compute/Calcul Canada
www.scinet.utoronto.ca - www.computecanada.org
University of Toronto
256 McCaul Street, Room 235
Toronto, ON, M5T1W5
P: 416-978-2755
C: 416-505-1477
----------------------------------------------------------------
This message was sent using IMP at SciNet Consortium, University of
Toronto.
----- Message from Jonathan Buzzard <jonathan at buzzard.me.uk> on Mon, 11 Apr
2016 16:02:45 +0100 -----
To: gpfsug-discuss at spectrumscale.org
Subject: Re: [gpfsug-discuss] backup and disaster
recovery solutions
On Mon, 2016-04-11 at 10:34 -0400, Jaime Pinto wrote:
> Do you want backups or periodic frozen snapshots of the file system?
>
> Backups can entail some level of version control, so that you or
> end-users can get files back on certain points in time, in case of
> accidental deletions. Besides 1.5PB is a lot of material, so you may
> not want to take full snapshots that often. In that case, a
> combination of daily incremental backups using TSM with GPFS's
> mmbackup can be a good option. TSM also does a very good job at
> controlling how material is distributed across multiple tapes, and
> that is something that requires a lot of micro-management if you want
> a home grown solution of rsync+LTFS.
Is there any other viable option other than TSM for backing up 1.5PB of
data? All other backup software does not handle this at all well.
> On the other hand, you could use gpfs built-in tools such a
> mmapplypolicy to identify candidates for incremental backup, and send
> them to LTFS. Just more micro management, and you may have to come up
> with your own tool to let end-users restore their stuff, or you'll
> have to act on their behalf.
>
I was not aware of a way of letting end users restore their stuff from
*backup* for any of the major backup software while respecting the file
system level security of the original file system. If you let the end
user have access to the backup they can restore any file to any location
which is generally not a good idea.
I do have a concept of creating a read only Fuse mounted file system
from a TSM point in time synthetic backup, and then using the shadow
copy feature of Samba to enable restores using the "Previous Versions"
feature of windows file manager.
I got as far as getting a directory tree you could browse through but
then had an enforced change of jobs and don't have access to a TSM
server any more to continue development.
Note if anyone from IBM is listening that would be a super cool feature.
JAB.
--
Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.
----- Message from "Marc A Kaplan" <makaplan at us.ibm.com> on Mon, 11 Apr
2016 11:11:24 -0400 -----
To: gpfsug main discussion list
<gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] backup and disaster recovery
solutions
Since you write "so if the file is not touched for number of days, it's
moved to a tape" -
that is what we call the HSM feature. This is additional function beyond
backup. IBM has two implementations.
(1) TSM/HSM now called IBM Spectrum Protect.
http://www-03.ibm.com/software/products/en/spectrum-protect-for-space-management
(2) HPSS http://www.hpss-collaboration.org/
The GPFS (Spectrum Scale File System) policy feature supports both, so that
mmapplypolicy and GPFS policy rules can be used to perform
accelerated metadata scans to identify which files should be migrated.
Also, GPFS supports on-demand recall (on application reads) of data from
long term storage (tape) to GPFS storage (disk or SSD). See also DMAPI.
Marc A Kaplan
From: Damir Krstic <damir.krstic at gmail.com>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: 04/11/2016 09:16 AM
Subject: [gpfsug-discuss] backup and disaster recovery solutions
Sent by: gpfsug-discuss-bounces at spectrumscale.org
We have implemented 1.5PB ESS solution recently in our HPC environment.
Today we are kicking of backup and disaster recovery discussions so I was
wondering what everyone else is using for their backup?
In our old storage environment we simply rsync-ed home and software
directories and projects were not backed up.
With ESS we are looking for more of a GPFS based backup solution -
something to tape possibly and also something that will have life cycle
feature - so if the file is not touched for number of days, it's moved to a
tape (something like LTFS).
Thanks in advance.
DAmir _______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160411/d2b7b64d/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160411/d2b7b64d/attachment-0002.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0E436792.gif
Type: image/gif
Size: 21994 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20160411/d2b7b64d/attachment-0003.gif>
More information about the gpfsug-discuss
mailing list