[gpfsug-discuss] Fwd: FW: Backing up GPFS with Rsync
Steven Daniels
sadaniel at us.ibm.com
Thu Mar 11 16:08:11 GMT 2021
Also, be aware there have been massive improvements in AFM, in terms of
usability, reliablity and performance.
I just completed a project where we moved about 3/4 PB during 7x24
operations to retire a very old storage system (1st Gen IBM GSS) to a new
ESS. We were able to get considerable performance but not without effort,
it allowed the client to continue operations and migrate to new hardware
seamlessly.
The new v5.1 AFM feature supports filesystem level AFM which would have
greatly simplified the effort and I believe will make AFM vastly easier to
implement in the general case.
I'll leave it to Venkat and others on the development team to share more
details about improvements.
Steven A. Daniels
Cross-brand Client Architect
Senior Certified IT Specialist
National Programs
Fax and Voice: 3038101229
sadaniel at us.ibm.com
http://www.ibm.com
From: Stephen Ulmer <ulmer at ulmer.org>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Cc: bill.burke.860 at gmail.com
Date: 03/11/2021 06:47 AM
Subject: [EXTERNAL] Re: [gpfsug-discuss] Fwd: FW: Backing up GPFS with
Rsync
Sent by: gpfsug-discuss-bounces at spectrumscale.org
Thank you! Would you mind letting me know in what era you made your
evaluation?
I’m not suggesting you should change anything at all, but when I make
recommendations for my own customers I like to be able to associate the
level of GPFS with the anecdotes. I view the software as more of a stream
of features and capabilities than as a set product.
Different clients have different requirements, so every implementation
could be different. When I add someone else’s judgement to my own, I just
like getting as close to their actual evaluation scenario as possible.
Your original post was very thoughtful, and I appreciate your time.
--
Stephen
On Mar 11, 2021, at 7:58 AM, Tagliavini, Enrico
<enrico.tagliavini at fmi.ch> wrote:
Hello Stephen,
actually not a dumb question at all. We evaluated AFM quite a bit
before turning it down.
The horror stories about it and massive data loss are too scary. Plus
we had actual reports of very bad performance. Personally I think AFM
is very complicated, overcomplicated for what we need. We need the
data safe, we don't need active / active DR or anything like that.
While AFM can technically do what we need the complexity of its
design makes it too easy to make a mistake and cause a service
disruption or, even worst, data loss. We are a very small institute
with a small IT team, so investing time in making it right was also
not really worth it due to the high TCO.
Kind regards.
--
Enrico Tagliavini
Systems / Software Engineer
enrico.tagliavini at fmi.ch
Friedrich Miescher Institute for Biomedical Research
Infomatics
Maulbeerstrasse 66
4058 Basel
Switzerland
On Thu, 2021-03-11 at 08:17 -0500, Stephen Ulmer wrote:
I’m going to ask what may be a dumb question:
Given that you have GPFS on both ends, what made you decide to NOT
use AFM?
--
Stephen
On Mar 11, 2021, at 3:56 AM, Tagliavini, Enrico
<enrico.tagliavini at fmi.ch> wrote:
Hello William,
I've got your email forwarded my another user and I decided to
subscribe to give you my two cents.
I would like to warn you about the risk of dong what you have in
mind. Using the GPFS policy engine to get a list of file to rsync
is
easily going to get you with missing data in the backup. The
problem is that there are cases that are not covered by it. For
example
if you mv a folder with a lot of nested subfolders and files none
of the subfolders would show up in your list of files to be
updated.
DM API would be the way to go, as you could replicate the mv on
the backup side, but you must not miss any event, which scares me
enough not to go that route.
What I ended up doing instead: we run GPFS on both side, main and
backup storage. So I use the policy engine on both sides and just
build up the differences. We have about 250 million files and this
is surprisingly fast. On top of that add all the files for which
the ctime changes in the last couple of days (to update metadata
info).
Good luck.
Kind regards.
--
Enrico Tagliavini
Systems / Software Engineer
enrico.tagliavini at fmi.ch
Friedrich Miescher Institute for Biomedical Research
Infomatics
Maulbeerstrasse 66
4058 Basel
Switzerland
-------- Forwarded Message --------
-----Original Message-----
From: gpfsug-discuss-bounces at spectrumscale.org
<gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of Ryan
Novosielski
Sent: Wednesday, March 10, 2021 3:22 AM
To: gpfsug main discussion list
<gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Backing up GPFS with Rsync
Yup, you want to use the policy engine:
https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adv.doc/bl1adv_policyrules.htm
Something in here ought to help. We do something like this (but
I’m reluctant to provide examples as I’m actually suspicious
that we
don’t have it quite right and are passing far too much stuff to
rsync).
--
#BlackLivesMatter
____
\\UTGERS,
|---------------------------*O*---------------------------
_// the State | Ryan Novosielski -
novosirj at rutgers.edu
\\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~
RBHS Campus
\\ of NJ | Office of Advanced Research Computing - MSB
C630, Newark
`'
On Mar 9, 2021, at 9:19 PM, William Burke
<bill.burke.860 at gmail.com> wrote:
I would like to know what files were modified/created/deleted
(only for the current day) on the GPFS's file system so that I
could rsync ONLY those files to a predetermined external
location. I am running GPFS 4.2.3.9
Is there a way to access the GPFS's metadata directly so that I
do not have to traverse the filesystem looking for these files?
If
i use the rsync tool it will scan the file system which is 400+
million files. Obviously this will be problematic to complete
a
scan in a day, if it would ever complete single-threaded. There
are tools or scripts that run multithreaded rsync but it's
still a
brute force attempt. and it would be nice to know where the
delta of files that have changed.
I began looking at Spectrum Scale Data Management (DM) API but
I am not sure if this is the best approach to looking at the
GPFS
metadata - inodes, modify times, creation times, etc.
--
Best Regards,
William Burke (he/him)
Lead HPC Engineer
Advance Research Computing
860.255.8832 m | LinkedIn
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=hSVvvIGpqQhKt_u_TKHdjoXyU-z7P14pCBQ5pA7MMFA&s=g2hkl0Raj7QbLvqRZfDk6nska0crl4Peh4kd8YwiO6k&e=
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210311/c7c77d22/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 1A816397.jpg
Type: image/jpeg
Size: 4919 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210311/c7c77d22/attachment-0002.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210311/c7c77d22/attachment-0002.gif>
More information about the gpfsug-discuss
mailing list