<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
I've been hesitant to reply to this thread as I am not a TSM/Storage Protect expert even after dealing with it for more than a decade and a half, but we've been backing up some moderately sized GPFS file systems (8-16P) with it for many years now so I do have
some experience with it. We've had all the problems you can have from a corrupted DB2, to issues with a zillion tiny files taking more than 24 hours, to the once a week or so issues with users uploading/creating file with the dreaded "newline" or control
characters in the names. </div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
>If You run mmbackup on a large file system where a lot of changes are ongoing, we always see in the final report of mmbackup that some files failed.</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
As this thread started out asking about "validating" the backups let's start here and address a couple different issues from different folks.</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Are you not using snapshots to do the mmbackup? Other than 'bad characters' which GPFS allows and SP does not(GPFS is MUCH more permissive than TSM about that even with things like WILDCARDSARELITERAL/QUOTESARELITERAL), there shouldn't really be any issues.
I HIGHLY recommend using a snapshot at the minimum for backups. This will bypass any 'file not found' issues at least. Though this DOES introduce the whole 'issues with the snaps' you might see if compute nodes are too overloaded and will not pause for the
snaps to be created/deleted. </div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Snapshots will also allow the validation testing to have a perfect copy of the backed up data to validate with. Backups and snapshots are both 'point in time' on a live/production file system, so validation is difficult without being at that point in time.
Just keep the snapshot around as long as you can stand, for the tests. (I realize not everyone can keep snapshots for a long time due to inode/block issues) Maybe a scripted 'xxx#sum' for everything backed up? Up to you how you do this. </div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
For the comment about sending the data to multiple initial SP servers, that sounds like a major shadow file problem waiting to happen. Which server was it built with last? Which is the source of truth?, etc. Either send to a single SP server and have it
replicate (disaster recovery is your friend), and/or divide up the file system by filesets and send them to different servers. Using snapshots with multiple SP servers and gpfs filesets allows for much better parallelization of the backup itself. As we already
use independent filesets by 'project code' for quotas, this is quite helpful for us.</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Apologies if I used GPFS or TSM in here and folks were confused. Just replace those with Storage Scale and Storage Protect.</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Ed Wahl</div>
<div style="font-family: "Times New Roman", Times, serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Ohio Supercomputer Center</div>
<hr style="display: inline-block; width: 98%;">
<div id="divRplyFwdMsg">
<div style="direction: ltr; font-family: Calibri, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<b>From:</b> gpfsug-discuss <gpfsug-discuss-bounces@gpfsug.org> on behalf of Stephan Graf <st.graf@fz-juelich.de><br>
<b>Sent:</b> Friday, January 16, 2026 4:29 AM<br>
<b>To:</b> gpfsug-discuss@gpfsug.org <gpfsug-discuss@gpfsug.org><br>
<b>Subject:</b> Re: [gpfsug-discuss] mmbackup valiating the backups</div>
<div style="direction: ltr;"> </div>
</div>
<div style="font-size: 11pt;">Hi,<br>
<br>
I also recommend to use the '-q' option in a regularly way to be sure<br>
not to get some kind of 'split brain' situation: The TSM DB content and<br>
the shadow database.<br>
<br>
If You run mmbackup on a large file system where a lot of changes are<br>
ongoing, we always see in the final report of mmbackup that some files<br>
failed.<br>
As far as I remember the situation was, that the file was deleted by the<br>
user before the dsmc was started.<br>
But it is hard to check it for all files so we ignore those failures.<br>
But it could be that something different is the root cause of not<br>
backing up this file. In the past we saw it when users used strange<br>
characters in their file name.<br>
<br>
For this it would be nice to get some kind of better report from mmbackup.<br>
So for example a filelist of all files which failed. May be with the reason<br>
- file not found<br>
- dsmc backup failed<br>
- ...<br>
<br>
This is a question to the IBM Devs: What do You think? Is it worth of<br>
opening an IBM IDEA?<br>
<br>
Stephan<br>
<br>
<br>
On 1/16/26 07:53, Timm Stamer wrote:<br>
> Hi ,<br>
><br>
> we're running mmbackup with -q option once a quarter to be sure<br>
> everything is backed up.<br>
> I think this is much simpler than your approach but maybe does not<br>
> cover all your needs.<br>
><br>
><br>
> -q<br>
> Performs a query operation before issuing mmbackup. The IBM Storage<br>
> Protect server might have data stored already that is not recognized as<br>
> having been backed up by mmbackup and its own shadow database. To<br>
> properly compute the set of files that currently need to be backed up,<br>
> mmbackup can perform an IBM Storage Protect query and process the<br>
> results to update its shadow database. Use the -q switch to perform<br>
> this query and then immediately commence the requested backup<br>
> operation.<br>
><br>
> <a data-auth="NotApplicable" class="OWAAutoLink" id="OWAe20bf59c-8016-a982-6490-5c5d9d9de3b9" href="https://urldefense.com/v3/__https://www.ibm.com/docs/en/storage-scale/5.2.3?topic=reference-mmbackup-command__;!!KGKeukY!2Gc-96JLXSQApClIceidsjnn3c3yGp2w_RSrcXcylKv3vSYzH4zqmFX0o222MdyEb07sy4QfsZ9bKkKyvQNr$">
https://urldefense.com/v3/__https://www.ibm.com/docs/en/storage-scale/5.2.3?topic=reference-mmbackup-command__;!!KGKeukY!2Gc-96JLXSQApClIceidsjnn3c3yGp2w_RSrcXcylKv3vSYzH4zqmFX0o222MdyEb07sy4QfsZ9bKkKyvQNr$</a><br>
><br>
><br>
> [...]<br>
> we=$(LC_TIME=C date +%A)<br>
> dm=$(date +%d)<br>
> dmonth=$(date +%m)<br>
> checkMonths=("01" "04" "07" "10")<br>
> [[ ${checkMonths[@]} =~ $dmonth ]] && [ "$we" = "Friday" ] && [ "$dm" -le 7 ] && QUERYBACKUP="-q"<br>
><br>
> mmbackup ... ${QUERYBACKUP} ...<br>
> [...]<br>
><br>
><br>
> Kind regards<br>
><br>
> Timm Stamer<br>
><br>
><br>
><br>
><br>
> Am Donnerstag, dem 15.01.2026 um 18:21 +0000 schrieb Peter Childs:<br>
>><br>
>> Hi All,<br>
>><br>
>> We use mmbackup to backup our Scale Storage to two IBM Protect<br>
>> Instances.<br>
>><br>
>> We would like to validate our backups to ensure that the files we<br>
>> have really are backed up and we don't have any problems.<br>
>><br>
>> So far I've worked out I can query the backups using "query backup --<br>
>> detail" in Protect and get times and dates, From this I can compare<br>
>> with `stat` and `ls` (or using mmapplypolicy (which should be<br>
>> faster)) to check the three indexes match, and check the contents of<br>
>> each directory. (Using --subdir=yes looks great on paper but the<br>
>> output takes days to appear and checking it file by file can be done<br>
>> incrementally and I can use mmapplypolicy to run the report.<br>
>><br>
>> Given the stats from Protect the two backup servers are reporting<br>
>> different occupancy figures which suggests to me we may have some<br>
>> inconsistencies. (We're talking a 1/2 TB difference between the two<br>
>> servers according to Protect Query Occupancy) but I'm aware that<br>
>> Protect's figures are estimates and not always accurate.<br>
>><br>
>> Verify the backup is always a good plan even if your 101% sure its<br>
>> correct anyway (and I'm not maybe). (Your backups are only as good as<br>
>> the last time you recalled them and all that)<br>
>><br>
>> We could use the shadow database. As this looks to be what Storage<br>
>> Archive does to say yes this file is backed up before it is archived.<br>
>><br>
>> Does anyone know the format of the shadow database, which fields are<br>
>> which, as I think knowing the format might allow us to at least know<br>
>> what the differences; and increase our confidence in the backup.<br>
>><br>
>> Thanks in advance.<br>
>><br>
>> Peter Childs<br>
>> _______________________________________________<br>
>> gpfsug-discuss mailing list<br>
>> gpfsug-discuss at gpfsug.org<br>
>> <a data-auth="NotApplicable" class="OWAAutoLink" id="OWA2035307a-fba9-de16-3e4f-e42c1478c645" href="https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org__;!!KGKeukY!2Gc-96JLXSQApClIceidsjnn3c3yGp2w_RSrcXcylKv3vSYzH4zqmFX0o222MdyEb07sy4QfsZ9bKh7-Yo7f$">
https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org__;!!KGKeukY!2Gc-96JLXSQApClIceidsjnn3c3yGp2w_RSrcXcylKv3vSYzH4zqmFX0o222MdyEb07sy4QfsZ9bKh7-Yo7f$</a><br>
> _______________________________________________<br>
> gpfsug-discuss mailing list<br>
> gpfsug-discuss at gpfsug.org<br>
> <a data-auth="NotApplicable" class="OWAAutoLink" id="OWAedaa8283-1fda-a3d0-47d3-597bbde4deb6" href="https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org__;!!KGKeukY!2Gc-96JLXSQApClIceidsjnn3c3yGp2w_RSrcXcylKv3vSYzH4zqmFX0o222MdyEb07sy4QfsZ9bKh7-Yo7f$">
https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org__;!!KGKeukY!2Gc-96JLXSQApClIceidsjnn3c3yGp2w_RSrcXcylKv3vSYzH4zqmFX0o222MdyEb07sy4QfsZ9bKh7-Yo7f$</a><br>
<br>
--<br>
Stephan Graf<br>
#GernePerDu<br>
HPC, Cloud and Data Systems and Services<br>
Jlich Supercomputing Centre<br>
<br>
Phone: +49-2461-61-6578<br>
Fax: +49-2461-61-6656<br>
E-mail: st.graf@fz-juelich.de<br>
WWW: <a data-auth="NotApplicable" class="OWAAutoLink" id="OWA6b537056-213e-ca3e-23dc-e57e56858247" href="https://urldefense.com/v3/__http://www.fz-juelich.de/jsc/__;!!KGKeukY!2Gc-96JLXSQApClIceidsjnn3c3yGp2w_RSrcXcylKv3vSYzH4zqmFX0o222MdyEb07sy4QfsZ9bKkMvcWKi$">
https://urldefense.com/v3/__http://www.fz-juelich.de/jsc/__;!!KGKeukY!2Gc-96JLXSQApClIceidsjnn3c3yGp2w_RSrcXcylKv3vSYzH4zqmFX0o222MdyEb07sy4QfsZ9bKkMvcWKi$</a><br>
---------------------------------------------------------------------------------------------<br>
---------------------------------------------------------------------------------------------<br>
Forschungszentrum Jlich GmbH<br>
52425 Jlich<br>
Sitz der Gesellschaft: Jlich<br>
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498<br>
Vorsitzender des Aufsichtsrats: MinDir Stefan Mller<br>
Geschftsfhrung: Prof. Dr. Astrid Lambrecht (Vorsitzende),<br>
Dr. Stephanie Bauer (stellv. Vorsitzende)<br>
Prof. Dr. Ir. Pieter Jansens, Prof. Dr. Laurens Kuipers<br>
---------------------------------------------------------------------------------------------<br>
---------------------------------------------------------------------------------------------<br>
<br>
</div>
</body>
</html>