[gpfsug-discuss] Handling bad file names in policies?

Wahl, Edward ewahl at osc.edu
Fri Oct 8 21:44:14 BST 2021

This is an interesting idea, but not at all what I was working towards, and is getting me off track.  (and I'm known to get distracted and explore interesting Rabbit Holes, red herrings, et al)   I've next to no issues with the filenames in day to day operations.

On the positive side, this is a one off.   What I need is a LIST policy, and the return leaves off the entire filename.

Ed Wahl

From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Alec <anacreo at gmail.com>
Sent: Friday, October 8, 2021 3:36 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Handling bad file names in policies?

Why not just configure a file placement policy using a non existent pool or a bad encryption key to prevent files with non-printables characters from even being created in the first place.


On Fri, Oct 8, 2021, 11:49 AM Wahl, Edward <ewahl at osc.edu<mailto:ewahl at osc.edu>> wrote:
This goes back as far as I can recall to <=GPFS 3.5 days. And no, I cannot recall what version of TSM-EE that was.   But newline has been the only stopping point, for what seems like forever.
Having filed many an mmbackup bug, I don't recall ever crashing on filenames.  (tons of OTHER reasons, but not character set)   We even generate an error report from this and email users to fix it.
We accept basically almost everything else, and I have to say, we see some really crazy things sometimes.   I think my current favorite is the full windows paths as a filename.
(eg:  "Y:\Temp\temp\290\work\0\Material_ERTi-5.in" )

Current IBM documentation doesn't go backwards past 4.2 but it says:

"For IBM Spectrum Scale™ file systems with special characters frequently used in the names of files or directories, backup failures might occur. Known special characters that require special handling include: *, ?, ", ’, carriage return, and the new line character.

In such cases, enable the Tivoli Storage Manager client options WILDCARDSARELITERAL and QUOTESARELITERAL on all nodes that are used in backup activities and make sure that the mmbackup option --noquote is used when invoking mmbackup."

So maybe we could handle newlines somehow.   But my lazy searches didn't show what TSM doesn't accept.

Ed Wahl

-----Original Message-----
From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> On Behalf Of Jonathan Buzzard
Sent: Monday, October 4, 2021 7:29 PM
To: gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Handling bad file names in policies?

On 04/10/2021 23:23, Wahl, Edward wrote:

> I know I've run into this before way back, but my notes on how I
> solved this aren't getting the job done in Scale and my notes
> are from 3.5.  😉
> Anyone know a way to get a LIST policy to properly feed bad filenames
> into the output or an external script?
> When I say bad I mean things like control characters, spaces, etc.
> Not concerned about the dreaded 'newline' as we force users to fix
> those or the files do not get backed up in Tivoli.

Since when? Last time I checked which was admittedly circa 2008, TSM would backup files with newlines in them no problem. mmbackup on the other hand in that time frame would simply die and backup nothing if there was a single file on the file system with a newline in it.

I would take a look at the mmbackup scripts which can handle such stuff (least ways in >4.2) which would also suggest dsmc can handle it.

As an aside I now think I know how you end up with newlines in file names. Basically you cut and paste the file name complete with newlines (most likely at the end) into a text field when saving the file.
Personally I think any program should baulk at that point but what do I know.


Jonathan A. Buzzard                         Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG _______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://urldefense.com/v3/__http://spectrumscale.org__;!!KGKeukY!jPEGEXlh4N27v0ev7VeN2w8CsqZiWAWqEtQpQ7eHaetmvPuD0-JVJrZx0hAA$>
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://urldefense.com/v3/__http://spectrumscale.org__;!!KGKeukY!jPEGEXlh4N27v0ev7VeN2w8CsqZiWAWqEtQpQ7eHaetmvPuD0-JVJrZx0hAA$>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20211008/f12eb7e8/attachment-0002.htm>

More information about the gpfsug-discuss mailing list