[gpfsug-discuss] /tmp/mmfs vanishes randomly?

Pavel Safre PSAFRE at de.ibm.com
Fri Nov 19 13:49:11 GMT 2021

Hello Heiner,

just a heads up for you and the other storage admins, regularly cleaning 
up /tmp, regarding one aspect to keep in mind:

- If you are using Spectrum Scale software call home (mmcallhome), it 
would be using the directory ${dataStructureDump}/callhome to save the 
copies of the uploaded data.
        This would be /tmp/mmfs/callhome/ in your case, which you would be 
automatically regularly removing.
- These copies are used by one of the features of call home: "mmcallhome 
status diff"
        - This feature allows to see an overview of the Spectrum Scale 
configuration changes, that occurred between 2 different points in time.
        - This effectively allows to quickly find out if any config 
changes occurred prior to an outage, thereby helping to find the root 
cause of self-caused problems in the Scale cluster.
        - It was added in Scale
        See IBM KC for more details: 

        - As a source of the "config snapshots", mmcallhome status diff is 
using the DC packages inside of ${dataStructureDump}/callhome, which you 
would be regularly deleting, thereby hugely reducing the usability of this 
particular feature.
- Of course, software call home automatically makes sure, it will not use 
too much space in dataStructureDump and it automatically removes the 
oldest entries, keeping at most 2GB or 300 files inside (default values, 

Mit freundlichen Grüßen / Kind regards

Pavel Safre

Software Engineer
IBM Systems Group, IBM Spectrum Scale Development
Dept. M925


 IBM Deutschland Research & Development GmbH

psafre at de.ibm.com
 Wilhelm-Fay-Straße 32

 65936 Frankfurt am Main

IBM Data Privacy Statement 
IBM Deutschland Research & Development GmbH / Vorsitzender des 
Aufsichtsrats: Gregor Pillen
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, 
HRB 243294 

From:   "Billich  Heinrich Rainer (ID SD)" <heinrich.billich at id.ethz.ch>
To:     "gpfsug main discussion list" <gpfsug-discuss at spectrumscale.org>
Date:   16.11.2021 17:44
Subject:        [EXTERNAL] Re: [gpfsug-discuss] /tmp/mmfs vanishes 
Sent by:        gpfsug-discuss-bounces at spectrumscale.org

Hello Olaf,
Thank you,  you are right. I was ignorant about the systemd-tmpfiles* 
services and timers. The cleanup in /tmp wasn’t present in RHEL7, at least 
not on our nodes. I consider to modify the configuration a bit to keep the 
directory  /tmp/mmfs  - or even create it – but to clean it’s content .
Best regards,
From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Olaf Weiser 
<olaf.weiser at de.ibm.com>
Reply to: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: Monday, 8 November 2021 at 10:53
To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Cc: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] /tmp/mmfs vanishes randomly?
Hallo Heiner,
multiple levels of answers..
(1st) ... it the directory is not there, the gpfs trace would create it 
automatically - just like this:
[root at ess5-ems1 ~]# ls -l /tmp/mmfs 
ls: cannot access '/tmp/mmfs': No such file or directory
[root at ess5-ems1 ~]# mmtracectl --start -N ems5k.mmfsd.net
mmchconfig: Command successfully completed
mmchconfig: Propagating the cluster configuration data to all
 affected nodes.  This is an asynchronous process.
[root at ess5-ems1 ~]# 
[root at ess5-ems1 ~]# 
[root at ess5-ems1 ~]# ls -l /tmp/mmfs 
total 0
-rw-r--r-- 1 root root 0 Nov  8 10:47 lxtrace.trcerr.ems5k
[root at ess5-ems1 ~]# 

(2nd) I think - the cleaning of /tmp is something done by the OS -
please check - 
systemctl status systemd-tmpfiles-setup.service
or look at this config file
[root at ess5-ems1 ~]# cat /usr/lib/tmpfiles.d/tmp.conf 
#  This file is part of systemd.
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published 
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

# See tmpfiles.d(5) for details

# Clear tmp directories separately, to make them easier to override
q /tmp 1777 root root 10d
q /var/tmp 1777 root root 30d

# Exclude namespace mountpoints created with PrivateTmp=yes
x /tmp/systemd-private-%b-*
X /tmp/systemd-private-%b-*/tmp
x /var/tmp/systemd-private-%b-*
X /var/tmp/systemd-private-%b-*/tmp

# Remove top-level private temporary directories on each boot
R! /tmp/systemd-private-*
R! /var/tmp/systemd-private-*
[root at ess5-ems1 ~]# 
hope this helps -
Mit freundlichen Grüßen / Kind regards
Olaf Weiser
IBM Systems, SpectrumScale Client Adoption
IBM Deutschland
IBM Allee 1
71139 Ehningen
Phone: +49-170-579-44-66
E-Mail: olaf.weiser at de.ibm.com
IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter
Geschäftsführung: Gregor Pillen (Vorsitzender), Agnes Heftberger, Norbert 
Janzen, Markus Koerner, Christian Noll, Nicole Reimer
Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, 
HRB 14562 / WEEE-Reg.-Nr. DE 99369940
----- Ursprüngliche Nachricht -----
Von: "Billich Heinrich Rainer (ID SD)" <heinrich.billich at id.ethz.ch>
Gesendet von: gpfsug-discuss-bounces at spectrumscale.org
An: "gpfsug main discussion list" <gpfsug-discuss at spectrumscale.org>
Betreff: [EXTERNAL] [gpfsug-discuss] /tmp/mmfs vanishes randomly?
Datum: Mo, 8. Nov 2021 10:35

We use /tmp/mmfs as dataStructureDump directory. Since a while I notice 
that this directory randomly vanishes. Mmhealth does not complain but just 
notes that it will no longer monitor the directory. Still I doubt that 
trace collection and similar will create the directory when needed?

Do you know of any spectrum scale internal mechanism that could cause 
/tmp/mmfs to get deleted? It happens on ESS nodes, with a plain IBM 
installation, too. It happens just on one or two nodes at a time, it's no 
cluster-wide cleanup or similar. We run scale 5.0.5 and ESS and

Thank you,

Mmhealth message:
local_fs_path_not_found   INFO       The configured dataStructureDump path 
/tmp/mmfs does not exists. Skipping monitoring.

Kind regards,

Heinrich Billich
ETH Zürich
Tel.: +41 44 632 72 56
heinrich.billich at id.ethz.ch

gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org

gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20211119/af0fd962/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 1851 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20211119/af0fd962/attachment-0002.gif>

More information about the gpfsug-discuss mailing list