[gpfsug-discuss] /tmp/mmfs vanishes randomly?
Pavel Safre
PSAFRE at de.ibm.com
Fri Nov 19 13:49:11 GMT 2021
Hello Heiner,
just a heads up for you and the other storage admins, regularly cleaning
up /tmp, regarding one aspect to keep in mind:
- If you are using Spectrum Scale software call home (mmcallhome), it
would be using the directory ${dataStructureDump}/callhome to save the
copies of the uploaded data.
This would be /tmp/mmfs/callhome/ in your case, which you would be
automatically regularly removing.
- These copies are used by one of the features of call home: "mmcallhome
status diff"
- This feature allows to see an overview of the Spectrum Scale
configuration changes, that occurred between 2 different points in time.
- This effectively allows to quickly find out if any config
changes occurred prior to an outage, thereby helping to find the root
cause of self-caused problems in the Scale cluster.
- It was added in Scale 5.0.5.0
See IBM KC for more details:
https://www.ibm.com/docs/en/spectrum-scale/5.1.0?topic=cch-use-cases-detecting-system-changes-by-using-mmcallhome-command
- As a source of the "config snapshots", mmcallhome status diff is
using the DC packages inside of ${dataStructureDump}/callhome, which you
would be regularly deleting, thereby hugely reducing the usability of this
particular feature.
- Of course, software call home automatically makes sure, it will not use
too much space in dataStructureDump and it automatically removes the
oldest entries, keeping at most 2GB or 300 files inside (default values,
configurable).
Mit freundlichen Grüßen / Kind regards
Pavel Safre
Software Engineer
IBM Systems Group, IBM Spectrum Scale Development
Dept. M925
Phone:
IBM Deutschland Research & Development GmbH
Email:
psafre at de.ibm.com
Wilhelm-Fay-Straße 32
65936 Frankfurt am Main
IBM Data Privacy Statement
IBM Deutschland Research & Development GmbH / Vorsitzender des
Aufsichtsrats: Gregor Pillen
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart,
HRB 243294
From: "Billich Heinrich Rainer (ID SD)" <heinrich.billich at id.ethz.ch>
To: "gpfsug main discussion list" <gpfsug-discuss at spectrumscale.org>
Date: 16.11.2021 17:44
Subject: [EXTERNAL] Re: [gpfsug-discuss] /tmp/mmfs vanishes
randomly?
Sent by: gpfsug-discuss-bounces at spectrumscale.org
Hello Olaf,
Thank you, you are right. I was ignorant about the systemd-tmpfiles*
services and timers. The cleanup in /tmp wasn’t present in RHEL7, at least
not on our nodes. I consider to modify the configuration a bit to keep the
directory /tmp/mmfs - or even create it – but to clean it’s content .
Best regards,
Heiner
From: <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Olaf Weiser
<olaf.weiser at de.ibm.com>
Reply to: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: Monday, 8 November 2021 at 10:53
To: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Cc: "gpfsug-discuss at spectrumscale.org" <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] /tmp/mmfs vanishes randomly?
Hallo Heiner,
multiple levels of answers..
(1st) ... it the directory is not there, the gpfs trace would create it
automatically - just like this:
[root at ess5-ems1 ~]# ls -l /tmp/mmfs
ls: cannot access '/tmp/mmfs': No such file or directory
[root at ess5-ems1 ~]# mmtracectl --start -N ems5k.mmfsd.net
mmchconfig: Command successfully completed
mmchconfig: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
[root at ess5-ems1 ~]#
[root at ess5-ems1 ~]#
[root at ess5-ems1 ~]# ls -l /tmp/mmfs
total 0
-rw-r--r-- 1 root root 0 Nov 8 10:47 lxtrace.trcerr.ems5k
[root at ess5-ems1 ~]#
(2nd) I think - the cleaning of /tmp is something done by the OS -
please check -
systemctl status systemd-tmpfiles-setup.service
or look at this config file
[root at ess5-ems1 ~]# cat /usr/lib/tmpfiles.d/tmp.conf
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published
by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
# See tmpfiles.d(5) for details
# Clear tmp directories separately, to make them easier to override
q /tmp 1777 root root 10d
q /var/tmp 1777 root root 30d
# Exclude namespace mountpoints created with PrivateTmp=yes
x /tmp/systemd-private-%b-*
X /tmp/systemd-private-%b-*/tmp
x /var/tmp/systemd-private-%b-*
X /var/tmp/systemd-private-%b-*/tmp
# Remove top-level private temporary directories on each boot
R! /tmp/systemd-private-*
R! /var/tmp/systemd-private-*
[root at ess5-ems1 ~]#
hope this helps -
cheers
Mit freundlichen Grüßen / Kind regards
Olaf Weiser
IBM Systems, SpectrumScale Client Adoption
-------------------------------------------------------------------------------------------------------------------------------------------
IBM Deutschland
IBM Allee 1
71139 Ehningen
Phone: +49-170-579-44-66
E-Mail: olaf.weiser at de.ibm.com
-------------------------------------------------------------------------------------------------------------------------------------------
IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter
Geschäftsführung: Gregor Pillen (Vorsitzender), Agnes Heftberger, Norbert
Janzen, Markus Koerner, Christian Noll, Nicole Reimer
Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart,
HRB 14562 / WEEE-Reg.-Nr. DE 99369940
----- Ursprüngliche Nachricht -----
Von: "Billich Heinrich Rainer (ID SD)" <heinrich.billich at id.ethz.ch>
Gesendet von: gpfsug-discuss-bounces at spectrumscale.org
An: "gpfsug main discussion list" <gpfsug-discuss at spectrumscale.org>
CC:
Betreff: [EXTERNAL] [gpfsug-discuss] /tmp/mmfs vanishes randomly?
Datum: Mo, 8. Nov 2021 10:35
Hello,
We use /tmp/mmfs as dataStructureDump directory. Since a while I notice
that this directory randomly vanishes. Mmhealth does not complain but just
notes that it will no longer monitor the directory. Still I doubt that
trace collection and similar will create the directory when needed?
Do you know of any spectrum scale internal mechanism that could cause
/tmp/mmfs to get deleted? It happens on ESS nodes, with a plain IBM
installation, too. It happens just on one or two nodes at a time, it's no
cluster-wide cleanup or similar. We run scale 5.0.5 and ESS 6.0.2.2 and
6.0.2.2.
Thank you,
Mmhealth message:
local_fs_path_not_found INFO The configured dataStructureDump path
/tmp/mmfs does not exists. Skipping monitoring.
Kind regards,
Heiner
---
=======================
Heinrich Billich
ETH Zürich
Informatikdienste
Tel.: +41 44 632 72 56
heinrich.billich at id.ethz.ch
========================
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20211119/af0fd962/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 1851 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20211119/af0fd962/attachment-0002.gif>
More information about the gpfsug-discuss
mailing list