<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:12pt" ><div dir="ltr" >Have you looked a the mmaddcallback command and specifically the file system mount callbacks?</div>
<div dir="ltr" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div dir="ltr" ><br><font size="2" face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" ><span style="font-size:1.143em;" >Fred<br>__________________________________________________<br>Fred Stock | IBM Pittsburgh Lab | 720-430-8821<br>stockf@us.ibm.com</span></font></div></div></div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Original message -----<br>From: Ulrich Sibiller <u.sibiller@science-computing.de><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: gpfsug-discuss@gpfsug.org<br>Cc:<br>Subject: [EXTERNAL] [gpfsug-discuss] wait for mount during gpfs startup<br>Date: Tue, Apr 28, 2020 7:05 AM<br>
<div><font size="2" face="Default Monospace,Courier New,Courier,monospace" >Hi,<br><br>when the gpfs systemd service returns from startup the filesystems are usually not mounted. So<br>having another service depending on gpfs is not feasible if you require the filesystem(s).<br><br>Therefore we have added a script to the systemd gpfs service that waits for all local gpfs<br>filesystems being mounted. We have added that script via ExecStartPost:<br><br>------------------------------------------------------------<br># cat /etc/systemd/system/gpfs.service.d/waitmount.conf<br>[Service]<br>ExecStartPost=/usr/local/sc-gpfs/sbin/wait-for-all_local-mounts.sh<br>TimeoutStartSec=200<br>-------------------------------------------------------------<br><br>The script itself is not doing much:<br>-------------------------------------------------------------<br>#!/bin/bash<br>#<br># wait until all _local_ gpfs filesystems are mounted. It ignored<br># filesystems where mmlsfs -A does not report "yes".<br>#<br># returns 0 if all fs are mounted (or none are found in gpfs configuration)<br># returns non-0 otherwise<br><br># wait for max. TIMEOUT seconds<br>TIMEOUT=180<br><br># leading space is required!<br>FS=" $(/usr/lpp/mmfs/bin/mmlsfs all_local -Y 2>/dev/null | grep :automaticMountOption:yes: | cut -d:<br>-f7 | xargs; exit ${PIPESTATUS[0]})"<br># RC=1 and no output means there are no such filesystems configured in GPFS<br>[ $? -eq 1 ] && [ "$FS" = " " ] && exit 0<br><br># uncomment this line for testing<br>#FS="$FS gpfsdummy"<br><br>while [ $TIMEOUT -gt 0 ]; do<br> for fs in ${FS}; do<br> if findmnt $fs -n &>/dev/null; then<br> FS=${FS/ $fs/}<br> continue 2;<br> fi<br> done<br> [ -z "${FS// /}" ] && break<br> (( TIMEOUT -= 5 ))<br> sleep 5<br>done<br><br>if [ -z "${FS// /}" ]; then<br> exit 0<br>else<br> echo >&2 "ERROR: filesystem(s) not found in time:${FS}"<br> exit 2<br>fi<br>--------------------------------------------------<br><br><br>This works without problems on _most_ of our clusters. However, not on all. Some of them show what I<br>believe is a race condition and fail to startup after a reboot:<br><br>----------------------------------------------------------------------<br># journalctl -u gpfs<br>-- Logs begin at Fri 2020-04-24 17:11:26 CEST, end at Tue 2020-04-28 12:47:34 CEST. --<br>Apr 24 17:12:13 myhost systemd[1]: Starting General Parallel File System...<br>Apr 24 17:12:17 myhost mmfs[5720]: [X] Cannot open configuration file /var/mmfs/gen/mmfs.cfg.<br>Apr 24 17:13:44 myhost systemd[1]: gpfs.service start-post operation timed out. Stopping.<br>Apr 24 17:13:44 myhost mmremote[8966]: Shutting down!<br>Apr 24 17:13:48 myhost mmremote[8966]: Unloading modules from<br>/lib/modules/3.10.0-1062.18.1.el7.x86_64/extra<br>Apr 24 17:13:48 myhost mmremote[8966]: Unloading module mmfs26<br>Apr 24 17:13:48 myhost mmremote[8966]: Unloading module mmfslinux<br>Apr 24 17:13:48 myhost systemd[1]: Failed to start General Parallel File System.<br>Apr 24 17:13:48 myhost systemd[1]: Unit gpfs.service entered failed state.<br>Apr 24 17:13:48 myhost systemd[1]: gpfs.service failed.<br>----------------------------------------------------------------------<br><br>The mmfs.log shows a bit more:<br>----------------------------------------------------------------------<br># less /var/adm/ras/mmfs.log.previous<br>2020-04-24_17:12:14.609+0200: runmmfs starting (4254)<br>2020-04-24_17:12:14.622+0200: [I] Removing old /var/adm/ras/mmfs.log.* files:<br>2020-04-24_17:12:14.658+0200: runmmfs: [I] Unloading modules from<br>/lib/modules/3.10.0-1062.18.1.el7.x86_64/extra<br>2020-04-24_17:12:14.692+0200: runmmfs: [I] Unloading module mmfs26<br>2020-04-24_17:12:14.901+0200: runmmfs: [I] Unloading module mmfslinux<br>2020-04-24_17:12:15.018+0200: runmmfs: [I] Unloading module tracedev<br>2020-04-24_17:12:15.057+0200: runmmfs: [I] Loading modules from<br>/lib/modules/3.10.0-1062.18.1.el7.x86_64/extra<br>Module Size Used by<br>mmfs26 2657452 0<br>mmfslinux 809734 1 mmfs26<br>tracedev 48618 2 mmfs26,mmfslinux<br><br><br>2020-04-24_17:12:16.720+0200: Node rebooted. Starting mmautoload...<br>2020-04-24_17:12:17.011+0200: [I] This node has a valid standard license<br>2020-04-24_17:12:17.011+0200: [I] Initializing the fast condition variables at 0x5561DFC365C0 ...<br>2020-04-24_17:12:17.011+0200: [I] mmfsd initializing. {Version: 5.0.4.2 Built: Jan 27 2020<br>12:13:06} ...<br>2020-04-24_17:12:17.011+0200: [I] Cleaning old shared memory ...<br>2020-04-24_17:12:17.012+0200: [I] First pass parsing mmfs.cfg ...<br>2020-04-24_17:12:17.013+0200: [X] Cannot open configuration file /var/mmfs/gen/mmfs.cfg.<br><br>2020-04-24_17:12:20.667+0200: mmautoload: Starting GPFS ...<br>2020-04-24_17:13:44.846+0200: mmremote: Initiating GPFS shutdown ...<br>2020-04-24_17:13:47.861+0200: mmremote: Starting the mmsdrserv daemon ...<br>2020-04-24_17:13:47.955+0200: mmremote: Unloading GPFS kernel modules ...<br>2020-04-24_17:13:48.165+0200: mmremote: Completing GPFS shutdown ...<br>--------------------------------------------------------------------------<br><br>Starting the gpfs service again manually then works without problems. Interestingly the missing<br>mmfs.cfg _is there_ after the shutdown, it gets created shortly after the failure. That's why I am<br>assuming a race condition:<br>--------------------------------------------------------------------------<br># stat /var/mmfs/gen/mmfs.cfg<br> File: ‘/var/mmfs/gen/mmfs.cfg’<br> Size: 408 Blocks: 8 IO Block: 4096 regular file<br>Device: fd00h/64768d Inode: 268998265 Links: 1<br>Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)<br>Context: system_u:object_r:var_t:s0<br>Access: 2020-04-27 17:12:19.801060073 +0200<br>Modify: 2020-04-24 17:12:17.617823441 +0200<br>Change: 2020-04-24 17:12:17.659823405 +0200<br> Birth: -<br>--------------------------------------------------------------------------<br><br><br>Now, the interesting part:<br>- removing the ExecStartPost script makes the issue vanish. Reboot is always startign gpfs successfully<br>- reducing the ExecStartPost to simply one line ("exit 0") makes the issue stay. gpfs startup always<br>fails.<br><br>Unfortunately IBM is refusing support because "the script is not coming with gpfs".<br><br>So I am searching for a solution that makes the script work on those servers again. Or a better way<br>to wait for all local gpfs mounts being ready. Has anyone written something like that already?<br><br>Thank you,<br><br>Uli<br>--<br>Science + Computing AG<br>Vorstandsvorsitzender/Chairman of the board of management:<br>Dr. Martin Matzke<br>Vorstand/Board of Management:<br>Matthias Schempp, Sabine Hohenstein<br>Vorsitzender des Aufsichtsrats/<br>Chairman of the Supervisory Board:<br>Philippe Miltin<br>Aufsichtsrat/Supervisory Board:<br>Martin Wibbe, Ursula Morgenstern<br>Sitz/Registered Office: Tuebingen<br>Registergericht/Registration Court: Stuttgart<br>Registernummer/Commercial Register No.: HRB 382196<br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> </font><br> </div></blockquote>
<div dir="ltr" > </div></div><BR>