<div dir="ltr">Alternative solution we're trying...<div><br></div><div>Create the file /etc/systemd/system/gpfs.service.d/delay.conf containing:</div><div><br></div><div><div>[Service]</div><div> ExecStartPre=/bin/sleep 60</div></div><div><br></div><div><br></div><div>Then I expect we should have long enough delay for infiniband to start before starting gpfs..</div><div><br></div><div><br></div><div><br></div><div> -jf</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Mar 16, 2018 at 1:05 PM, Frederick Stock <span dir="ltr"><<a href="mailto:stockf@us.ibm.com" target="_blank">stockf@us.ibm.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><font size="3" face="Arial">I have my doubts that mmdiag can be used in
this script. In general the guidance is to avoid or be very careful
with mm* commands in a callback due to the potential for deadlock.</font><span class=""><br><br><font size="3" face="sans-serif">Fred<br>______________________________<wbr>____________________<br>Fred Stock | IBM Pittsburgh Lab | 720-430-8821<br><a href="mailto:stockf@us.ibm.com" target="_blank">stockf@us.ibm.com</a></font><br><br><br><br></span><span class=""><font size="1" color="#5f5f5f" face="sans-serif">From:
</font><font size="1" face="sans-serif">Jan-Frode Myklebust
<<a href="mailto:janfrode@tanso.net" target="_blank">janfrode@tanso.net</a>></font><br><font size="1" color="#5f5f5f" face="sans-serif">To:
</font><font size="1" face="sans-serif">gpfsug main discussion
list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.<wbr>org</a>></font><br></span><font size="1" color="#5f5f5f" face="sans-serif">Date:
</font><font size="1" face="sans-serif">03/16/2018 04:30 AM</font><div><div class="h5"><br><font size="1" color="#5f5f5f" face="sans-serif">Subject:
</font><font size="1" face="sans-serif">Re: [gpfsug-discuss]
GPFS autoload - wait for IB ports tobecomeactive</font><br><font size="1" color="#5f5f5f" face="sans-serif">Sent by:
</font><font size="1" face="sans-serif"><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@<wbr>spectrumscale.org</a></font><br><hr noshade><br><br><br><font size="3">Thanks Olaf, but we don't use NetworkManager on this cluster..</font><br><br><font size="3">I now created this simple script:</font><br><br><font size="2" color="#2f2f2f" face="Arial"><br>------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>-------</font><br><font size="3">#! /bin/bash -</font><br><font size="3">#</font><br><font size="3"># Fail mmstartup if not all configured IB ports are active.</font><br><font size="3">#</font><br><font size="3"># Install with:</font><br><font size="3">#</font><br><font size="3"># mmaddcallback fail-if-ibfail --command /var/mmfs/etc/fail-if-ibfail
--event preStartup --sync --onerror shutdown</font><br><font size="3">#</font><br><br><font size="3">for port in $(/usr/lpp/mmfs/bin/mmdiag --config|grep verbsPorts
| cut -f 4- -d " ")</font><br><font size="3">do</font><br><font size="3">grep -q ACTIVE /sys/class/infiniband/${port%/<wbr>*}/ports/${port##*/}/state
|| exit 1</font><br><font size="3">done</font><br><font size="3">------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>-------</font><br><br><font size="3">which I haven't tested, but assume should work. Suggestions
for improvements would be much appreciated!</font><br><br><br><br><font size="3"> -jf</font><br><br><br><font size="3">On Thu, Mar 15, 2018 at 6:30 PM, Olaf Weiser <</font><a href="mailto:olaf.weiser@de.ibm.com" target="_blank"><font size="3" color="blue"><u>olaf.weiser@de.ibm.com</u></font></a><font size="3">>
wrote:</font><br><font size="2" face="sans-serif"><br>you can try :<br>systemctl enable NetworkManager-wait-online<br>ln -s '/usr/lib/systemd/system/<wbr>NetworkManager-wait-online.<wbr>service' '/etc/systemd/system/multi-<wbr>user.target.wants/<wbr>NetworkManager-wait-online.<wbr>service'</font><font size="3"><br></font><font size="2" face="sans-serif"><br>in many cases .. it helps .. </font><font size="3"><br><br><br><br><br></font><font size="1" color="#5f5f5f" face="sans-serif"><br>From: </font><font size="1" face="sans-serif">Jan-Frode
Myklebust <</font><a href="mailto:janfrode@tanso.net" target="_blank"><font size="1" color="blue" face="sans-serif"><u>janfrode@tanso.net</u></font></a><font size="1" face="sans-serif">></font><font size="1" color="#5f5f5f" face="sans-serif"><br>To: </font><font size="1" face="sans-serif">gpfsug
main discussion list <</font><a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank"><font size="1" color="blue" face="sans-serif"><u>gpfsug-discuss@spectrumscale.<wbr>org</u></font></a><font size="1" face="sans-serif">></font><font size="1" color="#5f5f5f" face="sans-serif"><br>Date: </font><font size="1" face="sans-serif">03/15/2018
06:18 PM</font><font size="1" color="#5f5f5f" face="sans-serif"><br>Subject: </font><font size="1" face="sans-serif">Re:
[gpfsug-discuss] GPFS autoload - wait for IB ports to
becomeactive</font><font size="1" color="#5f5f5f" face="sans-serif"><br>Sent by: </font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank"><font size="1" color="blue" face="sans-serif"><u>gpfsug-discuss-bounces@<wbr>spectrumscale.org</u></font></a><font size="3"><br></font><hr noshade><font size="3"><br><br><br>I found some discussion on this at </font><a href="https://www.ibm.com/developerworks/community/forums/html/threadTopic?id=77777777-0000-0000-0000-000014471957&ps=25" target="_blank"><font size="3" color="blue"><u>https://www.ibm.com/<wbr>developerworks/community/<wbr>forums/html/threadTopic?id=<wbr>77777777-0000-0000-0000-<wbr>000014471957&ps=25</u></font></a><font size="3">and
there it's claimed that none of the callback events are early enough to
resolve this. That we need a pre-preStartup trigger. Any idea if this has
changed -- or is the callback option then only to do a "</font><font size="1" color="#2f2f2f" face="Helvetica Neue">--onerror
shutdown"</font><font size="3"> if it has failed to connect IB ?</font><br><font size="3"><br><br>On Thu, Mar 8, 2018 at 1:42 PM, Frederick Stock <</font><a href="mailto:stockf@us.ibm.com" target="_blank"><font size="3" color="blue"><u>stockf@us.ibm.com</u></font></a><font size="3">>
wrote:</font><font size="3" face="Arial"><br>You could also use the GPFS prestartup callback (mmaddcallback) to execute
a script synchronously that waits for the IB ports to become available
before returning and allowing GPFS to continue. Not systemd integrated
but it should work.</font><font size="3" face="sans-serif"><br><br>Fred<br>______________________________<wbr>____________________<br>Fred Stock | IBM Pittsburgh Lab | </font><a href="tel:(720)%20430-8821" target="_blank"><font size="3" color="blue" face="sans-serif"><u>720-430-8821</u></font></a><font size="3" color="blue"><u><br></u></font><a href="mailto:stockf@us.ibm.com" target="_blank"><font size="3" color="blue" face="sans-serif"><u>stockf@us.ibm.com</u></font></a><font size="3"><br><br></font><font size="1" color="#5f5f5f" face="sans-serif"><br><br>From: </font><a href="mailto:david_johnson@brown.edu" target="_blank"><font size="1" color="blue" face="sans-serif"><u>david_johnson@brown.edu</u></font></a><font size="1" color="#5f5f5f" face="sans-serif"><br>To: </font><font size="1" face="sans-serif">gpfsug
main discussion list <</font><a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank"><font size="1" color="blue" face="sans-serif"><u>gpfsug-discuss@spectrumscale.<wbr>org</u></font></a><font size="1" face="sans-serif">></font><font size="1" color="#5f5f5f" face="sans-serif"><br>Date: </font><font size="1" face="sans-serif">03/08/2018
07:34 AM</font><font size="1" color="#5f5f5f" face="sans-serif"><br>Subject: </font><font size="1" face="sans-serif">Re:
[gpfsug-discuss] GPFS autoload - wait for IB ports to become
active</font><font size="1" color="#5f5f5f" face="sans-serif"><br>Sent by: </font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank"><font size="1" color="blue" face="sans-serif"><u>gpfsug-discuss-bounces@<wbr>spectrumscale.org</u></font></a><font size="3"><br></font><hr noshade><font size="3"><br><br><br><br>Until IBM provides a solution, here is my workaround. Add it so it runs
before the gpfs script, I call it from our custom xcat diskless boot scripts.
Based on rhel7, not fully systemd integrated. YMMV!<br><br>Regards, <br> — ddj<br>——-<br>[ddj@storage041 ~]$ cat /etc/init.d/ibready <br>#! /bin/bash<br>#<br># chkconfig: 2345 06 94<br># /etc/rc.d/init.d/ibready<br># written in 2016 David D Johnson (ddj <at> </font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__brown.edu&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=u-EMob09-dkE6jZbD3dTjBi3vWhmDXtxiOK3nqFyIgY&s=DZ8S9rlTWQ8XfqHR6o5CWfRorBROzg9akyebO0kFd0M&e=" target="_blank"><font size="3" color="blue"><u>brown.edu</u></font></a><font size="3">)<br>#<br>### BEGIN INIT INFO<br># Provides: ibready<br># Required-Start:<br># Required-Stop:<br># Default-Stop:<br># Description: Block until infiniband is ready<br># Short-Description: Block until infiniband is ready<br>### END INIT INFO<br><br>RETVAL=0<br>if [[ -d /sys/class/infiniband ]] <br>then<br> IBDEVICE=$(dirname $(grep -il infiniband /sys/class/infiniband/*/ports/<wbr>1/link*
| head -n 1))<br>fi<br># See how we were called.<br>case "$1" in<br> start)<br> if [[ -n $IBDEVICE && -f $IBDEVICE/state
]]<br> then<br> echo -n "Polling
for InfiniBand link up: "<br> for (( count =
60; count > 0; count-- ))<br> do<br>
if grep -q ACTIVE $IBDEVICE/state<br>
then<br>
echo ACTIVE<br>
break<br>
fi<br>
echo -n "."<br>
sleep 5<br> done<br> if (( count <=
0 ))<br> then<br>
echo DOWN - $0 timed out<br> fi<br> fi<br> ;;<br> stop|restart|reload|force-<wbr>reload|condrestart|try-<wbr>restart)<br> ;;<br> status)<br> if [[ -n $IBDEVICE && -f $IBDEVICE/state
]]<br> then<br> echo "$IBDEVICE
is $(< $IBDEVICE/state) $(< $IBDEVICE/rate)"<br> else<br> echo "No IBDEVICE
found"<br> fi<br> ;;<br> *)<br> echo "Usage: ibready {start|stop|status|restart|<wbr>reload|force-reload|<wbr>condrestart|try-restart}"<br> exit 2<br>esac<br>exit ${RETVAL}<br>————<br><br> -- ddj<br>Dave Johnson<br><br>On Mar 8, 2018, at 6:10 AM, Caubet Serrabou Marc (PSI) <</font><a href="mailto:marc.caubet@psi.ch" target="_blank"><font size="3" color="blue"><u>marc.caubet@psi.ch</u></font></a><font size="3">>
wrote:</font><font size="2" face="Tahoma"><br><br>Hi all,<br><br>with autoload = yes we do not ensure that GPFS will be started after the
IB link becomes up. Is there a way to force GPFS waiting to start until
IB ports are up? This can be probably done by adding something like After=network-online.target
and Wants=network-online.target in the systemd file but I would like to
know if this is natively possible from the GPFS configuration.<br><br>Thanks a lot,<br>Marc <br>______________________________<wbr>___________<br>Paul Scherrer Institut <br>High Performance Computing<br>Marc Caubet Serrabou<br>WHGA/036<br>5232 Villigen PSI<br>Switzerland<br><br>Telephone: </font><a href="tel:+41%2056%20310%2046%2067" target="_blank"><font size="2" color="blue" face="Tahoma"><u>+41
56 310 46 67</u></font></a><font size="2" face="Tahoma"><br>E-Mail: </font><a href="mailto:marc.caubet@psi.ch" target="_blank"><font size="2" color="blue" face="Tahoma"><u>marc.caubet@psi.ch</u></font></a><font size="3"><br>______________________________<wbr>_________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=u-EMob09-dkE6jZbD3dTjBi3vWhmDXtxiOK3nqFyIgY&s=Cn4NIxkWXmTOrwjnMFpO8KxH1BvuZLdC5_C9fwPSQCg&e=" target="_blank"><font size="3" color="blue"><u>spectrumscale.org</u></font></a><font size="3" color="blue"><u><br></u></font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=u-EMob09-dkE6jZbD3dTjBi3vWhmDXtxiOK3nqFyIgY&s=JCfJgq6pZnKUI6d-rIgJXVcdZh7vmA5ypB1_goP_FFA&e=" target="_blank"><font size="3" color="blue"><u>http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</u></font></a><tt><font size="2">_______<wbr>______________________________<wbr>__________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font></tt><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org_&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=xImYTxt4pm1o5znVn5Vdoka2uxgsTRpmlCGdEWhB9vw&s=j35YX5vYr7_YZ5e8mzqvyCel2GUSQqjP2s7dBECkOQw&e=" target="_blank"><tt><font size="2" color="blue"><u>spectrumscale.org</u></font></tt></a><font size="3" color="blue"><u><br></u></font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=u-EMob09-dkE6jZbD3dTjBi3vWhmDXtxiOK3nqFyIgY&s=JCfJgq6pZnKUI6d-rIgJXVcdZh7vmA5ypB1_goP_FFA&e=" target="_blank"><tt><font size="2" color="blue"><u>https://urldefense.proofpoint.<wbr>com/v2/url?u=http-3A__gpfsug.<wbr>org_mailman_listinfo_gpfsug-<wbr>2Ddiscuss&d=DwICAg&c=jf_<wbr>iaSHvJObTbx-siA1ZOg&r=p_<wbr>1XEUyoJ7-VJxF_w8h9gJh8_<wbr>Wj0Pey73LCLLoxodpw&m=u-EMob09-<wbr>dkE6jZbD3dTjBi3vWhmDXtxiOK3nqF<wbr>yIgY&s=JCfJgq6pZnKUI6d-<wbr>rIgJXVcdZh7vmA5ypB1_goP_FFA&e=</u></font></tt></a><font size="3"><br><br><br><br><br>______________________________<wbr>_________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org_&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=xImYTxt4pm1o5znVn5Vdoka2uxgsTRpmlCGdEWhB9vw&s=j35YX5vYr7_YZ5e8mzqvyCel2GUSQqjP2s7dBECkOQw&e=" target="_blank"><font size="3" color="blue"><u>spectrumscale.org</u></font></a><font size="3" color="blue"><u><br></u></font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=xImYTxt4pm1o5znVn5Vdoka2uxgsTRpmlCGdEWhB9vw&s=veOZZz80aBzoCTKusx6WOpVlYs64eNkp5pM9kbHgvic&e=" target="_blank"><font size="3" color="blue"><u>http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</u></font></a><font size="3"><br></font><tt><font size="2"><br>______________________________<wbr>_________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font></tt><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=xImYTxt4pm1o5znVn5Vdoka2uxgsTRpmlCGdEWhB9vw&s=ocL9CBYdvYLa3eMuhGzZkyyDKzVCWSbQGeSj7t-OYTA&e=" target="_blank"><tt><font size="2" color="blue"><u>spectrumscale.org</u></font></tt></a><font size="3" color="blue"><u><br></u></font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=xImYTxt4pm1o5znVn5Vdoka2uxgsTRpmlCGdEWhB9vw&s=veOZZz80aBzoCTKusx6WOpVlYs64eNkp5pM9kbHgvic&e=" target="_blank"><tt><font size="2" color="blue"><u>http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</u></font></tt></a><font size="3"><br><br></font><br><font size="3"><br><br>______________________________<wbr>_________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=xImYTxt4pm1o5znVn5Vdoka2uxgsTRpmlCGdEWhB9vw&s=ocL9CBYdvYLa3eMuhGzZkyyDKzVCWSbQGeSj7t-OYTA&e=" target="_blank"><font size="3" color="blue"><u>spectrumscale.org</u></font></a><font size="3" color="blue"><u><br></u></font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=xImYTxt4pm1o5znVn5Vdoka2uxgsTRpmlCGdEWhB9vw&s=veOZZz80aBzoCTKusx6WOpVlYs64eNkp5pM9kbHgvic&e=" target="_blank"><font size="3" color="blue"><u>http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</u></font></a><font size="3"><br></font><br><tt><font size="2">______________________________<wbr>_________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br></font></tt></div></div><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=xImYTxt4pm1o5znVn5Vdoka2uxgsTRpmlCGdEWhB9vw&s=veOZZz80aBzoCTKusx6WOpVlYs64eNkp5pM9kbHgvic&e=" target="_blank"><tt><font size="2">https://urldefense.proofpoint.<wbr>com/v2/url?u=http-3A__gpfsug.<wbr>org_mailman_listinfo_gpfsug-<wbr>2Ddiscuss&d=DwICAg&c=jf_<wbr>iaSHvJObTbx-siA1ZOg&r=p_<wbr>1XEUyoJ7-VJxF_w8h9gJh8_<wbr>Wj0Pey73LCLLoxodpw&m=<wbr>xImYTxt4pm1o5znVn5Vdoka2uxgsTR<wbr>pmlCGdEWhB9vw&s=<wbr>veOZZz80aBzoCTKusx6WOpVlYs64eN<wbr>kp5pM9kbHgvic&e=</font></tt></a><tt><font size="2"><br></font></tt><br><br><br>
<br>______________________________<wbr>_________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</a><br>
<br></blockquote></div><br></div>