<div dir="ltr">Thanks Olaf, but we don't use NetworkManager on this cluster..<div><br></div><div>I now created this simple script:</div><div><br></div><div><div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br class="gmail-Apple-interchange-newline">-------------------------------------------------------------------------------------------------------------------------------------------------------------</div></div><div>#! /bin/bash -</div><div>#</div><div># Fail mmstartup if not all configured IB ports are active.</div><div>#</div><div># Install with:</div><div>#</div><div>#<span style="white-space:pre"> </span>mmaddcallback fail-if-ibfail --command /var/mmfs/etc/fail-if-ibfail --event preStartup --sync --onerror shutdown</div><div>#</div><div><br></div><div>for port in $(/usr/lpp/mmfs/bin/mmdiag --config|grep verbsPorts | cut -f 4- -d " ")</div><div>do</div><div><span style="white-space:pre"> </span>grep -q ACTIVE /sys/class/infiniband/${port%/*}/ports/${port##*/}/state || exit 1</div><div>done</div></div><div>-------------------------------------------------------------------------------------------------------------------------------------------------------------</div><div><br></div><div>which I haven't tested, but assume should work. Suggestions for improvements would be much appreciated!</div><div><br></div><div><br></div><div><br></div><div> -jf</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Mar 15, 2018 at 6:30 PM, Olaf Weiser <span dir="ltr"><<a href="mailto:olaf.weiser@de.ibm.com" target="_blank">olaf.weiser@de.ibm.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><br><font size="2" face="sans-serif">you can try :</font><br><font size="2" face="sans-serif">systemctl enable NetworkManager-wait-online<br>ln -s '/usr/lib/systemd/system/<wbr>NetworkManager-wait-online.<wbr>service' '/etc/systemd/system/multi-<wbr>user.target.wants/<wbr>NetworkManager-wait-online.<wbr>service'</font><br><br><font size="2" face="sans-serif">in many cases .. it helps .. </font><br><br><br><br><br><br><font size="1" color="#5f5f5f" face="sans-serif">From:
</font><font size="1" face="sans-serif">Jan-Frode Myklebust
<<a href="mailto:janfrode@tanso.net" target="_blank">janfrode@tanso.net</a>></font><span class=""><br><font size="1" color="#5f5f5f" face="sans-serif">To:
</font><font size="1" face="sans-serif">gpfsug main discussion
list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.<wbr>org</a>></font><br></span><font size="1" color="#5f5f5f" face="sans-serif">Date:
</font><font size="1" face="sans-serif">03/15/2018 06:18 PM</font><span class=""><br><font size="1" color="#5f5f5f" face="sans-serif">Subject:
</font><font size="1" face="sans-serif">Re: [gpfsug-discuss]
GPFS autoload - wait for IB ports to becomeactive</font><br><font size="1" color="#5f5f5f" face="sans-serif">Sent by:
</font><font size="1" face="sans-serif"><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@<wbr>spectrumscale.org</a></font><br><hr noshade><br><br><br></span><font size="3">I found some discussion on this at </font><a href="https://www.ibm.com/developerworks/community/forums/html/threadTopic?id=77777777-0000-0000-0000-000014471957&ps=25" target="_blank"><font size="3" color="blue"><u>https://www.ibm.com/<wbr>developerworks/community/<wbr>forums/html/threadTopic?id=<wbr>77777777-0000-0000-0000-<wbr>000014471957&ps=25</u></font></a><font size="3">and there it's claimed that none of the callback events are early enough
to resolve this. That we need a pre-preStartup trigger. Any idea if this
has changed -- or is the callback option then only to do a "</font><font size="1" color="#2f2f2f" face="Helvetica Neue">--onerror
shutdown"</font><font size="3"> if it has failed to connect IB ?</font><div><div class="h5"><br><br><font size="3">On Thu, Mar 8, 2018 at 1:42 PM, Frederick Stock <</font><a href="mailto:stockf@us.ibm.com" target="_blank"><font size="3" color="blue"><u>stockf@us.ibm.com</u></font></a><font size="3">>
wrote:</font><br><font size="3" face="Arial">You could also use the GPFS prestartup callback
(mmaddcallback) to execute a script synchronously that waits for the IB
ports to become available before returning and allowing GPFS to continue.
Not systemd integrated but it should work.</font><font size="3"><br></font><font size="3" face="sans-serif"><br>Fred<br>______________________________<wbr>____________________<br>Fred Stock | IBM Pittsburgh Lab | </font><a href="tel:(720)%20430-8821" target="_blank"><font size="3" color="blue" face="sans-serif"><u>720-430-8821</u></font></a><font size="3" color="blue" face="sans-serif"><u><br></u></font><a href="mailto:stockf@us.ibm.com" target="_blank"><font size="3" color="blue" face="sans-serif"><u>stockf@us.ibm.com</u></font></a><font size="3"><br><br><br></font><font size="1" color="#5f5f5f" face="sans-serif"><br>From: </font><a href="mailto:david_johnson@brown.edu" target="_blank"><font size="1" color="blue" face="sans-serif"><u>david_johnson@brown.edu</u></font></a><font size="1" color="#5f5f5f" face="sans-serif"><br>To: </font><font size="1" face="sans-serif">gpfsug
main discussion list <</font><a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank"><font size="1" color="blue" face="sans-serif"><u>gpfsug-discuss@spectrumscale.<wbr>org</u></font></a><font size="1" face="sans-serif">></font><font size="1" color="#5f5f5f" face="sans-serif"><br>Date: </font><font size="1" face="sans-serif">03/08/2018
07:34 AM</font><font size="1" color="#5f5f5f" face="sans-serif"><br>Subject: </font><font size="1" face="sans-serif">Re:
[gpfsug-discuss] GPFS autoload - wait for IB ports to become
active</font><font size="1" color="#5f5f5f" face="sans-serif"><br>Sent by: </font><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank"><font size="1" color="blue" face="sans-serif"><u>gpfsug-discuss-bounces@<wbr>spectrumscale.org</u></font></a><font size="3"><br></font><hr noshade><br><font size="3"><br><br><br>Until IBM provides a solution, here is my workaround. Add it so it runs
before the gpfs script, I call it from our custom xcat diskless boot scripts.
Based on rhel7, not fully systemd integrated. YMMV!<br><br>Regards, <br> — ddj<br>——-<br>[ddj@storage041 ~]$ cat /etc/init.d/ibready <br>#! /bin/bash<br>#<br># chkconfig: 2345 06 94<br># /etc/rc.d/init.d/ibready<br># written in 2016 David D Johnson (ddj <at> </font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__brown.edu&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=u-EMob09-dkE6jZbD3dTjBi3vWhmDXtxiOK3nqFyIgY&s=DZ8S9rlTWQ8XfqHR6o5CWfRorBROzg9akyebO0kFd0M&e=" target="_blank"><font size="3" color="blue"><u>brown.edu</u></font></a><font size="3">)<br>#<br>### BEGIN INIT INFO<br># Provides: ibready<br># Required-Start:<br># Required-Stop:<br># Default-Stop:<br># Description: Block until infiniband is ready<br># Short-Description: Block until infiniband is ready<br>### END INIT INFO<br><br>RETVAL=0<br>if [[ -d /sys/class/infiniband ]] <br>then<br> IBDEVICE=$(dirname $(grep -il infiniband /sys/class/infiniband/*/ports/<wbr>1/link*
| head -n 1))<br>fi<br># See how we were called.<br>case "$1" in<br> start)<br> if [[ -n $IBDEVICE && -f $IBDEVICE/state
]]<br> then<br> echo -n "Polling
for InfiniBand link up: "<br> for (( count =
60; count > 0; count-- ))<br> do<br>
if grep -q ACTIVE $IBDEVICE/state<br>
then<br>
echo ACTIVE<br>
break<br>
fi<br>
echo -n "."<br>
sleep 5<br> done<br> if (( count <=
0 ))<br> then<br>
echo DOWN - $0 timed out<br> fi<br> fi<br> ;;<br> stop|restart|reload|force-<wbr>reload|condrestart|try-<wbr>restart)<br> ;;<br> status)<br> if [[ -n $IBDEVICE && -f $IBDEVICE/state
]]<br> then<br> echo "$IBDEVICE
is $(< $IBDEVICE/state) $(< $IBDEVICE/rate)"<br> else<br> echo "No IBDEVICE
found"<br> fi<br> ;;<br> *)<br> echo "Usage: ibready {start|stop|status|restart|<wbr>reload|force-reload|<wbr>condrestart|try-restart}"<br> exit 2<br>esac<br>exit ${RETVAL}<br>————<br><br> -- ddj<br>Dave Johnson<br><br>On Mar 8, 2018, at 6:10 AM, Caubet Serrabou Marc (PSI) <</font><a href="mailto:marc.caubet@psi.ch" target="_blank"><font size="3" color="blue"><u>marc.caubet@psi.ch</u></font></a><font size="3">>
wrote:<br></font><font size="2" face="Tahoma"><br>Hi all,<br><br>with autoload = yes we do not ensure that GPFS will be started after the
IB link becomes up. Is there a way to force GPFS waiting to start until
IB ports are up? This can be probably done by adding something like After=network-online.target
and Wants=network-online.target in the systemd file but I would like to
know if this is natively possible from the GPFS configuration.<br><br>Thanks a lot,<br>Marc <br>______________________________<wbr>___________<br>Paul Scherrer Institut <br>High Performance Computing<br>Marc Caubet Serrabou<br>WHGA/036<br>5232 Villigen PSI<br>Switzerland<br><br>Telephone: </font><a href="tel:+41%2056%20310%2046%2067" target="_blank"><font size="2" color="blue" face="Tahoma"><u>+41
56 310 46 67</u></font></a><font size="2" face="Tahoma"><br>E-Mail: </font><a href="mailto:marc.caubet@psi.ch" target="_blank"><font size="2" color="blue" face="Tahoma"><u>marc.caubet@psi.ch</u></font></a><font size="3"><br>______________________________<wbr>_________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=u-EMob09-dkE6jZbD3dTjBi3vWhmDXtxiOK3nqFyIgY&s=Cn4NIxkWXmTOrwjnMFpO8KxH1BvuZLdC5_C9fwPSQCg&e=" target="_blank"><font size="3" color="blue"><u>spectrumscale.org</u></font></a><br><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=u-EMob09-dkE6jZbD3dTjBi3vWhmDXtxiOK3nqFyIgY&s=JCfJgq6pZnKUI6d-rIgJXVcdZh7vmA5ypB1_goP_FFA&e=" target="_blank"><font size="3" color="blue"><u>http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</u></font></a><tt><font size="2">_______<wbr>______________________________<wbr>__________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font></tt><a href="http://spectrumscale.org/" target="_blank"><tt><font size="2" color="blue"><u>spectrumscale.org</u></font></tt></a><font size="3" color="blue"><u><br></u></font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=u-EMob09-dkE6jZbD3dTjBi3vWhmDXtxiOK3nqFyIgY&s=JCfJgq6pZnKUI6d-rIgJXVcdZh7vmA5ypB1_goP_FFA&e=" target="_blank"><tt><font size="2" color="blue"><u>https://urldefense.proofpoint.<wbr>com/v2/url?u=http-3A__gpfsug.<wbr>org_mailman_listinfo_gpfsug-<wbr>2Ddiscuss&d=DwICAg&c=jf_<wbr>iaSHvJObTbx-siA1ZOg&r=p_<wbr>1XEUyoJ7-VJxF_w8h9gJh8_<wbr>Wj0Pey73LCLLoxodpw&m=u-EMob09-<wbr>dkE6jZbD3dTjBi3vWhmDXtxiOK3nqF<wbr>yIgY&s=JCfJgq6pZnKUI6d-<wbr>rIgJXVcdZh7vmA5ypB1_goP_FFA&e=</u></font></tt></a><font size="3"><br><br><br><br><br>______________________________<wbr>_________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="http://spectrumscale.org/" target="_blank"><font size="3" color="blue"><u>spectrumscale.org</u></font></a><font size="3" color="blue"><u><br></u></font><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><font size="3" color="blue"><u>http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</u></font></a><font size="3"><br></font><br><tt><font size="2">______________________________<wbr>_________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br></font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><tt><font size="2">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</font></tt></a><tt><font size="2"><br></font></tt><br><br></div></div></div><br>
<br>______________________________<wbr>_________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</a><br>
<br></blockquote></div><br></div>