<div dir="ltr">I found some discussion on this at <a href="https://www.ibm.com/developerworks/community/forums/html/threadTopic?id=77777777-0000-0000-0000-000014471957&ps=25">https://www.ibm.com/developerworks/community/forums/html/threadTopic?id=77777777-0000-0000-0000-000014471957&ps=25</a> and there it's claimed that none of the callback events are early enough to resolve this. That we need a pre-preStartup trigger. Any idea if this has changed -- or is the callback option then only to do a "<span style="color:rgb(34,34,34);font-family:"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:12px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;display:inline;float:none">--onerror shutdown"</span> if it has failed to connect IB ?<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Mar 8, 2018 at 1:42 PM, Frederick Stock <span dir="ltr"><<a href="mailto:stockf@us.ibm.com" target="_blank">stockf@us.ibm.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><font size="3" face="Arial">You could also use the GPFS prestartup callback
(mmaddcallback) to execute a script synchronously that waits for the IB
ports to become available before returning and allowing GPFS to continue.
Not systemd integrated but it should work.</font><br><br><font size="3" face="sans-serif">Fred<br>______________________________<wbr>____________________<br>Fred Stock | IBM Pittsburgh Lab | <a href="tel:(720)%20430-8821" value="+17204308821" target="_blank">720-430-8821</a><br><a href="mailto:stockf@us.ibm.com" target="_blank">stockf@us.ibm.com</a></font><br><br><br><br><font size="1" face="sans-serif" color="#5f5f5f">From:
</font><font size="1" face="sans-serif"><a href="mailto:david_johnson@brown.edu" target="_blank">david_johnson@brown.edu</a></font><br><font size="1" face="sans-serif" color="#5f5f5f">To:
</font><font size="1" face="sans-serif">gpfsug main discussion
list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.<wbr>org</a>></font><br><font size="1" face="sans-serif" color="#5f5f5f">Date:
</font><font size="1" face="sans-serif">03/08/2018 07:34 AM</font><br><font size="1" face="sans-serif" color="#5f5f5f">Subject:
</font><font size="1" face="sans-serif">Re: [gpfsug-discuss]
GPFS autoload - wait for IB ports to become active</font><br><font size="1" face="sans-serif" color="#5f5f5f">Sent by:
</font><font size="1" face="sans-serif"><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@<wbr>spectrumscale.org</a></font><br><hr noshade><div><div class="h5"><br><br><br><font size="3">Until IBM provides a solution, here is my workaround.
Add it so it runs before the gpfs script, I call it from our custom xcat
diskless boot scripts. Based on rhel7, not fully systemd integrated. YMMV!</font><br><br><font size="3">Regards, </font><br><font size="3"> — ddj</font><br><font size="3">——-</font><br><font size="3">[ddj@storage041 ~]$ cat /etc/init.d/ibready </font><br><font size="3">#! /bin/bash</font><br><font size="3">#</font><br><font size="3"># chkconfig: 2345 06 94</font><br><font size="3"># /etc/rc.d/init.d/ibready</font><br><font size="3"># written in 2016 David D Johnson (ddj <at> </font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__brown.edu&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=u-EMob09-dkE6jZbD3dTjBi3vWhmDXtxiOK3nqFyIgY&s=DZ8S9rlTWQ8XfqHR6o5CWfRorBROzg9akyebO0kFd0M&e=" target="_blank"><font size="3" color="blue"><u>brown.edu</u></font></a><font size="3">)</font><br><font size="3">#</font><br><font size="3">### BEGIN INIT INFO</font><br><font size="3"># Provides:
ibready</font><br><font size="3"># Required-Start:</font><br><font size="3"># Required-Stop:</font><br><font size="3"># Default-Stop:</font><br><font size="3"># Description: Block until infiniband is ready</font><br><font size="3"># Short-Description: Block until infiniband is ready</font><br><font size="3">### END INIT INFO</font><br><br><font size="3">RETVAL=0</font><br><font size="3">if [[ -d /sys/class/infiniband ]] </font><br><font size="3">then</font><br><font size="3"> IBDEVICE=$(dirname $(grep
-il infiniband /sys/class/infiniband/*/ports/<wbr>1/link* | head -n 1))</font><br><font size="3">fi</font><br><font size="3"># See how we were called.</font><br><font size="3">case "$1" in</font><br><font size="3"> start)</font><br><font size="3"> if [[ -n $IBDEVICE &&
-f $IBDEVICE/state ]]</font><br><font size="3"> then</font><br><font size="3">
echo -n "Polling for InfiniBand link up: "</font><br><font size="3">
for (( count = 60; count > 0; count-- ))</font><br><font size="3">
do</font><br><font size="3">
if grep -q ACTIVE $IBDEVICE/state</font><br><font size="3">
then</font><br><font size="3">
echo ACTIVE</font><br><font size="3">
break</font><br><font size="3">
fi</font><br><font size="3">
echo -n "."</font><br><font size="3">
sleep 5</font><br><font size="3">
done</font><br><font size="3">
if (( count <= 0 ))</font><br><font size="3">
then</font><br><font size="3">
echo DOWN - $0 timed out</font><br><font size="3">
fi</font><br><font size="3"> fi</font><br><font size="3"> ;;</font><br><font size="3"> stop|restart|reload|force-<wbr>reload|condrestart|try-<wbr>restart)</font><br><font size="3"> ;;</font><br><font size="3"> status)</font><br><font size="3"> if [[ -n $IBDEVICE &&
-f $IBDEVICE/state ]]</font><br><font size="3"> then</font><br><font size="3">
echo "$IBDEVICE is $(< $IBDEVICE/state) $(< $IBDEVICE/rate)"</font><br><font size="3"> else</font><br><font size="3">
echo "No IBDEVICE found"</font><br><font size="3"> fi</font><br><font size="3"> ;;</font><br><font size="3"> *)</font><br><font size="3"> echo "Usage: ibready
{start|stop|status|restart|<wbr>reload|force-reload|<wbr>condrestart|try-restart}"</font><br><font size="3"> exit 2</font><br><font size="3">esac</font><br><font size="3">exit ${RETVAL}</font><br><font size="3">————</font><br><br><font size="3"> -- ddj</font><br><font size="3">Dave Johnson</font><br><font size="3"><br>On Mar 8, 2018, at 6:10 AM, Caubet Serrabou Marc (PSI) <</font><a href="mailto:marc.caubet@psi.ch" target="_blank"><font size="3" color="blue"><u>marc.caubet@psi.ch</u></font></a><font size="3">>
wrote:<br></font><br><font size="2" face="Tahoma">Hi all,</font><br><font size="2" face="Tahoma"><br>with autoload = yes we do not ensure that GPFS will be started after the
IB link becomes up. Is there a way to force GPFS waiting to start until
IB ports are up? This can be probably done by adding something like After=network-online.target
and Wants=network-online.target in the systemd file but I would like to
know if this is natively possible from the GPFS configuration.<br><br>Thanks a lot,<br>Marc </font><br><font size="2" face="Tahoma">______________________________<wbr>___________<br>Paul Scherrer Institut <br>High Performance Computing<br>Marc Caubet Serrabou<br>WHGA/036<br>5232 Villigen PSI<br>Switzerland<br><br>Telephone: <a href="tel:+41%2056%20310%2046%2067" value="+41563104667" target="_blank">+41 56 310 46 67</a><br>E-Mail: </font><a href="mailto:marc.caubet@psi.ch" target="_blank"><font size="2" face="Tahoma" color="blue"><u>marc.caubet@psi.ch</u></font></a><br><font size="3">______________________________<wbr>_________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at </font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=u-EMob09-dkE6jZbD3dTjBi3vWhmDXtxiOK3nqFyIgY&s=Cn4NIxkWXmTOrwjnMFpO8KxH1BvuZLdC5_C9fwPSQCg&e=" target="_blank"><font size="3" color="blue"><u>spectrumscale.org</u></font></a><font size="3" color="blue"><u><br></u></font></div></div><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=u-EMob09-dkE6jZbD3dTjBi3vWhmDXtxiOK3nqFyIgY&s=JCfJgq6pZnKUI6d-rIgJXVcdZh7vmA5ypB1_goP_FFA&e=" target="_blank"><font size="3" color="blue"><u>http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</u></font></a><tt><font size="2">_______<wbr>______________________________<wbr>__________<span class=""><br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br></span></font></tt><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=u-EMob09-dkE6jZbD3dTjBi3vWhmDXtxiOK3nqFyIgY&s=JCfJgq6pZnKUI6d-rIgJXVcdZh7vmA5ypB1_goP_FFA&e=" target="_blank"><tt><font size="2">https://urldefense.proofpoint.<wbr>com/v2/url?u=http-3A__gpfsug.<wbr>org_mailman_listinfo_gpfsug-<wbr>2Ddiscuss&d=DwICAg&c=jf_<wbr>iaSHvJObTbx-siA1ZOg&r=p_<wbr>1XEUyoJ7-VJxF_w8h9gJh8_<wbr>Wj0Pey73LCLLoxodpw&m=u-EMob09-<wbr>dkE6jZbD3dTjBi3vWhmDXtxiOK3nqF<wbr>yIgY&s=JCfJgq6pZnKUI6d-<wbr>rIgJXVcdZh7vmA5ypB1_goP_FFA&e=</font></tt></a><tt><font size="2"><br></font></tt><br><br><br>
<br>______________________________<wbr>_________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</a><br>
<br></blockquote></div><br></div>