<div dir="ltr">*All* nodes needs to be able to communicate on the daemon network. If they don't have access to this network, they can't join the cluster. It doesn't need to be same subnet, it can be routed. But they all have to be able to reach each other. If you use IPoIB, you likely need something to route between the IPoIB network and the outside world to reach the IP you have on your VM. I don't think you will be able to use an IP address in the IPoIB range for your VM, unless your vmware hypervisor is connected to the IB fabric, and can bridge it.. (doubt that's possible).<div><br>I've seen some customers avoid using IPoIB, and rather mix an ethernet for daemon network, and dedicate the infiniband network to RDMA. </div><div><br></div><div> -jf</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jun 17, 2021 at 8:35 AM Leonardo Sala <<a href="mailto:leonardo.sala@psi.ch">leonardo.sala@psi.ch</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div>
<p><font face="monospace">Hallo everybody</font></p>
<p><font face="monospace">thanks for the feedback! So, what it is
suggested is to create on the VM (in my case hosted on vSphere,
with only one NIC) a secondary IP within the IPoIP range, and
create a route for that IP range to go over the public IP (and
create a similar route on my bare-metal servers, so that the VM
IPoIB IPs are reached over the public network) - is that
correct?</font></p>
<p><font face="monospace">The only other options would be to ditch
IPoIB as daemon network, right? What happens if some nodes have
access to the daemon network over IPoIB, and other not - GPFS
goes back to public ip cluster wide, or else?</font></p>
<p><font face="monospace">Thanks again!</font></p>
<p><font face="monospace">regards</font></p>
<p><font face="monospace">leo</font><br>
</p>
<pre cols="72">Paul Scherrer Institut
Dr. Leonardo Sala
Group Leader High Performance Computing
Deputy Section Head Science IT
Science IT
WHGA/036
Forschungstrasse 111
5232 Villigen PSI
Switzerland
Phone: +41 56 310 3369
<a href="mailto:leonardo.sala@psi.ch" target="_blank">leonardo.sala@psi.ch</a>
<a href="http://www.psi.ch" target="_blank">www.psi.ch</a></pre>
<div>On 07.06.21 21:49, Jan-Frode Myklebust
wrote:<br>
</div>
<blockquote type="cite">
<div><br>
</div>
<div dir="auto">I’ve done this a few times. Once with IPoIB as
daemon network, and then created a separate routed network on
the hypervisor to bridge (?) between VM and IPoIB network.</div>
<div dir="auto"><br>
</div>
<div dir="auto">Example RHEL config where bond0 is an IP-over-IB
bond on the hypervisor:</div>
<div dir="auto">————————</div>
<div dir="auto">
<div style="border-color:rgb(255,255,255)" dir="auto"><br>
</div>
<div style="border-color:rgb(255,255,255)" dir="auto">To give
the VMs access to the daemon network, we need create an
internal network for the VMs, that is then routed into the
IPoIB network on the hypervisor.</div>
<div style="border-color:rgb(255,255,255)" dir="auto"><br>
</div>
<div style="border-color:rgb(255,255,255)" dir="auto">~~~</div>
<div style="border-color:rgb(255,255,255)" dir="auto"># cat
<<EOF > routed34.xml</div>
<div style="border-color:rgb(255,255,255)" dir="auto"><network></div>
<div style="border-color:rgb(255,255,255)" dir="auto">
<name>routed34</name></div>
<div style="border-color:rgb(255,255,255)" dir="auto">
<forward mode='route' dev='bond0'/></div>
<div style="border-color:rgb(255,255,255)" dir="auto">
<bridge name='virbr34' stp='on' delay='2'/></div>
<div style="border-color:rgb(255,255,255)" dir="auto"> <ip
address='10.0.0.1' netmask='255.255.255.0'></div>
<div style="border-color:rgb(255,255,255)" dir="auto">
<dhcp></div>
<div style="border-color:rgb(255,255,255)" dir="auto">
<range start='10.0.0.128' end='10.0.0.254'/></div>
<div style="border-color:rgb(255,255,255)" dir="auto">
</dhcp></div>
<div style="border-color:rgb(255,255,255)" dir="auto">
</ip></div>
<div style="border-color:rgb(255,255,255)" dir="auto"></network></div>
<div style="border-color:rgb(255,255,255)" dir="auto">EOF</div>
<div style="border-color:rgb(255,255,255)" dir="auto"># virsh
net-define routed34.xml</div>
<div style="border-color:rgb(255,255,255)" dir="auto">Network
routed34 defined from routed34.xml</div>
<div style="border-color:rgb(255,255,255)" dir="auto"><br>
</div>
<div style="border-color:rgb(255,255,255)" dir="auto"># virsh
net-start routed34</div>
<div style="border-color:rgb(255,255,255)" dir="auto">Network
routed34 started</div>
<div style="border-color:rgb(255,255,255)" dir="auto"><br>
</div>
<div style="border-color:rgb(255,255,255)" dir="auto"># virsh
net-autostart routed34</div>
<div style="border-color:rgb(255,255,255)" dir="auto">Network
routed34 marked as autostarted</div>
<div style="border-color:rgb(255,255,255)" dir="auto"><br>
</div>
<div style="border-color:rgb(255,255,255)" dir="auto"># virsh
net-list --all</div>
<div style="border-color:rgb(255,255,255)" dir="auto"> Name
State Autostart Persistent</div>
<div style="border-color:rgb(255,255,255)" dir="auto">----------------------------------------------------------</div>
<div style="border-color:rgb(255,255,255)" dir="auto"> default
active yes yes</div>
<div style="border-color:rgb(255,255,255)" dir="auto"> routed34
active yes yes</div>
<div style="border-color:rgb(255,255,255)" dir="auto"><br>
</div>
<div style="border-color:rgb(255,255,255)" dir="auto">~~~</div>
<div style="border-color:rgb(255,255,255)" dir="auto"><br>
</div>
</div>
<div dir="auto">————————-</div>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">I see no issue with it — but beware that the FAQ
lists some required tunings if the VM is to host desconly disks
(paniconiohang?)… </div>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
<div dir="auto"> -jf</div>
<div dir="auto"><br>
</div>
<div><br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">man. 7. jun. 2021 kl. 14:55
skrev Leonardo Sala <<a href="mailto:leonardo.sala@psi.ch" target="_blank">leonardo.sala@psi.ch</a>>:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)">
<div>
<p><font style="font-family:monospace;color:rgb(0,0,0)" face="monospace">Hallo,</font></p>
<p><font style="font-family:monospace;color:rgb(0,0,0)" face="monospace">we do have multiple bare-metal GPFS
clusters with infiniband fabric, and I am actually
considering adding some VMs in the mix, to perform
admin tasks (so that the bare metal servers do not
need passwordless ssh keys) and quorum nodes. Has
anybody tried this? What could be the drawbacks /
issues at GPFS level?</font></p>
<p><font style="font-family:monospace;color:rgb(0,0,0)" face="monospace">Thanks a lot for the insights!</font></p>
<p><font style="font-family:monospace;color:rgb(0,0,0)" face="monospace">cheers</font></p>
<p><font style="font-family:monospace;color:rgb(0,0,0)" face="monospace">leo</font><br>
</p>
<pre cols="72" style="font-family:monospace">--
Paul Scherrer Institut
Dr. Leonardo Sala
Group Leader High Performance Computing
Deputy Section Head Science IT
Science IT
WHGA/036
Forschungstrasse 111
5232 Villigen PSI
Switzerland
Phone: +41 56 310 3369
<a href="mailto:leonardo.sala@psi.ch" style="font-family:monospace" target="_blank">leonardo.sala@psi.ch</a>
<a href="http://www.psi.ch" style="font-family:monospace" target="_blank">www.psi.ch</a></pre>
</div>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
</blockquote>
</div>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a>
</pre>
</blockquote>
</div>
</blockquote></div>