<div><br></div><div dir="auto">I’ve done this a few times. Once with IPoIB as daemon network, and then created a separate routed network on the hypervisor to bridge (?) between VM and IPoIB network.</div><div dir="auto"><br></div><div dir="auto">Example RHEL config where bond0 is an IP-over-IB bond on the hypervisor:</div><div dir="auto">————————</div><div dir="auto"><div style="border-color:rgb(255,255,255)" dir="auto"><br></div><div style="border-color:rgb(255,255,255)" dir="auto">To give the VMs access to the daemon network, we need create an internal network for the VMs, that is then routed into the IPoIB network on the hypervisor.</div><div style="border-color:rgb(255,255,255)" dir="auto"><br></div><div style="border-color:rgb(255,255,255)" dir="auto">~~~</div><div style="border-color:rgb(255,255,255)" dir="auto"># cat <<EOF > routed34.xml</div><div style="border-color:rgb(255,255,255)" dir="auto"><network></div><div style="border-color:rgb(255,255,255)" dir="auto"> <name>routed34</name></div><div style="border-color:rgb(255,255,255)" dir="auto"> <forward mode='route' dev='bond0'/></div><div style="border-color:rgb(255,255,255)" dir="auto"> <bridge name='virbr34' stp='on' delay='2'/></div><div style="border-color:rgb(255,255,255)" dir="auto"> <ip address='10.0.0.1' netmask='255.255.255.0'></div><div style="border-color:rgb(255,255,255)" dir="auto"> <dhcp></div><div style="border-color:rgb(255,255,255)" dir="auto"> <range start='10.0.0.128' end='10.0.0.254'/></div><div style="border-color:rgb(255,255,255)" dir="auto"> </dhcp></div><div style="border-color:rgb(255,255,255)" dir="auto"> </ip></div><div style="border-color:rgb(255,255,255)" dir="auto"></network></div><div style="border-color:rgb(255,255,255)" dir="auto">EOF</div><div style="border-color:rgb(255,255,255)" dir="auto"># virsh net-define routed34.xml</div><div style="border-color:rgb(255,255,255)" dir="auto">Network routed34 defined from routed34.xml</div><div style="border-color:rgb(255,255,255)" dir="auto"><br></div><div style="border-color:rgb(255,255,255)" dir="auto"># virsh net-start routed34</div><div style="border-color:rgb(255,255,255)" dir="auto">Network routed34 started</div><div style="border-color:rgb(255,255,255)" dir="auto"><br></div><div style="border-color:rgb(255,255,255)" dir="auto"># virsh net-autostart routed34</div><div style="border-color:rgb(255,255,255)" dir="auto">Network routed34 marked as autostarted</div><div style="border-color:rgb(255,255,255)" dir="auto"><br></div><div style="border-color:rgb(255,255,255)" dir="auto"># virsh net-list --all</div><div style="border-color:rgb(255,255,255)" dir="auto"> Name State Autostart Persistent</div><div style="border-color:rgb(255,255,255)" dir="auto">----------------------------------------------------------</div><div style="border-color:rgb(255,255,255)" dir="auto"> default active yes yes</div><div style="border-color:rgb(255,255,255)" dir="auto"> routed34 active yes yes</div><div style="border-color:rgb(255,255,255)" dir="auto"><br></div><div style="border-color:rgb(255,255,255)" dir="auto">~~~</div><div style="border-color:rgb(255,255,255)" dir="auto"><br></div></div><div dir="auto">————————-</div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto">I see no issue with it — but beware that the FAQ lists some required tunings if the VM is to host desconly disks (paniconiohang?)… </div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"> -jf</div><div dir="auto"><br></div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">man. 7. jun. 2021 kl. 14:55 skrev Leonardo Sala <<a href="mailto:leonardo.sala@psi.ch">leonardo.sala@psi.ch</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)">
<div>
<p><font face="monospace" style="font-family:monospace;color:rgb(0,0,0)">Hallo,</font></p>
<p><font face="monospace" style="font-family:monospace;color:rgb(0,0,0)">we do have multiple bare-metal GPFS
clusters with infiniband fabric, and I am actually considering
adding some VMs in the mix, to perform admin tasks (so that the
bare metal servers do not need passwordless ssh keys) and quorum
nodes. Has anybody tried this? What could be the drawbacks /
issues at GPFS level?</font></p>
<p><font face="monospace" style="font-family:monospace;color:rgb(0,0,0)">Thanks a lot for the insights!</font></p>
<p><font face="monospace" style="font-family:monospace;color:rgb(0,0,0)">cheers</font></p>
<p><font face="monospace" style="font-family:monospace;color:rgb(0,0,0)">leo</font><br>
</p>
<pre cols="72" style="font-family:monospace">--
Paul Scherrer Institut
Dr. Leonardo Sala
Group Leader High Performance Computing
Deputy Section Head Science IT
Science IT
WHGA/036
Forschungstrasse 111
5232 Villigen PSI
Switzerland
Phone: +41 56 310 3369
<a href="mailto:leonardo.sala@psi.ch" target="_blank" style="font-family:monospace">leonardo.sala@psi.ch</a>
<a href="http://www.psi.ch" target="_blank" style="font-family:monospace">www.psi.ch</a></pre>
</div>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
</blockquote></div></div>