<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:12pt" ><div dir="ltr" >Hi</div>
<div dir="ltr" > </div>
<div dir="ltr" >Also beware that KVM host and KVM guest cannot use the same FS nor belong to the same cluster</div>
<div dir="ltr" > </div>
<div dir="ltr" >Please see FAQ <a href="https://www.ibm.com/docs/en/spectrum-scale?topic=STXKQY/gpfsclustersfaq.html#gpfsclustersfaqAugust2016-gen4__virtualizationlimitsq">https://www.ibm.com/docs/en/spectrum-scale?topic=STXKQY/gpfsclustersfaq.html#gpfsclustersfaqAugust2016-gen4__virtualizationlimitsq</a></div>
<div dir="ltr" > </div>
<div dir="ltr" >Not saying it does not work, stating what the current documentation says that specific case. It is ok if KVM host is not part of Scale cluster at all.</div>
<div dir="ltr" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt" ><div dir="ltr" ><div style="font-size: 12pt; font-weight: bold; font-family: sans-serif; color: #7C7C5F;" ><span style="font-size:10pt;" ><font face="Arial" >--<br>Ystävällisin terveisin / Kind regards / Saludos cordiales / Salutations</font> / <font face="Arial" >Salutacions</font><br><font face="Arial" >Luis Bolinches</font></span></div>
<div style="font-size: 12pt; font-weight: bold; font-family: sans-serif; color: #7C7C5F;" ><div><span style="font-size:10pt;" ><font face="Arial" >Consultant IT Specialist</font></span></div>
<div><span style="font-size:10pt;" ><font face="Arial" >IBM Spectrum Scale development</font></span></div>
<div><span style="font-size:10pt;" ><font face="Arial" >Mobile Phone: +358503112585</font></span></div>
<div> </div>
<div><span style="font-size:10pt;" ><a href="https://www.youracclaim.com/user/luis-bolinches" rel="noopener" target="_blank"><b><font face="Arial" >https://www.youracclaim.com/user/luis-bolinches</font></b></a></span></div>
<div> </div>
<div><div><span style="font-size:10pt;" ><font face="Arial" >Ab IBM Finland Oy</font></span></div>
<div><span style="font-size:10pt;" ><font face="Arial" >Laajalahdentie 23</font></span></div>
<div><span style="font-size:10pt;" ><font face="Arial" >00330 Helsinki</font></span></div>
<div><span style="font-size:10pt;" ><font face="Arial" >Uusimaa - Finland</font></span></div><br><span style="font-size:10pt;" ><b><font face="Arial" color="#7C7C5F" >"If you always give you will always have" -- Anonymous</font></b></span></div></div>
<div style="font-size: 8pt; font-family: sans-serif; margin-top: 10px;" ><div> </div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Original message -----<br>From: Jan-Frode Myklebust <janfrode@tanso.net><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Cc:<br>Subject: [EXTERNAL] Re: [gpfsug-discuss] Using VMs as quorum / admin nodes in a GPFS infiniband cluster<br>Date: Mon, Jun 7, 2021 22:49<br> <br><!-- BaNnErBlUrFlE-HeAdEr-start --> <!-- BaNnErBlUrFlE-HeAdEr-end --> <!-- BaNnErBlUrFlE-BoDy-start --> <!-- Preheader Text : BEGIN --> <span style="display:none !important;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;max-width:0px;opacity:0;overflow:hidden;" > I’ve done this a few times. Once with IPoIB as daemon network, and then created a separate routed network on the hypervisor to bridge (?) between VM and IPoIB network. Example RHEL config where bond0 is an IP-over-IB bond on the hypervisor: </span> <!-- Preheader Text : END --> <!-- Email Banner : BEGIN --> <span style="display:none !important;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;max-width:0px;opacity:0;overflow:hidden;" >ZjQcmQRYFpfptBannerStart</span> <!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 3px 0px 16px 0px;" ><tr><td>
<table class="pfptBannerTableMSO" border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%;border-radius:4px;border-top:4px solid #90a4ae;background-color:#d0d8dc;" ><tr><td valign="top" >
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px" >
<tr><td><span class="pfptTitleMSO" style="color:#000000 !important;font-family: 'Arial', sans-serif;font-weight:bold !important;font-size:14px !important;" >
This Message Is From an External Sender
</span></td></tr>
<tr><td><span class="pfptSubtitleMSO" style="color:#000000 !important;font-weight:normal !important;font-family: 'Arial', sans-serif; font-size:12px !important;" >
This message came from outside your organization.
</span></td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]--> <!--[if !((ie)|(mso))]-->
<div style="margin:16px 0px 16px 0px; padding:8px 16px 8px 16px; border-radius: 4px; min-width: 200px;background-color: #d0d8dc; border-top: 4px solid #90a4ae;" ><div style="display: inline-block; margin: 0px 0px 1px 0px; max-width: 600px;" ><div style="color:#000000 !important;font-family: 'Arial', sans-serif !important;font-weight:bold !important;font-size:14px !important;line-height:18px;display:block;" >This Message Is From an External Sender</div>
<div style="color:#000000 !important;font-weight:normal !important;font-family: 'Arial', sans-serif !important;font-size:12px !important;line-height:18px;margin-top:2px;display:block" >This message came from outside your organization.</div></div></div><!--[endif]--> <span style="display:none !important;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;max-width:0px;opacity:0;overflow:hidden;" >ZjQcmQRYFpfptBannerEnd</span><br><!-- Email Banner : END --> <!-- BaNnErBlUrFlE-BoDy-end -->
<div> </div>
<div dir="auto" >I’ve done this a few times. Once with IPoIB as daemon network, and then created a separate routed network on the hypervisor to bridge (?) between VM and IPoIB network.</div>
<div dir="auto" > </div>
<div dir="auto" >Example RHEL config where bond0 is an IP-over-IB bond on the hypervisor:</div>
<div dir="auto" >————————</div>
<div dir="auto" ><div dir="auto" style="border-color:rgb(255,255,255)" > </div>
<div dir="auto" style="border-color:rgb(255,255,255)" >To give the VMs access to the daemon network, we need create an internal network for the VMs, that is then routed into the IPoIB network on the hypervisor.</div>
<div dir="auto" style="border-color:rgb(255,255,255)" > </div>
<div dir="auto" style="border-color:rgb(255,255,255)" >~~~</div>
<div dir="auto" style="border-color:rgb(255,255,255)" ># cat <<EOF > routed34.xml</div>
<div dir="auto" style="border-color:rgb(255,255,255)" ><network></div>
<div dir="auto" style="border-color:rgb(255,255,255)" > <name>routed34</name></div>
<div dir="auto" style="border-color:rgb(255,255,255)" > <forward mode='route' dev='bond0'/></div>
<div dir="auto" style="border-color:rgb(255,255,255)" > <bridge name='virbr34' stp='on' delay='2'/></div>
<div dir="auto" style="border-color:rgb(255,255,255)" > <ip address='10.0.0.1' netmask='255.255.255.0'></div>
<div dir="auto" style="border-color:rgb(255,255,255)" > <dhcp></div>
<div dir="auto" style="border-color:rgb(255,255,255)" > <range start='10.0.0.128' end='10.0.0.254'/></div>
<div dir="auto" style="border-color:rgb(255,255,255)" > </dhcp></div>
<div dir="auto" style="border-color:rgb(255,255,255)" > </ip></div>
<div dir="auto" style="border-color:rgb(255,255,255)" ></network></div>
<div dir="auto" style="border-color:rgb(255,255,255)" >EOF</div>
<div dir="auto" style="border-color:rgb(255,255,255)" ># virsh net-define routed34.xml</div>
<div dir="auto" style="border-color:rgb(255,255,255)" >Network routed34 defined from routed34.xml</div>
<div dir="auto" style="border-color:rgb(255,255,255)" > </div>
<div dir="auto" style="border-color:rgb(255,255,255)" ># virsh net-start routed34</div>
<div dir="auto" style="border-color:rgb(255,255,255)" >Network routed34 started</div>
<div dir="auto" style="border-color:rgb(255,255,255)" > </div>
<div dir="auto" style="border-color:rgb(255,255,255)" ># virsh net-autostart routed34</div>
<div dir="auto" style="border-color:rgb(255,255,255)" >Network routed34 marked as autostarted</div>
<div dir="auto" style="border-color:rgb(255,255,255)" > </div>
<div dir="auto" style="border-color:rgb(255,255,255)" ># virsh net-list --all</div>
<div dir="auto" style="border-color:rgb(255,255,255)" > Name State Autostart Persistent</div>
<div dir="auto" style="border-color:rgb(255,255,255)" >----------------------------------------------------------</div>
<div dir="auto" style="border-color:rgb(255,255,255)" > default active yes yes</div>
<div dir="auto" style="border-color:rgb(255,255,255)" > routed34 active yes yes</div>
<div dir="auto" style="border-color:rgb(255,255,255)" > </div>
<div dir="auto" style="border-color:rgb(255,255,255)" >~~~</div>
<div dir="auto" style="border-color:rgb(255,255,255)" > </div></div>
<div dir="auto" >————————-</div>
<div dir="auto" > </div>
<div dir="auto" > </div>
<div dir="auto" >I see no issue with it — but beware that the FAQ lists some required tunings if the VM is to host desconly disks (paniconiohang?)… </div>
<div dir="auto" > </div>
<div dir="auto" > </div>
<div dir="auto" > </div>
<div dir="auto" > -jf</div>
<div dir="auto" > </div>
<div>
<div><div dir="ltr" >man. 7. jun. 2021 kl. 14:55 skrev Leonardo Sala <<a href="mailto:leonardo.sala@psi.ch" target="_blank">leonardo.sala@psi.ch</a>>:</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)" ><div><p><font style="font-family:monospace;color:rgb(0,0,0)" face="monospace" >Hallo,</font></p>
<p><font style="font-family:monospace;color:rgb(0,0,0)" face="monospace" >we do have multiple bare-metal GPFS clusters with infiniband fabric, and I am actually considering adding some VMs in the mix, to perform admin tasks (so that the bare metal servers do not need passwordless ssh keys) and quorum nodes. Has anybody tried this? What could be the drawbacks / issues at GPFS level?</font></p>
<p><font style="font-family:monospace;color:rgb(0,0,0)" face="monospace" >Thanks a lot for the insights!</font></p>
<p><font style="font-family:monospace;color:rgb(0,0,0)" face="monospace" >cheers</font></p>
<p><font style="font-family:monospace;color:rgb(0,0,0)" face="monospace" >leo</font></p>
<div><font size="2" face="Default Monospace,Courier New,Courier,monospace" >--<br>Paul Scherrer Institut<br>Dr. Leonardo Sala<br>Group Leader High Performance Computing<br>Deputy Section Head Science IT<br>Science IT<br>WHGA/036<br>Forschungstrasse 111<br>5232 Villigen PSI<br>Switzerland<br><br>Phone: +41 56 310 3369<br><a href="mailto:leonardo.sala@psi.ch" style="font-family:monospace" target="_blank">leonardo.sala@psi.ch</a><br><a href="http://www.psi.ch" style="font-family:monospace" target="_blank">www.psi.ch</a></font></div></div>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></blockquote></div></div>
<div><font size="2" face="Default Monospace,Courier New,Courier,monospace" >_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> </font></div></blockquote>
<div dir="ltr" > </div></div><BR>
<BR>
<BR>
Ellei edellä ole toisin mainittu: / Unless stated otherwise above:<BR>
<BR>
Oy IBM Finland Ab<BR>
<BR>
PL 265, 00101 Helsinki, Finland<BR>
<BR>
Business ID, Y-tunnus: 0195876-3 <BR>
<BR>
Registered in Finland<BR>
<BR>
<BR>