<html><head></head><body><div>
<meta charset="utf-8"> <div id="compose-body-wrapper" dir="auto"><div dir="auto">Hi Leonardo,</div><div dir="auto"><br></div><div dir="auto">I’ve found it easier to think and use the subnet parameter as a “what is my preferred high speed network” parameter and can be used to allow common nodes to use a common high speed network.</div><div dir="auto"><br></div><div dir="auto">All nodes must have a common network for admin and data, however once and only once at mmstartup it will compare the subnet parameter to connect to the declared high speed networks (in addition to the common data network). It always treats RDMA as the first subnet regardless to what is defined in the subnets parameter.</div><div dir="auto"><br></div><div dir="auto">I commonly use this when</div><div dir="auto"><br></div><div dir="auto">2 node groups, Group A and Group B</div><div dir="auto"><br></div><div dir="auto">Both groups have different slurm queues and they use different filesets of gpfs.</div><div dir="auto"><br></div><div dir="auto">Group A has 1G and 10G</div><div dir="auto">Group B have 1G and 100G</div><div dir="auto"><br></div><div dir="auto">1G is a flat layer 2 network, let’s say 10.0.0.0/24</div><div dir="auto"><br></div><div dir="auto">10G 10.0.10.0/24</div><div dir="auto">100G 10.0.100.0/24</div><div dir="auto"><br></div><div dir="auto">The 10G and 100G are separate networks, no routing between them.</div><div dir="auto"><br></div><div dir="auto">In this instance I would define both the admin and data networks as 10.0.0.0/24</div><div dir="auto"><br></div><div dir="auto">But I would define the 10G and 100G networks under the subnets parameter.</div><div dir="auto"><br></div><div dir="auto">On mmstartup all nodes use the 1G for admin and data networks, but then they would compare subnet availability and escalate the data network to the subnet that is available to them. (Remembering RDMA is always the first subnet, regardless of definition and if available)</div><div dir="auto"><br></div><div dir="auto">Now in this instance the workload on the file system is separated by slurm and filesets, so there is minimal internode communication when accessing files, so you can get away with a “slow” 1G common network when Group A and Group B have to communicate. </div><div dir="auto">But Group A will communicate with group A over 10G and Group B over 1G. Then Group B will communicate to Group B over 100G but to Group A over 1G.</div><div dir="auto"><br></div><div dir="auto">You can also use this to define preferred routed subnets etc, however it is important to note that this is not an “LNET” feature/replacement, nor is it a HA/failover/tailback feature. All nodes must have a common network, whether routed or otherwise, the subnet parameter just allows you to define the preferred high speed network, and thus interface to use. It can help create flexibility in a gpfs cluster, but it can also be the source of an unexplained “slow” file system when a node fails to escalate from 1G to a high speed network slowing the entire Group down to 1G during a job.</div><div dir="auto"><br></div><div dir="auto">I hope this helps.</div><div dir="auto"><br></div><div dir="auto">— Lauz</div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto" id="tmjah_g_1299">Get <a href="https://bluemail.me">BlueMail</a> for Mobile</div></div><div class="replyHeader" dir="auto">Leonardo Sala wrote:</div><br><br><div><blockquote cite="mid:47e7d3ec-75a1-e53f-9125-8d5bbe0acaff@psi.ch" type="cite" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <p><font face="monospace">Dear all,</font></p> <p><font face="monospace">I do have a very naive question about the subnets parameter. I read the various docs [*], but it is not clear to me how do best define it in my use case. As an example, I have a CL1 storage cluster, and a CL2 client cluster that remotely mounts CL1 storage, and they are both connected to a private IB network 192.168.0.0. Where should I define subnets=192.168.0.0/CL1;CL2 : on CL1, CL2 or both? Some directionality seems implied in the docs, but it was not totally clear to me...<br> </font></p> <p><font face="monospace">Thanks a lot!</font></p> <p><font face="monospace">cheers</font></p> <p><font face="monospace">leo<br> </font></p> <p><font face="monospace">[*]</font></p> <p><font face="monospace"><a class="moz-txt-link-freetext" href="https://www.ibm.com/docs/en/spectrum-scale/5.1.7?topic=system-using-remote-access-multiple-network-definitions">https://www.ibm.com/docs/en/spectrum-scale/5.1.7?topic=system-using-remote-access-multiple-network-definitions</a></font></p> <p><font face="monospace"><a class="moz-txt-link-freetext" href="https://www.ibm.com/docs/en/spectrum-scale/5.1.7?topic=communication-using-public-private-ip-addresses-gpfs-nodes">https://www.ibm.com/docs/en/spectrum-scale/5.1.7?topic=communication-using-public-private-ip-addresses-gpfs-nodes</a></font><br> </p> <pre class="moz-signature" cols="72">-- Paul Scherrer InstitutDr. Leonardo SalaGroup Leader Data Analysis and Research InfrastructureDeputy Department Head a.i Science IT Infrastructure and Services departmentScience IT Infrastructure and Services department (AWI)WHGA/036Forschungstrasse <a href="tel:1115232" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="telephone" x-apple-data-detectors-result="4">1115232</a> Villigen PSISwitzerlandPhone: <a href="tel:+41%2056%20310" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="telephone" x-apple-data-detectors-result="5">+41 56 310</a> 3369<a class="moz-txt-link-abbreviated" href="mailto:leonardo.sala@psi.ch">leonardo.sala@psi.ch</a><a class="moz-txt-link-abbreviated" href="http://www.psi.ch">www.psi.ch</a></pre> </blockquote></div><div><blockquote cite="mid:47e7d3ec-75a1-e53f-9125-8d5bbe0acaff@psi.ch" type="cite" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <p><font face="monospace">Dear all,</font></p> <p><font face="monospace">I do have a very naive question about the subnets parameter. I read the various docs [*], but it is not clear to me how do best define it in my use case. As an example, I have a CL1 storage cluster, and a CL2 client cluster that remotely mounts CL1 storage, and they are both connected to a private IB network 192.168.0.0. Where should I define subnets=192.168.0.0/CL1;CL2 : on CL1, CL2 or both? Some directionality seems implied in the docs, but it was not totally clear to me...<br> </font></p> <p><font face="monospace">Thanks a lot!</font></p> <p><font face="monospace">cheers</font></p> <p><font face="monospace">leo<br> </font></p> <p><font face="monospace">[*]</font></p> <p><font face="monospace"><a class="moz-txt-link-freetext" href="https://www.ibm.com/docs/en/spectrum-scale/5.1.7?topic=system-using-remote-access-multiple-network-definitions">https://www.ibm.com/docs/en/spectrum-scale/5.1.7?topic=system-using-remote-access-multiple-network-definitions</a></font></p> <p><font face="monospace"><a class="moz-txt-link-freetext" href="https://www.ibm.com/docs/en/spectrum-scale/5.1.7?topic=communication-using-public-private-ip-addresses-gpfs-nodes">https://www.ibm.com/docs/en/spectrum-scale/5.1.7?topic=communication-using-public-private-ip-addresses-gpfs-nodes</a></font><br> </p> <pre class="moz-signature" cols="72">-- Paul Scherrer InstitutDr. Leonardo SalaGroup Leader Data Analysis and Research InfrastructureDeputy Department Head a.i Science IT Infrastructure and Services departmentScience IT Infrastructure and Services department (AWI)WHGA/036Forschungstrasse <a href="tel:1115232" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="telephone" x-apple-data-detectors-result="11">1115232</a> Villigen PSISwitzerlandPhone: <a href="tel:+41%2056%20310" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="telephone" x-apple-data-detectors-result="12">+41 56 310</a> 3369<a class="moz-txt-link-abbreviated" href="mailto:leonardo.sala@psi.ch">leonardo.sala@psi.ch</a><a class="moz-txt-link-abbreviated" href="http://www.psi.ch">www.psi.ch</a></pre> </blockquote></div>
</div></body></html>