[gpfsug-discuss] Question about subnets parameter and remote cluster mounts

Wed May 3 15:00:10 BST 2023

Hi Leonardo,
I’ve found it easier to think and use the subnet parameter as a “what is my preferred high speed network” parameter and can be used to allow common nodes to use a common high speed network.
All nodes must have a common network for admin and data, however once and only once at mmstartup it will compare the subnet parameter to connect to the declared high speed networks (in addition to the common data network). It always treats RDMA as the first subnet regardless to what is defined in the subnets parameter.
I commonly use this when
2 node groups, Group A and Group B
Both groups have different slurm queues and they use different filesets of gpfs.
Group A has 1G and 10G
Group B have 1G and 100G
1G is a flat layer 2 network, let’s say 10.0.0.0/24
10G 10.0.10.0/24
100G 10.0.100.0/24
The 10G and 100G are separate networks, no routing between them.
In this instance I would define both the admin and data networks as 10.0.0.0/24
But I would define the 10G and 100G networks under the subnets parameter.
On mmstartup all nodes use the 1G for admin and data networks, but then they would compare subnet availability and escalate the data network to the subnet that is available to them. (Remembering RDMA is always the first subnet, regardless of definition and if available)
Now in this instance the workload on the file system is separated by slurm and filesets, so there is minimal internode communication when accessing files, so you can get away with a “slow” 1G common network when Group A and Group B have to communicate. 
But Group A will communicate with group A over 10G and Group B over 1G. Then Group B will communicate to Group B over 100G but to Group A over 1G.
You can also use this to define preferred routed subnets etc, however it is important to note that this is not an “LNET” feature/replacement, nor is it a HA/failover/tailback feature. All nodes must have a common network, whether routed or otherwise, the subnet parameter just allows you to define the preferred high speed network, and thus interface to use. It can help create flexibility in a gpfs cluster, but it can also be the source of an unexplained “slow” file system when a node fails to escalate from 1G to a high speed network slowing the entire Group down to 1G during a job.
I hope this helps.
— Lauz
Get BlueMail <https://bluemail.me> for Mobile
Leonardo Sala wrote:
Dear all,
I do have a very naive question about the subnets parameter. I read the various docs [*], but it is not clear to me how do best define it in my use case. As an example, I have a CL1 storage cluster, and a CL2 client cluster that remotely mounts CL1 storage, and they are both connected to a private IB network 192.168.0.0. Where should I define subnets=192.168.0.0/CL1;CL2 : on CL1, CL2 or both? Some directionality seems implied in the docs, but it was not totally clear to me...
Thanks a lot!
cheers
leo
[*]
https://www.ibm.com/docs/en/spectrum-scale/5.1.7?topic=system-using-remote-access-multiple-network-definitions <https://www.ibm.com/docs/en/spectrum-scale/5.1.7?topic=system-using-remote-access-multiple-network-definitions> 
https://www.ibm.com/docs/en/spectrum-scale/5.1.7?topic=communication-using-public-private-ip-addresses-gpfs-nodes <https://www.ibm.com/docs/en/spectrum-scale/5.1.7?topic=communication-using-public-private-ip-addresses-gpfs-nodes> 
-- Paul Scherrer InstitutDr. Leonardo SalaGroup Leader Data Analysis and Research InfrastructureDeputy Department Head a.i Science IT Infrastructure and Services departmentScience IT Infrastructure and Services department (AWI)WHGA/036Forschungstrasse 1115232 <tel:1115232> Villigen PSISwitzerlandPhone: +41 56 310 <tel:+41%2056%20310> 3369leonardo.sala at psi.ch <mailto:leonardo.sala at psi.ch> www.psi.ch <http://www.psi.ch> 
Dear all,
I do have a very naive question about the subnets parameter. I read the various docs [*], but it is not clear to me how do best define it in my use case. As an example, I have a CL1 storage cluster, and a CL2 client cluster that remotely mounts CL1 storage, and they are both connected to a private IB network 192.168.0.0. Where should I define subnets=192.168.0.0/CL1;CL2 : on CL1, CL2 or both? Some directionality seems implied in the docs, but it was not totally clear to me...
Thanks a lot!
cheers
leo
[*]
https://www.ibm.com/docs/en/spectrum-scale/5.1.7?topic=system-using-remote-access-multiple-network-definitions <https://www.ibm.com/docs/en/spectrum-scale/5.1.7?topic=system-using-remote-access-multiple-network-definitions> 
https://www.ibm.com/docs/en/spectrum-scale/5.1.7?topic=communication-using-public-private-ip-addresses-gpfs-nodes <https://www.ibm.com/docs/en/spectrum-scale/5.1.7?topic=communication-using-public-private-ip-addresses-gpfs-nodes> 
-- Paul Scherrer InstitutDr. Leonardo SalaGroup Leader Data Analysis and Research InfrastructureDeputy Department Head a.i Science IT Infrastructure and Services departmentScience IT Infrastructure and Services department (AWI)WHGA/036Forschungstrasse 1115232 <tel:1115232> Villigen PSISwitzerlandPhone: +41 56 310 <tel:+41%2056%20310> 3369leonardo.sala at psi.ch <mailto:leonardo.sala at psi.ch> www.psi.ch <http://www.psi.ch> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20230503/43dc0620/attachment-0002.htm>