[gpfsug-discuss] data interface and management infercace.
Salvatore Di Nardo
sdinardo at ebi.ac.uk
Tue Jul 14 09:15:26 BST 2015
Thanks,
this has already been done ( without too much success).
We need to rearrange the networking and since somebody experience was to
add a copper interface for management i want to do the same, so i'm
digging a bit to aundertsand the best way yo do it.
Regards,
Salvatore
On 14/07/15 08:31, Hagley Birgit wrote:
> Hello Salvatore,
>
> as you wrote that you have about 700 clients, maybe also the tuning
> recommendations for large GPFS clusters are helpful for you. They are
> on the developerworks GPFS wiki:
>
> https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20%28GPFS%29/page/Best%20Practices%20Network%20Tuning
> <https://www.ibm.com/developerworks/community/wikis/home?lang=en#%21/wiki/General%20Parallel%20File%20System%20%28GPFS%29/page/Best%20Practices%20Network%20Tuning>
>
>
> To my experience especially "failureDetectionTime" and
> "minMissedPingTimeout" may help in case of expelled nodes.
>
>
> In case you use InfiniBand, for RDMA, there also is a "Best Practices
> RDMA Tuning" page:
>
> https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20%28GPFS%29/page/Best%20Practices%20RDMA%20Tuning
> <https://www.ibm.com/developerworks/community/wikis/home?lang=en#%21/wiki/General%20Parallel%20File%20System%20%28GPFS%29/page/Best%20Practices%20RDMA%20Tuning>
>
>
>
> Regards
> Birgit
>
> ------------------------------------------------------------------------
> *From:* gpfsug-discuss-bounces at gpfsug.org
> [gpfsug-discuss-bounces at gpfsug.org] on behalf of Salvatore Di Nardo
> [sdinardo at ebi.ac.uk]
> *Sent:* Monday, July 13, 2015 3:29 PM
> *To:* Vic Cornell
> *Cc:* gpfsug main discussion list
> *Subject:* Re: [gpfsug-discuss] data interface and management infercace.
>
> Hello Vic.
> We are currently draining our gpfs to do all the recabling to add a
> management network, but looking what the admin interface does ( man
> mmchnode ) it says something different:
>
> --admin-interface={hostname | ip_address}
> Specifies the name of the node to be used by GPFS
> administration commands when communicating between nodes. The
> admin node name must be specified as an IP
> address or a hostname that is resolved by the host command to
> the desired IP address. If the keyword DEFAULT is specified,
> the admin interface for the
> node is set to be equal to the daemon interface for the node.
>
>
> So, seems used only for commands propagation, hence have nothing to
> do with the node-to-node traffic. Infact the other interface
> description is:
>
> --daemon-interface={hostname | ip_address}
> Specifies the host name or IP address _*to be used by the GPFS
> daemons for node-to-node communication*_. The host name or IP
> address must refer to the commu-
> nication adapter over which the GPFS daemons communicate.
> Alias interfaces are not allowed. Use the original address or
> a name that is resolved by the
> host command to that original address.
>
>
> The "expired lease" issue and file locking mechanism a( most of our
> expells happens when 2 clients try to write in the same file) are
> exactly node-to node-comunication, so im wondering what's the point to
> separate the "admin network". I want to be sure to plan the right
> changes before we do a so massive task. We are talking about adding a
> new interface on 700 clients, so the recabling work its not small.
>
>
> Regards,
> Salvatore
>
>
>
> On 13/07/15 14:00, Vic Cornell wrote:
>> Hi Salavatore,
>>
>> Does your GSS have the facility for a 1GbE “management” network? If
>> so I think that changing the “admin” node names of the cluster
>> members to a set of IPs on the management network would give you the
>> split that you need.
>>
>> What about the clients? Can they also connect to a separate admin
>> network?
>>
>> Remember that if you are using multi-cluster all of the nodes in both
>> networks must share the same admin network.
>>
>> Kind Regards,
>>
>> Vic
>>
>>
>>> On 13 Jul 2015, at 13:31, Salvatore Di Nardo <sdinardo at ebi.ac.uk
>>> <mailto:sdinardo at ebi.ac.uk>> wrote:
>>>
>>> Anyone?
>>>
>>> On 10/07/15 11:07, Salvatore Di Nardo wrote:
>>>> Hello guys.
>>>> Quite a while ago i mentioned that we have a big expel issue on
>>>> our gss ( first gen) and white a lot people suggested that the root
>>>> cause could be that we use the same interface for all the traffic,
>>>> and that we should split the data network from the admin network.
>>>> Finally we could plan a downtime and we are migrating the data out
>>>> so, i can soon safelly play with the change, but looking what
>>>> exactly i should to do i'm a bit puzzled. Our mmlscluster looks
>>>> like this:
>>>>
>>>> GPFS cluster information
>>>> ========================
>>>> GPFS cluster name: GSS.ebi.ac.uk <http://GSS.ebi.ac.uk>
>>>> GPFS cluster id: 17987981184946329605
>>>> GPFS UID domain: GSS.ebi.ac.uk <http://GSS.ebi.ac.uk>
>>>> Remote shell command: /usr/bin/ssh
>>>> Remote file copy command: /usr/bin/scp
>>>>
>>>> GPFS cluster configuration servers:
>>>> -----------------------------------
>>>> Primary server: gss01a.ebi.ac.uk <http://gss01a.ebi.ac.uk>
>>>> Secondary server: gss02b.ebi.ac.uk
>>>> <http://gss02b.ebi.ac.uk>
>>>>
>>>> Node Daemon node name IP address Admin node
>>>> name Designation
>>>> -----------------------------------------------------------------------
>>>> 1 gss01a.ebi.ac.uk <http://gss01a.ebi.ac.uk>
>>>> 10.7.28.2 gss01a.ebi.ac.uk <http://gss01a.ebi.ac.uk>
>>>> quorum-manager
>>>> 2 gss01b.ebi.ac.uk <http://gss01b.ebi.ac.uk>
>>>> 10.7.28.3 gss01b.ebi.ac.uk <http://gss01b.ebi.ac.uk>
>>>> quorum-manager
>>>> 3 gss02a.ebi.ac.uk <http://gss02a.ebi.ac.uk>
>>>> 10.7.28.67 gss02a.ebi.ac.uk <http://gss02a.ebi.ac.uk>
>>>> quorum-manager
>>>> 4 gss02b.ebi.ac.uk <http://gss02b.ebi.ac.uk>
>>>> 10.7.28.66 gss02b.ebi.ac.uk <http://gss02b.ebi.ac.uk>
>>>> quorum-manager
>>>> 5 gss03a.ebi.ac.uk <http://gss03a.ebi.ac.uk>
>>>> 10.7.28.34 gss03a.ebi.ac.uk <http://gss03a.ebi.ac.uk>
>>>> quorum-manager
>>>> 6 gss03b.ebi.ac.uk <http://gss03b.ebi.ac.uk>
>>>> 10.7.28.35 gss03b.ebi.ac.uk <http://gss03b.ebi.ac.uk>
>>>> quorum-manager
>>>>
>>>>
>>>> It was my understanding that the "admin node" should use a
>>>> different interface ( a 1g link copper should be fine), while the
>>>> daemon node is where the data was passing , so should point to the
>>>> bonded 10g interfaces. but when i read the mmchnode man page i
>>>> start to be quite confused. It says:
>>>>
>>>> --daemon-interface={hostname | ip_address}
>>>> Specifies the host name or IP address _*to be used by the GPFS
>>>> daemons for node-to-node communication*_. The host name or IP
>>>> address must refer to the communication adapter over which the GPFS
>>>> daemons communicate.
>>>> Alias interfaces are not allowed. Use the
>>>> original address or a name that is resolved by the host command to
>>>> that original address.
>>>>
>>>> --admin-interface={hostname | ip_address}
>>>> Specifies the name of the node to be used by GPFS administration
>>>> commands when communicating between nodes. The admin node name must
>>>> be specified as an IP address or a hostname that is resolved by
>>>> the host command
>>>> tothe desired IP address. If the keyword
>>>> DEFAULT is specified, the admin interface for the node is set to be
>>>> equal to the daemon interface for the node.
>>>>
>>>> What exactly means "node-to node-communications" ?
>>>> Means DATA or also the "lease renew", and the token communication
>>>> between the clients to get/steal the locks to be able to manage
>>>> concurrent write to thr same file?
>>>> Since we are getting expells ( especially when several clients
>>>> contends the same file ) i assumed i have to split this type of
>>>> packages from the data stream, but reading the documentation it
>>>> looks to me that those internal comunication between nodes use the
>>>> daemon-interface wich i suppose are used also for the data. so HOW
>>>> exactly i can split them?
>>>>
>>>>
>>>> Thanks in advance,
>>>> Salvatore
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> gpfsug-discuss mailing list
>>>> gpfsug-discuss atgpfsug.org <http://gpfsug.org>
>>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>>
>>> _______________________________________________
>>> gpfsug-discuss mailing list
>>> gpfsug-discuss at gpfsug.org <http://gpfsug.org>
>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20150714/8aefd8ac/attachment-0003.htm>
More information about the gpfsug-discuss
mailing list