[gpfsug-discuss] Odd networking/name resolution issue

[Jaime Pinto]

The rationale for my suggestion doesn't have much to do with the central DNS server, but everything to do with the DNS client side of the service.
If you have a very busy cluster at times, and a number of nodes really busy with 1000+ IOPs for instance, so much that the OS on the client can't barely spare a cycle to query the DSN server on what the IP associated with the name of interface leading to the GPFS infrastructure is, or even process that response when it returns, on the same interface where it's having contentions and trying to process all the gpfs data transactions, you can have temporary catch 22 situations. This can generate a backlog of waiters, and eventual expelling of some nodes when the cluster managers don't hear from them in reasonable time.

It's doesn't really matter if you have a central DNS server in steroids.

Jaime

On 5/10/2020 03:35:29, TURNER Aaron wrote:
> Following on from Jonathan Buzzards comments, I'd also like to point out that I've never known a central DNS failure in a UK HEI for as long as I can remember, and it was certainly not my intention to suggest that as I think a central DNS issue is highly unlikely. And indeed, as I originally noted, the standard command-line tools on the nodes resolve the names as expected, so whatever is going on looks like it affects GPFS only. It may even be that the repetition of the domain names in the logs is just a function of something it is doing when logging when a node is failing to connect for some other reason entirely. It's just not something I recall having seen before and wanted to see if anyone else had seen it.
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> *From:* gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> on behalf of Jonathan Buzzard <jonathan.buzzard at strath.ac.uk>
> *Sent:* 09 May 2020 23:22
> *To:* gpfsug-discuss at spectrumscale.org <gpfsug-discuss at spectrumscale.org>
> *Subject:* Re: [gpfsug-discuss] Odd networking/name resolution issue
> On 09/05/2020 12:06, Jaime Pinto wrote:
>> DNS shouldn't be relied upon on a GPFS cluster for internal 
>> communication/management or data.
>> 
> 
> The 1980's have called and want their lack of IP resolution protocols
> back :-)
> 
> I would kindly disagree. If your DNS is not working then your cluster is
> fubar anyway and a zillion other things will also break very rapidly.
> For us at least half of the running jobs would be dead in a few minutes
> as failure to contact license servers would cause the software to stop.
> All authentication and account lookup is also going to fail as well.
> 
> You could distribute a hosts file but frankly outside of a storage only
> cluster (as opposed to one with hundreds if not thousands of compute
> nodes) that is frankly madness and will inevitably come to bite you in
> the ass because they *will* get out of sync. The only hosts entry we
> have is for the Salt Stack host because it tries to do things before the
> DNS resolvers have been setup and consequently breaks otherwise. Which
> IMHO is duff on it's behalf.
> 
> I would add I can't think of a time in the last 16 years where internal
> DNS at any University I have worked at has stopped working for even one
> millisecond. If DNS is that flaky at your institution then I suggest
> sacking the people responsible for it's maintenance as being incompetent
> twits. It is just such a vanishingly remote possibility that it's not
> worth bothering about. Frankly a aircraft falling out the sky and
> squishing your data centre seems more likely to me.
> 
> Finally in a world of IPv6 then anything other than DNS is a utter
> madness IMHO.
> 
> 
> JAB.
> 
> -- 
> Jonathan A. Buzzard                         Tel: +44141-5483420
> HPC System Administrator, ARCHIE-WeSt.
> University of Strathclyde, John Anderson Building, Glasgow. G4 0NG
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>