Asynchronous DNS client

[GoogleCodeExporter]

Currently, asyncio calls the standard getaddrinfo() function in a thread 
because this function is blocking. There is no standard asynchronous functio to 
resolve an host name. It would be interesting to try to plug one of the 
following asynchronous DNS client into asyncio:

http://www.gnu.org/software/adns/ => http://code.google.com/p/adns-python/
http://c-ares.haxx.se/ => https://pypi.python.org/pypi/pycares
https://github.com/getdnsapi/getdns
http://adns.sourceforge.net/

Old project:

http://pydns.sourceforge.net/ (no update since 11 years? blocking?)

Original issue reported on code.google.com by victor.s...@gmail.com on 6 Mar 2014 at 4:47

[1st1]

I want to resurrect this discussion. It would be really cool if we can have an API for setting custom DNS resolvers in asyncio in 3.5.

Writing a pure Python DNS client is a great idea, but it's not an easy feat. This can be done separately from asyncio if it has the API to plug such implementation in.

[gvanrossum]

Well, a DNS client is not what we need. We'd need a DNS client plus a cache plus a way to consult the system resolver plus who knows what else... Do you have an actual use case and a suitable implementation, or is this just a completeness dream?

[1st1]

I kind of liked Victor's patch: https://codereview.appspot.com/72270043/patch/20001/30001

Specifically, loop.set_resolver() method. Isn't cache something that resolvers can implement themselves?

[1st1]

So the actual use case it to: 1) try to integrate c-ares, 2) try to implement a cache policy for it. OTOH, this experimentation is something that is possible to do with subclassing.

[kaniini]

hi,

blindly calling getaddrinfo(3) is a really bad design choice, and is considered a bad pattern in C where it originates from. at the very least, a better approach would be to call getaddrinfo(3) when DNS is actually required (i.e. inet_pton(3) fails).

i think that adding an API to override this behaviour would be even better, consider that a +1000 from me, because it's really the only bad aspect of asyncio i've encountered.

[1st1]

@kaniini

blindly calling getaddrinfo(3) is a really bad design choice, and is considered a bad pattern in C where it originates from. at the very least, a better approach would be to call getaddrinfo(3) when DNS is actually required (i.e. inet_pton(3) fails).

That's what we already do, see https://github.com/python/asyncio/blob/master/asyncio/base_events.py#L90

[kaniini]

it is inconsistent, https://github.com/python/asyncio/blob/master/asyncio/base_events.py#L578-L593 - these functions are called by other dependencies such as asyncio.open_connection(). it would be desirable to ensure that there is consistency in what is done, i think. shipping off thousands of getaddrinfo(3) calls to a thread pool when the code is connecting to a well-defined IP address results in significant CPU overhead.

[1st1]

I'm sorry, what you're saying doesn't make any sense to me.

Current loop.getaddrinfo (in asyncio master branch) checks if the address is an IP address string, and if it is, it does not use threadpools or system getaddrinfo at all.

[kaniini]

oh, so it does. that's great! because on 3.5 it doesn't. :(

[gvanrossum]

That'll be fixed once 3.5.2 is out.

[graingert]

There's http://man7.org/linux/man-pages/man3/getaddrinfo_a.3.html but libuv doesn't like it: joyent/libuv#617

[kaniini]

getaddrinfo_a() just spawns a thread to do the DNS resolution.

the most ideal approach would be to allow installing a new set of DNS functions, allowing for a true async resolver to be used, i.e. loop.set_resolver() or similar, as discussed earlier in this issue.