dart:io does not use ipv6 on ubuntu 24.04 unless forced

[noctisbeta]

When making http requests to generativelanguage.googleapis.com Dart always goes through ipv4. For some reason, Hetzner's ipv4 addresses for their VPS are blocked. Going through ipv6 works. Other tools (curl, httpie, python) default to ipv6 (but fail when forced to use ipv4 eg. curl -4). Configuring the OS to always prefer ipv6 over ipv4 does not change the dart:io behaviour. The workaround is manually making a request to google's ipv6 address and handling the certificate failure, since this results in a mismatch. Or using a proxy in a custom http client to circumvent the ipv4 block. (this is speculative, but have found others reporting the same block for their ipv4 from Hetzner)

Surely there must be a cleaner way to tell IOClient or HttpClient to use ipv6 instead of ipv4.

This is using

• Dart 3.7.0 (stable) (None) on "linux_x64"
• on linux / Linux 6.8.0-52-generic # 53-Ubuntu SMP PREEMPT_DYNAMIC Sat Jan 11 00:06:25 UTC 2025
• locale is en_US.UTF-8
  on Ubuntu 24.04.1 LTS.

Edit: Is this the culprit, on line 702? Staggered lookup, which checks ipv4 first.

https://github.com/dart-lang/sdk/blob/main/sdk/lib/_internal/vm/bin/socket_patch.dart

Issue referenced: #50868

Edit 2: Since this looks like a dart wide issue, maybe this should be renamed to account for new information. Dart outright refusing to use ipv6 over ipv4 is in contradiction with other tools and works against expectations. This results in very hard to debug and hard to work with scenarios. That is, if I'm understanding this correctly.

[brianquinlan]

I've been looking at how other languages deal with this:

Java has a command-line flag or you can can call System.setProperty("java.net.preferIPv4Stack", "true"); programatically.
Python has nothing systematic but you can force IPv4 in urllib3 with requests.packages.urllib3.util.connection.HAS_IPV6 = False
Node has a command-line flag e.g. export NODE_OPTIONS=--dns-result-order=ipv4first or you can call dns.setDefaultResultOrder.

We could have a system like Java/Node where we have a command flag and maybe a static property on Socket. Options could be:

1. ipv4first (current behavior)
2. ipv6first
3. ipv4only (do not use ipv6)
4. ipv6only (do not use ipv4)
5. rfc (assume that the DNS server knows what it is doing and use the addresses in the order returned).

This would only apply when connecting to a host by name i.e. if you are connecting via InternetAddress then the family of that address would always be used. Likewise if you are connecting via a string that is interpretable as an address e.g. "127.0.0.1" or "::1".

I think that the awkward part would be testing all of this.

[brianquinlan]

I have a first draft implementation here: https://dart-review.googlesource.com/c/sdk/+/423980/

[mkustermann]

When making http requests to generativelanguage.googleapis.com Dart always goes through ipv4. For some reason, Hetzner's ipv4 addresses for their VPS are blocked. Going through ipv6 works. Other tools (curl, httpie, python) default to ipv6 (but fail when forced to use ipv4 eg. curl -4).

From reading the code, what should happen (if I read it correctly)

• we issue DNS hostname resolution for IPv4
• 10ms later we issue DNS hostname resolution for IPv6
• if we get IPv4 DNS answer before IPv6 we try to connect via IPv4
• if connection via IPv4 fails or takes longer than 250ms we try next address (if it succeeds, we cancel the IPv6 DNS lookup timer)
• if we get IPv6 DNS answer and IPv4 connection wasn't successful (or passed 250ms timeout) we should try IPv6 connection
• the IPv6 connection should then work

So from my reading the code, if the DNS lookups for IPv4 & IPv6 suceed but IPv4 is firewalled, then we should after roughtly 10ms+250ms try to connect via IPv6 which should then succeed.

@noctisbeta What exact behavior do you see? Does the connection take longer to establish, does it fail or does it timeout?

[noctisbeta]

@mkustermann The connection does not fail. It connects over IPv4 normally, then the google api gives an error response returning "unsupported user location" (this is the part where others are suspecting an ipv4 block). The IPv4 does not timeout or take longer. This does not happen over IPv6 (using manual dns lookup and a custom http client adapter), nor does it happen using other tools such as curl, httpie, javascript, python, etc (because they either favor IPv6 or try both at the same time).

The main problem is that dart "forces" you to try IPv4 first. This should not require workarounds as it is expected to work like everywhere else. I have not however looked into the original android bug that brought upon this change. Ideally another fix could be found and this hardcoded delay could be reverted.

[mkustermann]

Ah, so the root cause here is that the dart:io stack works fine and connects to the google servers fine (i.e. TCP connection is established). But the particular google service that is being used examines the client IP and issues an RPC error if the client IP is outside the allowed geographical regions / countries. In your particular case the IPv4 address your provider uses for outgoing traffic falls into a range that the Google service considers as not allowed to access those APIs.

Thanks, that clears clears things up and makes perfect sense!

Given you have this problem, you want a mechanism to force IPv6 when connecting to that particular generativelanguage.googleapis.com host.

This leads me also to conclude that maybe we shouldn't have a global setting telling to favor IPv4 vs IPv6. This use case wants to force IPv6 for that particular hostname. Maybe other API services may get denied if IPv6 is used but not IPv4. So one may want to configure this per hostname instead of globally.

OS configuration

Configuring the OS to always prefer ipv6 over ipv4 does not change the dart:io behaviour.

How do you configure the OS DNS resolution?

I would assume that if one makes DNS resolutions only return IPv6 addresses then this would be solved.
Similarly if all network interfaces have IPv6 address but no IPv4 addresses this may would be solved.

That being said, I'm not advocating for changing the OS settings - as many other use cases it may not even be possible to change OS settings.

Current dart:io and proposed CL

The current dart:io implementation doesn't force IPv4 - it just makes it much more likely to use IPv4 by giving it a 10ms advantage in DNS resolution. Even the proposed change in cl/423980 doesn't really give you what you want (at it's current state): It will allow configuring Socket.nameResolutionPolicy = NameResolutionPolicy.dnsOrder(would try the order it gets from addrinfo OS call - but depending on timing may end up connecting to IPv4 if the IPv6 connection is too slow) or Socket.nameResolutionPolicy = NameResolutionPolicy.ipv4 (would make IPv4 more likely) - but there's no guarantee it will use IPv6.

I do believe that the solution that was implemented in the past for the "IPv6 DNS resolution timing out" problem is not the right one. Instead of hard-coding a preference for IPv4 (by separate IPv4 & IPv6 lookups and delaying IPv6 DNS lookup by 10 ms) we should default to OS behavior (and trigger a 10ms delayed IPv4-only DNS lookup) -- this way we do most likely what OS tells us, only if the DNS lookup is slow/timing-out will we use IPv4 (for the cases where IPv6 DNS lookups would timeout / take seconds - the original issues observed on iOS and Android)

That behavior would probably make this particular issue mostly go away - even though there'd be still no guarantee of IPv6.

Solution for @noctisbeta

@noctisbeta I think for your use case you want to enforce IPv6 for that particular generateivelanguage.googleapis.com hostname (and not rely on some dart:io or OS behavior). Couldn't you do that via HttpClient.connectionFactory:

import 'dart:io';

Future main() async {
  final c = HttpClient();
  c.connectionFactory = (Uri url, String? proxyHost, int? proxyPort) async {
    final List<InternetAddress> addresses;
    if (url.host == 'generativelanguage.googleapis.com') {
      addresses = await InternetAddress.lookup(url.host, type: InternetAddressType.IPv6); // Enforce IPv6
    } else {
      addresses = await InternetAddress.lookup(url.host); // Default to OS lookup
    }
    if (addresses.isEmpty) throw 'DNS resolution for ${url.host} failed.';
    if (url.scheme == 'https') {
      return SecureSocket.startConnect(
          addresses.first, url.hasPort ? url.port : 443);
    } else {
      return Socket.startConnect(addresses.first, url.hasPort ? url.port : 80);
    }
  };

  final req =
      await c.getUrl(Uri.parse('http://generativelanguage.googleapis.com/foo'));
  print('HTTP remote address ${req.connectionInfo?.remoteAddress}');
  final res = await req.close();
  print('HTTP status: ${res.statusCode}');
  await res.drain();
  c.close();
}

Would that work for you?

[noctisbeta]

@mkustermann I tried setting priorities for linux's getaddrinfo function by editing /etc/gai.conf. I did this since if the IPv4 of my VPS really was on some sort of blacklist, this would make sure I always prefer IPv6, for other APIs as well. No matter how much I changed the values in gai.conf, dart:io wouldn't give me an IPv6. Maybe an error on my end though. From what I understand, this only changes the order in the list that is returned by the OS getaddrinfo (putting IPv6 in before IPv4), but then dart filters for IPv4 by doing a lookup for InternetAddressType.IPv4 first (because of the delay) so it doesn't matter. I assumed that if it looked up InternetAddressType.any and the OS was configured to return IPv6 first, it would get the IPv6. So this just added to the confusion before I found the delay in the socket patch.

In the end I tried a very similar solution to what you've suggested, using custom adapters, to make a workaround. I still wanted to open up an issue, even though my very specific IPv4 google api problem was resolved using manual lookup, because it took a very long time to figure out what was actually going wrong. I did not expect the delayed lookup code to exist, and even changing the OS didn't force it, it was quite a rabbit hole.

Thank you for providing an example for the solution. I believe this will make it easier for others to find out the cause of this issue and make it work.

[mkustermann]

Glad to hear you have a way to make this work for you, @noctisbeta .

And I'm in agreement that dart:io should try the list in same way the OS returns it in getaddrinfo by default and only resort to IPv4 specifically if the former DNS lookup times out (probably a very rare case when this fallback is needed). So maybe this is the only thing we should do @brianquinlan instead of exposing a global setting in dart:io?