Tailscale is slow: `wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument`

[marek22k]

What is the issue?

Hello,

I have a problem. For a few days now, Tailscale has been extremely slow and the following message keeps appearing in the log:

Aug 05 23:40:47 ciel tailscaled[876]: wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument
Aug 05 23:40:47 ciel tailscaled[876]: write /dev/net/tun: invalid argument
Aug 05 23:40:47 ciel tailscaled[876]: write /dev/net/tun: invalid argument
Aug 05 23:40:47 ciel tailscaled[876]: write /dev/net/tun: invalid argument
Aug 05 23:40:47 ciel tailscaled[876]: wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument
Aug 05 23:40:47 ciel tailscaled[876]: wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument
Aug 05 23:40:47 ciel tailscaled[876]: wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument
Aug 05 23:40:47 ciel tailscaled[876]: wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument
Aug 05 23:40:47 ciel tailscaled[876]: [RATELIMIT] format("wg: Failed to write packets to TUN device: %v")

Aug 05 23:46:09 ciel tailscaled[876]: [RATELIMIT] format("wg: Failed to write packets to TUN device: %v") (30 dropped)
Aug 05 23:46:09 ciel tailscaled[876]: wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument
Aug 05 23:46:09 ciel tailscaled[876]: write /dev/net/tun: invalid argument
Aug 05 23:46:09 ciel tailscaled[876]: write /dev/net/tun: invalid argument
Aug 05 23:46:09 ciel tailscaled[876]: wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument
Aug 05 23:46:09 ciel tailscaled[876]: wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument
Aug 05 23:46:09 ciel tailscaled[876]: wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument
Aug 05 23:46:09 ciel tailscaled[876]: wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument
Aug 05 23:46:09 ciel tailscaled[876]: [RATELIMIT] format("wg: Failed to write packets to TUN device: %v")

By slow I mean the following behavior (iperf3 is running on my tailscale exit node):

$ iperf3 --client vpn-lu.uaru-vimba.ts.net -R
Connecting to host vpn-lu.uaru-vimba.ts.net, port 5201
Reverse mode, remote host vpn-lu.uaru-vimba.ts.net is sending
[  5] local 100.97.92.115 port 44816 connected to 100.110.144.38 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   2.00-3.00   sec   128 KBytes  1.05 Mbits/sec                  
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   4.00-5.00   sec   128 KBytes  1.05 Mbits/sec                  
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   7.00-8.00   sec   128 KBytes  1.05 Mbits/sec                  
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   9.00-10.00  sec   128 KBytes  1.05 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.93  sec   609 KBytes   456 Kbits/sec  282             sender
[  5]   0.00-10.00  sec   512 KBytes   419 Kbits/sec                  receiver

iperf Done.

and for reference the other way around is works:

$ iperf3 --client vpn-lu.uaru-vimba.ts.net
Connecting to host vpn-lu.uaru-vimba.ts.net, port 5201
[  5] local 100.97.92.115 port 39542 connected to 100.110.144.38 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  2.50 MBytes  21.0 Mbits/sec    1   71.4 KBytes       
[  5]   1.00-2.00   sec  3.62 MBytes  30.4 Mbits/sec    0   99.3 KBytes       
[  5]   2.00-3.00   sec  3.88 MBytes  32.5 Mbits/sec    1   87.2 KBytes       
[  5]   3.00-4.00   sec  3.75 MBytes  31.5 Mbits/sec    1    102 KBytes       
[  5]   4.00-5.00   sec  3.88 MBytes  32.5 Mbits/sec    0    104 KBytes       
[  5]   5.00-6.00   sec  3.75 MBytes  31.5 Mbits/sec    1   93.2 KBytes       
[  5]   6.00-7.00   sec  4.00 MBytes  33.6 Mbits/sec    1   83.6 KBytes       
[  5]   7.00-8.00   sec  3.88 MBytes  32.5 Mbits/sec    0    111 KBytes       
[  5]   8.00-9.00   sec  3.62 MBytes  30.4 Mbits/sec    1   96.9 KBytes       
[  5]   9.00-10.00  sec  3.88 MBytes  32.5 Mbits/sec    1   84.8 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  36.8 MBytes  30.8 Mbits/sec    7             sender
[  5]   0.00-10.93  sec  36.5 MBytes  28.0 Mbits/sec                  receiver

iperf Done.

Steps to reproduce

No idea. I undid all the changes I made to tailscale (changing the systemd unit and disabling the support logs) for testing. It did not help.

OS

Arch Linux

OS version

$ uname -a
Linux ciel 6.6.44-2-lts #1 SMP PREEMPT_DYNAMIC Sat, 03 Aug 2024 18:10:33 +0000 x86_64 GNU/Linux

Tailscale version

$ tailscale version
1.70.0
  tailscale commit: 0e0a212418fbf8243cb3f06634367b61e81ea9db-dirty
  go version: go1.22.5

Bug report

$ tailscale bugreport
BUG-c75d50b30c42938e61b3d6dfd8683ea7f95df6373200ac4dafdee5700dbabca1-20240805234440Z-89d5c5a39dee5ae1

Additional information

Installed via pacman (tailscale package).

I also run a few other things like cjdns, Yggdrasil, libvirt and docker (-> tailscale is not the only tun interface) on the computer:

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: enp1s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 74:5d:22:f4:8d:6b brd ff:ff:ff:ff:ff:ff
3: wlp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 7e:1b:d9:a1:a2:d7 brd ff:ff:ff:ff:ff:ff permaddr 04:7b:cb:c7:41:10
    inet 192.168.41.[redacted]/24 brd 192.168.41.255 scope global dynamic noprefixroute wlp2s0
       valid_lft 42776sec preferred_lft 42776sec
    inet6 2a02:3100:[redacted]/64 scope global dynamic noprefixroute 
       valid_lft 232740sec preferred_lft 146340sec
    inet6 fd5f:28a6:[redacted]/64 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::[redacted]/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
4: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1304 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none 
    inet6 fcf6:773c:[redacted]/8 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::8f97:eaaa:dd2:ee16/64 scope link stable-privacy proto kernel_ll 
       valid_lft forever preferred_lft forever
5: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:b2:fa:7f brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.1/24 brd 192.168.100.255 scope global virbr0
       valid_lft forever preferred_lft forever
    inet6 fd6c:ffa:eb7a:74e2::1/64 scope global tentative 
       valid_lft forever preferred_lft forever
6: tailscale0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none 
    inet 100.97.92.115/32 scope global tailscale0
       valid_lft forever preferred_lft forever
    inet6 fd7a:115c:a1e0::8e21:5c73/128 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::7986:2c09:c665:1b26/64 scope link stable-privacy proto kernel_ll 
       valid_lft forever preferred_lft forever
7: tun1: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 65535 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none 
    inet6 200:[redacted]/7 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::b56c:e008:2405:5b50/64 scope link stable-privacy proto kernel_ll 
       valid_lft forever preferred_lft forever
9: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:70:f9:2d:b1 brd ff:ff:ff:ff:ff:ff
    inet 192.168.128.1/24 brd 192.168.128.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 dd64:6f63:6b65:72::1/80 scope global tentative 
       valid_lft forever preferred_lft forever

strace (when doing iperf3):

[pid 15854] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\3@\0@\6\260\213dn\220&da\\s\24Q"..., 1290) = 1290
[pid 15858] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\4@\0@\6\260\212dn\220&da\\s\24Q"..., 1290) = 1290
[pid 15858] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\5@\0@\6\260\211dn\220&da\\s\24Q"..., 1290) = 1290
[pid 15858] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\6@\0@\6\260\210dn\220&da\\s\24Q"..., 1290) = 1290
[pid 15826] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\7@\0@\6\260\207dn\220&da\\s\24Q"..., 1290) = 1290
[pid 15864] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\10@\0@\6\260\206dn\220&da\\s\24Q"..., 1290) = 1290
[pid 15826] write(11, "\1\1(\0\330\4\24\0\20\0E\0\t\330\320\t@\0@\6\253\255dn\220&da\\s\24Q"..., 2530) = -1 EINVAL (Invalid argument)
[pid 15826] write(11, "\1\1(\0\330\4\24\0\20\0E\0\t\330\320\v@\0@\6\253\253dn\220&da\\s\24Q"..., 2530) = -1 EINVAL (Invalid argument)
[pid 15826] write(11, "\1\1(\0\330\4\24\0\20\0E\0\t\330\320\r@\0@\6\253\251dn\220&da\\s\24Q"..., 2530) = -1 EINVAL (Invalid argument)
[pid 15826] write(11, "\1\1(\0\330\4\24\0\20\0E\0\t\330\320\17@\0@\6\253\247dn\220&da\\s\24Q"..., 2530) = -1 EINVAL (Invalid argument)
[pid 15864] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\21@\0@\6\260}dn\220&da\\s\24Q"..., 1290) = 1290
[pid 15858] write(11, "\1\1(\0\330\4\24\0\20\0E\0\t\330\320\22@\0@\6\253\244dn\220&da\\s\24Q"..., 2530) = -1 EINVAL (Invalid argument)
[pid 15864] write(11, "\1\1(\0\330\4\24\0\20\0E\0\t\330\320\24@\0@\6\253\242dn\220&da\\s\24Q"..., 2530) = -1 EINVAL (Invalid argument)
[pid 15864] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\26@\0@\6\260xdn\220&da\\s\24Q"..., 1290) = 1290
[pid 15858] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\27@\0@\6\260wdn\220&da\\s\24Q"..., 1290) = 1290
[pid 15858] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\30@\0@\6\260vdn\220&da\\s\24Q"..., 1290) = 1290
[pid 15826] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\31@\0@\6\260udn\220&da\\s\24Q"..., 1290) = 1290
[pid 15854] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\32@\0@\6\260tdn\220&da\\s\24Q"..., 1290) = 1290
[pid 15854] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\33@\0@\6\260sdn\220&da\\s\24Q"..., 1290) = 1290

If needed, I can also send the output of sudo strace -f -o strace.log tailscaled -verbose 1 > tailscale.log 2>&1 privately.

Another observation that might be interesting: I have a second laptop with pretty much the same setup (Arch Linux + Tailscale). I hadn't updated it for a few days / weeks, so kernel 6.6.35-2-lts was still installed. Tailscale worked there. I have now updated it to kernel 6.6.44-2-lts and since then the same problem occurs.

[raggi]

Does this reproduce for you on linux-mainline?

cc @jwhited

[marek22k]

The problem is also under 6.10.3-arch1-1, whether it is also under linux-mainline I have to see (assuming my computer manages to download and build the AUR package).

[jwhited]

[pid 15864] write(11, "\0\0\0\0\0\0\0\0\0\0E\0\5\0\320\10@\0@\6\260\206dn\220&da\\s\24Q"..., 1290) = 1290
[pid 15826] write(11, "\1\1(\0\330\4\24\0\20\0E\0\t\330\320\t@\0@\6\253\255dn\220&da\\s\24Q"..., 2530) = -1 EINVAL (Invalid argument)

The writes towards the TUN appear to return EINVAL when length is greater than MSS/MTU, i.e. TCP or UDP GRO is involved. Going by the fact you're running TCP iperf3, I'm going to assume TCP.

The virtio_net_hdr values at the front look well-formed at first glance: needs_csum, GSO_TCPv4, hdr len 51 (this is a little unexpected), gso size 1060, csum start 32...

This is the first EINVAL report I'm aware of involving TCP GSO on the TUN. We've tested up to 6.8.5 and haven't seen this.

Can you provide a pcap of the first 128 bytes from each packet when you run iperf3? sudo tcpdump -s 128 -w /tmp/tailscale0.pcap -nnni tailscale0 tcp

Is there anything on the system that could be modifying the tailscale TUN config? Perhaps picking the wrong FD. If you stop the other software that creates a TUN, and restart tailscale, does the problem still occur?

[simon--taylor]

I also experienced this problem on Arch Linux with linux 6.10.3.arch1-1 and Tailscale version 1.70.0. Downgrading the kernel to 6.10.2-arch1-2 resolved the issue.

[marek22k]

The problem is also under 6.10.3-arch1-1, whether it is also under linux-mainline I have to see (assuming my computer manages to download and build the AUR package).

Unfortunately I can't compile the AUR package - even GitHub Actions can't do it: https://github.com/marek22k/linux-mainline-builder/actions/runs/10259112100/job/28383031908

The writes towards the TUN appear to return EINVAL when length is greater than MSS/MTU, i.e. TCP or UDP GRO is involved. Going by the fact you're running TCP iperf3, I'm going to assume TCP.

Yes. As far as I understand it, iperf3 uses TCP (in the default setting which I uses).

Can you provide a pcap of the first 128 bytes from each packet when you run iperf3? sudo tcpdump -s 128 -w /tmp/tailscale0.pcap -nnni tailscale0 tcp

tailscale0.pcap.txt (remove the .txt as GitHub doesn't support uploading .pcap files)

Is there anything on the system that could be modifying the tailscale TUN config? Perhaps picking the wrong FD. If you stop the other software that creates a TUN, and restart tailscale, does the problem still occur?

If I stop the other TUN programs, it still does not work. Apart from that, I have not (as far as I know) made any changes relevant to TUN adapters.

[marek22k]

I also experienced this problem on Arch Linux with linux 6.10.3.arch1-1 and Tailscale version 1.70.0. Downgrading the kernel to 6.10.2-arch1-2 resolved the issue.

Downgrading the kernel to 6.6.35-2-lts also solved the problem for me.

Edit: Doesn't work on 6.6.44-2-lts, 6.6.44-1-lts; works on 6.6.43-1-lts, 6.6.42-1-lts.

[raggi]

I was able to partially reproduce this, specifically:

• Using arch LTS Linux archlinux 6.6.44-2-lts #1 SMP PREEMPT_DYNAMIC Sat, 03 Aug 2024 18:10:33 +0000 x86_64 GNU/Linux
• Observed iperf3 resulting in wg: Failed to write packets to TUN device: write /dev/net/tun: invalid argument
• Observed in dmesg: [ 96.578967] ens18: bad gso: type: 4, size: 1348

This appears to be scoped to IPv6.

I was not however able to reproduce this occurring persistently, in my test setup the issue certainly occurred, but it occurred for a short time and then throughput recovered, and the error stopped being logged, both on the tailscaled side and kernel side.

I was not able to reproduce on Linux archlinux 6.10.3-arch1-1 #1 SMP PREEMPT_DYNAMIC Sun, 04 Aug 2024 05:11:32 +0000 x86_64 GNU/Linux.

We will need to dig into what patches have been ported into LTS, there have been some activities in virtio header handling which we're initially now concerned about, but upstream has already worked on improvements there, it's entirely possible LTS brought in a bug that needs to be addressed with further backports.

[raggi]

there does not appear to be anything relevant in https://gitlab.archlinux.org/archlinux/packaging/packages/linux-lts recent history, or local patches

[raggi]

After some digging, so far it looks likely that the culprit is gregkh/linux@e269d79 and the fix is gregkh/linux@89add40

Next up, verify this.

[raggi]

We have a potential mitigation for this issue in our wireguard-go which appears to avoid the issue in my lab environment, however as the issue was not triggering so consistently there, it would be great to receive further confirmation.

If you are able to do so, testing the patch linked just above this comment would be great, a command based approach would be:

go install tailscale.com/cmd/tailscaled@ab9d3474d400590c53863f6d69f6e2bd98fe2d90
sudo install ~/go/bin/tailscaled `which tailscaled`
sudo systemctl restart tailscaled