internal/poll: transparently support new linux io_uring interface

[johanbrandhorst]

A document on the latest linux IO syscall has made the rounds of discussion forums on the internet:

http://kernel.dk/io_uring.pdf

I wanted to open a discussion on whether we could (and should) add transparent support for this on supported linux kernels.

LWN article: https://lwn.net/Articles/776703/

[ianlancetaylor]

It should be feasible to fit this approach into our current netpoll framework. For some programs I think it would reduce the number of threads doing file I/O. It should potentially reduce the number of system calls required to read from the network.

I'm concerned about coordinating access to the ring. The approach seems designed for high performance communication between the application and the kernel, but it seems easiest to use for an application that uses a single thread for I/O, or in which each I/O thread uses its own ring. In Go of course each goroutine is acting independently, and it seems infeasible for each thread to have a separate ring. so that means that goroutines will need to coordinate their access to the I/O ring. That's fine but on high GOMAXPROCS systems I would worry about contention for the ring, a contention that I think doesn't exist in the current epoll framework.

[ianlancetaylor]

Thinking about this further, it's not clear that it makes sense to use this new interface for network I/O. It seems that it can only handle a fixed number of concurrent I/O requests, and it's going to be quite hard to make that work transparently and efficiently in Go programs. Without knowing what the Go program plans to do, we can't allocate the ring appropriately.

If that is true, we would only use it for file I/O, where we can reasonably delay I/O operations when the ring is full. In that case it would not fit into the netpoll system. Instead, we might in effect send all file I/O requests to a single goroutine which would add them to the ring, while a second goroutine would sleep until events were ready and wake up the goroutine waiting for them. That should limit the number of threads we need for file I/O and reduce the number of system calls.

[CAFxX]

Without knowing what the Go program plans to do, we can't allocate the ring appropriately

I was wondering if we could use the map approach: when it becomes full we allocate a new, bigger one, and start submitting requests to the new one. Once the old one doesn't have pending requests anymore we deallocate it.

it seems easiest to use for an application that uses a single thread for I/O, or in which each I/O thread uses its own ring. In Go of course each goroutine is acting independently, and it seems infeasible for each thread to have a separate ring

Can you elaborate on the "infeasible" part? Assuming having multiple rings is feasible, wouldn't having per-P rings work (with the appropriate fallback slow paths in case the per-P ring is full)? I'm not so familiar with the poller, is the problem that the model mismatch is too big?

[coder543]

If that is true, we would only use it for file I/O

File I/O is the main reason I'm interested in io_uring. My understanding is that the current Linux kernel abstractions for async networking work just fine, but making file I/O appear to be interruptible so that goroutines don't block entire OS threads requires using a pool of I/O OS threads. If io_uring could remove the need for a pool of OS threads for file I/O, that seems to make it worthwhile.

Whether having a ring per P (as @CAFxX suggests) or just having a single thread dedicated to managing a ring... either solution seems fine. Unless there's some measurable advantage to io_uring for networking, I don't think it would be that important to switch the networking code at this point.

[ianlancetaylor]

Using a ring per P seems possible, but it means that when a P is idle some M will have to be sleeping in a io_uring getevent call. And then there will have to be a way to wake up that M if we need it, and some way to avoid confusion if the P gets assigned to a different M. It may be doable but it seems pretty complicated.

[JimChengLin]

I think it is a killer feature of Linux 5.1. libaio(a wrapper of Linux kernel native async file io api) is broken or at least flawed, which cannot do async buffered file io and could block unexpectedly. The author of io_uring claims all problems that libaio has are all solved. It would be great if Go can utilize the new feature seamlessly.

[dahankzter]

Apparently io_uring brings many benefits to networked io as well. Is that not the case or is it just hard to accommodate for the size of the ring?

[ianlancetaylor]

@dahankzter See the discussion above. When I looked at io_uring earlier I did not see how to fit it into the Go runtime for network I/O. I can certainly see how io_uring would work well for some network programs, but only ones that know exactly how many concurrent I/O requests they expect to have in flight. The Go runtime can never know that. This issue is about transparently using io_uring, and I don't see how to do that in Go for network connections. If you see how to do it, please explain. Thanks.

[axboe]

The fixed limit has been removed, and another networking concern was the fixed CQ ring size and the difficulty in sizing that. The latter was fixed with the IORING_FEAT_NODROP support, which guarantees that completion events are never dropped. I think that should take care of most of the concerns?

One thing I've been toying with is the idea of having a shared backend in terms of threads between rings. I think that'd work nicely and would allow you to just setup one ring per thread and not worry too much about overhead.

[ianlancetaylor]

@axboe Thanks for the update. Where is the current API documented?