🐢 Eliminate `!Send` resources by supporting `Send` and `!Send` `World` in the same binary

[joshua-holmes]

What problem does this solve or what need does it fill?

This issue is a replacement and extension of #17517, which attempts to solve the same problem by creating an optionally !Send World using generics. The problems with !Send resources listed in that issue are the same issues we are trying to solve here:

NonSend resources are confusing (they don't impl Resource!), badly named (#13481), require API duplication and hamper us when implementing valuable new ECS features like #17485.

That said, not everything we might want to stick in and access from the ECS is threadsafe! Audio, window handles, scripting language integrations and more are all inherently !Send. Simply removing them is not an option. Ultimately, the problem is that our model here is muddy, hampering our ability to teach, reason about and safely improve these areas of code.

Simultaneously, users have requested support for !Send components, particularly for use with web or scripting languages.

Through discussion on Discord and experimentation, we have determined that making World optionally !Send using generics is a bit messy and hacky, and we want a more long-term solution rather than a work-around.

In summary, we need to solve these problems:

1. Eliminate !Send resources so the resources-as-components effort can be unblocked, and...
2. Find a place to store the data from those !Send resources while still supporting their !Send status

What solution would you like?

Short term

Short term, we can replace !Send resources with thread_local!. This will unblock the resources-as-components effort. However, thread_local! requires std, which conflicts with Bevy's support for no_std, so this is not a good long-term solution.

Long term, @bushrat011899 brought up supporting both Send and !Send Worlds in the same binary. This would allow us to store the data that is currently in !Send resources in it's own !Send World as !Send components (since resources-as-components aims to remove the concept of resources altogether). If data needs to be shared from the Send World to the !Send World, @maniwani has suggested an API feature for sending functions to be run on other worlds. Usage might look something like this:

fn send_something_to_foo_world(
    mut worlds: Worlds,
    foo_world: Res<FooWorld>,
) {
    let id: WorldId = *foo_world.0;
    worlds.get_mut(id).block_on(move |mut world| {
        // Something to run on the world with non-send Foo stuff
    });
}

Long term

The long-term plan looks like this:

1. We make non-send resources thread_local! so that resources-as-components is unblocked 🐢 Make !Send resources thread_local! #17682
2. Create inter-thread executor that allows a system to execute arbitrary code on the main thread. This is needed because currently the only way to ensure a System runs on the main thread is to use !Send system params, which is getting removed. We need to ensure any thread_local!s are running on the same thread they're created in. 🐢 Create inter-thread executor API #18172
3. Create official support for Send and !Send World in a single binary a. More details to follow
4. Move thread_local! global vars into a new !Send World as !Send componentsMore issues will be created to break this work out further## What alternative(s) have you considered?
5. Move main World to a thread that is not main thread

Some ideas we have discussed:

1. Only officially supporting Send World (not supporting !Send World) and requiring that users store their own !Send data. However, that would come at the cost of worse UX
2. Only supporting either a Send World or a !Send World in the same binary, but not both simultaneously and using a separate data store for !Send global data to replace our !Send resource usage. However, this would require the creation of a new !Send data store, which can be solved automatically by just creating support for Send and !Send Worlds to co-exist. Imo, it is cleaner to use a !Send World as a !Send data store and doesn't require us to re-engineer something we already have implemented. Plus having a Send and !Send World co-exist in the same binary is more dynamically useful (it's more general and has potential to be more generally useful to end users and/or other bevy teams to solve a variety of other problems).

Why 🐢 turtle emoji?

Because that's the name of the Discord work group that discusses this issue. And turtles are cute.

[maniwani]

We extend the System API to allow configuration of which thread we are going to run the System on. This is needed because currently the only way to ensure a System runs on the main thread is to use !Send system params, which is getting removed. We need to ensure any thread_local!s are running on the same thread they're created in.

My recommendation was to create a channel that any system can use to send work (a function) to a specific receiving thread, and also to create a system param to encapsulate any boilerplate involved.

I don't know if systems themselves should be the thing users pick threads for. I think if we're eliminating the ECS presence of !Send resources, we should also aim to reduce scheduler complexity by eliminating "send-ness" as a variable. It's nicer if all systems are free to run on any thread in the pool.

Also I think sending units of work from inside systems will reduce how long those block the !Send-owning thread (which should be excluded from the thread pool that's visible to the scheduler).

[joshua-holmes]

@maniwani

My recommendation was to create a channel that any system can use to send work (a function) to a specific receiving thread, and also to create a system param to encapsulate any boilerplate involved.

You're right! I was thinking my idea (of extending the System API to allow configuration of which thread the system will run on) would be to unblock my effort to remove !Send resources, not necessarily thinking long term. I was considering your idea for when we setup the long term system of Send and !Send worlds living together in the same binary. Maybe your idea would help here as well and then I only need to implement 1 solution instead of 2. I would need to do some exploring/experimentation to really wrap my head around it though.