From 6e291ab1638fd296dbf2d1b23dddd078323d4b5f Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Tue, 12 May 2020 15:57:21 -0400 Subject: [PATCH 1/4] stream rfc draft --- rfc-drafts/stream.md | 186 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 186 insertions(+) create mode 100644 rfc-drafts/stream.md diff --git a/rfc-drafts/stream.md b/rfc-drafts/stream.md new file mode 100644 index 00000000..92a5d0fd --- /dev/null +++ b/rfc-drafts/stream.md @@ -0,0 +1,186 @@ +- Feature Name: `async_stream` +- Start Date: 2020-05-13 +- RFC PR: [rust-lang/rfcs#0000](https://github.com/rust-lang/rfcs/pull/0000) +- Rust Issue: [rust-lang/rust#0000](https://github.com/rust-lang/rust/issues/0000) + +# Summary +[summary]: #summary + +Introduce the `Stream` trait into the standard library, using the +design from `futures`. Redirect the futures-stream definition to the +standard library. + +# Motivation +[motivation]: #motivation + +XXX describe async streams + +discuss also some of the design goals + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + +A "stream" is the async version of an iterator. The `Stream` trait +matches the definition of an [iterator], except that the `next` method +is defined to "poll" for the next item. In other words, where the +`next` method on an iterator simply computes (and returns) the next +item in the sequence, the `poll_next` method on stream asks if the +next item is ready. If so, it will be returned, but otherwise +`poll_next` will return [`Poll::pending`]. Just as with a [`Future`], +returning [`Poll::pending`] implies that the stream has arranged for +the current task to be re-awoken when the data is ready. + +[iterator]: https://doc.rust-lang.org/std/iter/trait.Iterator.html +[`Future`]: https://doc.rust-lang.org/std/future/trait.Future.html +[`Poll::pending`]: https://doc.rust-lang.org/std/task/enum.Poll.html#variant.Pending + +```rust +pub trait Stream { + type Item; + + fn poll_next(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll>; + + #[inline] + fn size_hint(&self) -> (usize, Option) { + (0, None) + } +} +``` + +The arguments to `poll_next` match that of the [`Future::poll`] method: + +* The self must be a pinned reference, ensuring both unique access to + the stream and that the stream value itself will not move. Pinning + allows the stream to save pointers into itself when it suspends, + which will be required to support generator syntax at some point. +* The [context] `cx` defines details of the current task. In particular, + it gives access to the [`Waker`] for the task, which will allow the + task to be re-awoken once data is ready. + +[`Future::poll`]: https://doc.rust-lang.org/std/future/trait.Future.html#tymethod.poll +[pinned]: https://doc.rust-lang.org/std/pin/struct.Pin.html +[context]: https://doc.rust-lang.org/std/task/struct.Context.html +[Waker]: https://doc.rust-lang.org/std/task/struct.Waker.html + +# Reference-level explanation +[reference-level-explanation]: #reference-level-explanation + +This section goes into details about various aspects of the design and +why they ended up the way they did. + +## Why use a `poll` method? + +An alternative design for the stream trait would be to have a trait +that defines an async `next` method: + +```rust +trait Stream { + type Item; + + async fn next(&mut self) -> Option; +} +``` + +Unfortunately, async methods in traits are not currently supported, +and there [are a number of challenges to be +resolved](https://rust-lang.github.io/wg-async-foundations/design_notes/async_fn_in_traits.html) +before they can be added. + +Moreover, it is not clear yet how to make traits that contain async +functions be `dyn` safe, and it is imporant to be able to pass around `dyn +Stream` values without the need to monomorphize the functions that work +with them. + +Unfortunately, the use of poll does mean that it is harder to write +stream implementations. + +## What about combinators? + +The `Iterator` trait defines a number of useful combinators, like +`map`. The `Stream` trait being proposed here does not include any +such conveniences. Instead, they are available via extension traits, +such as the [`StreamExt`] trait offered by the [`futures`] crate. + +[`StreamExt`]: https://docs.rs/futures/0.3.5/futures/stream/trait.StreamExt.html +[`futures`]: https://crates.io/crates/futures + +The reason that we have chosen to exclude combinators is that a number +of them would require access to async closures. As of this writing, +async closures are unstable and there are a number of [outstanding +design issues] to be resolved before they are added. Therefore, we've +decided to enable progress on the stream trait by stabilizing a core, +and to come back to the problem of extending it with combinators. + +[outstanding design issues]: https://rust-lang.github.io/wg-async-foundations/design_notes/async_closures.html + +This path does carry some risk. Adding combinator methods can cause +existing code to stop compiling due to the ambiguities in method +resolution. We have had problems in the past with attempting to migate +iterator helper methods from `itertools` for this same reason. + +While such breakage is technically permitted by our semver guidelines, +it would obviously be best to avoid it, or at least to go to great +lengths to mitigate its effects. One option would be to extend the +language to allow method resolution to "favor" the extension trait in +existing code, perhaps as part of an edition migration. + +## "Attached" streams + +## Compatibility with future generator syntax + +# Drawbacks +[drawbacks]: #drawbacks + +Why should we *not* do this? + +# Rationale and alternatives +[rationale-and-alternatives]: #rationale-and-alternatives + +- Why is this design the best in the space of possible designs? +- What other designs have been considered and what is the rationale for not choosing them? +- What is the impact of not doing this? + +# Prior art +[prior-art]: #prior-art + +Discuss prior art, both the good and the bad, in relation to this proposal. +A few examples of what this can include are: + +- For language, library, cargo, tools, and compiler proposals: Does this feature exist in other programming languages and what experience have their community had? +- For community proposals: Is this done by some other community and what were their experiences with it? +- For other teams: What lessons can we learn from what other communities have done here? +- Papers: Are there any published papers or great posts that discuss this? If you have some relevant papers to refer to, this can serve as a more detailed theoretical background. + +This section is intended to encourage you as an author to think about the lessons from other languages, provide readers of your RFC with a fuller picture. +If there is no prior art, that is fine - your ideas are interesting to us whether they are brand new or if it is an adaptation from other languages. + +Note that while precedent set by other languages is some motivation, it does not on its own motivate an RFC. +Please also take into consideration that rust sometimes intentionally diverges from common language features. + +# Unresolved questions +[unresolved-questions]: #unresolved-questions + +- What parts of the design do you expect to resolve through the RFC process before this gets merged? +- What parts of the design do you expect to resolve through the implementation of this feature before stabilization? +- What related issues do you consider out of scope for this RFC that could be addressed in the future independently of the solution that comes out of this RFC? + +# Future possibilities +[future-possibilities]: #future-possibilities + +Think about what the natural extension and evolution of your proposal would +be and how it would affect the language and project as a whole in a holistic +way. Try to use this section as a tool to more fully consider all possible +interactions with the project and language in your proposal. +Also consider how the this all fits into the roadmap for the project +and of the relevant sub-team. + +This is also a good place to "dump ideas", if they are out of scope for the +RFC you are writing but otherwise related. + +If you have tried and cannot think of any future possibilities, +you may simply state that you cannot think of anything. + +Note that having something written down in the future-possibilities section +is not a reason to accept the current or a future RFC; such notes should be +in the section on motivation or rationale in this or subsequent RFCs. +The section merely provides additional information. From ad409b0270650ae4c40a99fc6b3c1b1b66c1a9f1 Mon Sep 17 00:00:00 2001 From: HackMD Date: Fri, 22 May 2020 15:48:40 +0000 Subject: [PATCH 2/4] expanded outline There is now an outline covering the major missing points. --- rfc-drafts/stream.md | 122 +++++++++++++++++++++++++------------------ 1 file changed, 70 insertions(+), 52 deletions(-) diff --git a/rfc-drafts/stream.md b/rfc-drafts/stream.md index 92a5d0fd..7a2bac89 100644 --- a/rfc-drafts/stream.md +++ b/rfc-drafts/stream.md @@ -13,9 +13,21 @@ standard library. # Motivation [motivation]: #motivation -XXX describe async streams - -discuss also some of the design goals +* Why include stream trait in the std library at all? + * Streams are a core async abstraction + * we want to enable portable libraries that produce/consume streams without being tied to particular executors + * examples of crates that are consuming streams? + * [async-h1](https://docs.rs/async-h1)'s server implementation takes `TcpStream` instances produced by a `TcpListener` in a loop. + * examples of crates that are producing streams? + * [async-sse](https://docs.rs/async-sse/) parses incoming buffers into a stream of messages. + * people can do this today using futures crate, but the stability guarantees are less clear + * e.g., if tokio wishes to declare a [5 year stability period](http://smallcultfollowing.com/babysteps/blog/2020/02/11/async-interview-6-eliza-weisman/#communicating-stability), having something in std means there are no concerns about trait changing during that time ([citation](http://smallcultfollowing.com/babysteps/blog/2019/12/23/async-interview-3-carl-lerche/#what-should-we-do-next-stabilize-stream)) + * We eventually want dedicated syntax for working with streams, which will require a shared trait + * Producing streams + * Consuming streams +* Why is the stream trait defined how it is? + * It is the "pollable iterator" + * dyn compatibility # Guide-level explanation [guide-level-explanation]: #guide-level-explanation @@ -92,41 +104,7 @@ Stream` values without the need to monomorphize the functions that work with them. Unfortunately, the use of poll does mean that it is harder to write -stream implementations. - -## What about combinators? - -The `Iterator` trait defines a number of useful combinators, like -`map`. The `Stream` trait being proposed here does not include any -such conveniences. Instead, they are available via extension traits, -such as the [`StreamExt`] trait offered by the [`futures`] crate. - -[`StreamExt`]: https://docs.rs/futures/0.3.5/futures/stream/trait.StreamExt.html -[`futures`]: https://crates.io/crates/futures - -The reason that we have chosen to exclude combinators is that a number -of them would require access to async closures. As of this writing, -async closures are unstable and there are a number of [outstanding -design issues] to be resolved before they are added. Therefore, we've -decided to enable progress on the stream trait by stabilizing a core, -and to come back to the problem of extending it with combinators. - -[outstanding design issues]: https://rust-lang.github.io/wg-async-foundations/design_notes/async_closures.html - -This path does carry some risk. Adding combinator methods can cause -existing code to stop compiling due to the ambiguities in method -resolution. We have had problems in the past with attempting to migate -iterator helper methods from `itertools` for this same reason. - -While such breakage is technically permitted by our semver guidelines, -it would obviously be best to avoid it, or at least to go to great -lengths to mitigate its effects. One option would be to extend the -language to allow method resolution to "favor" the extension trait in -existing code, perhaps as part of an edition migration. - -## "Attached" streams - -## Compatibility with future generator syntax +stream implementations. The long-term fix for this, discussed in the [Future possibilities][future-possibilities] section, is dedicated [generator syntax]. # Drawbacks [drawbacks]: #drawbacks @@ -167,20 +145,60 @@ Please also take into consideration that rust sometimes intentionally diverges f # Future possibilities [future-possibilities]: #future-possibilities -Think about what the natural extension and evolution of your proposal would -be and how it would affect the language and project as a whole in a holistic -way. Try to use this section as a tool to more fully consider all possible -interactions with the project and language in your proposal. -Also consider how the this all fits into the roadmap for the project -and of the relevant sub-team. +## Convenience methods + +The `Iterator` trait defines a number of useful combinators, like +`map`. The `Stream` trait being proposed here does not include any +such conveniences. Instead, they are available via extension traits, +such as the [`StreamExt`] trait offered by the [`futures`] crate. + +[`StreamExt`]: https://docs.rs/futures/0.3.5/futures/stream/trait.StreamExt.html +[`futures`]: https://crates.io/crates/futures + +The reason that we have chosen to exclude combinators is that a number +of them would require access to async closures. As of this writing, +async closures are unstable and there are a number of [outstanding +design issues] to be resolved before they are added. Therefore, we've +decided to enable progress on the stream trait by stabilizing a core, +and to come back to the problem of extending it with combinators. + +[outstanding design issues]: https://rust-lang.github.io/wg-async-foundations/design_notes/async_closures.html + +This path does carry some risk. Adding combinator methods can cause +existing code to stop compiling due to the ambiguities in method +resolution. We have had problems in the past with attempting to migrate +iterator helper methods from `itertools` for this same reason. + +While such breakage is technically permitted by our semver guidelines, +it would obviously be best to avoid it, or at least to go to great +lengths to mitigate its effects. One option would be to extend the +language to allow method resolution to "favor" the extension trait in +existing code, perhaps as part of an edition migration. + +Designing such a migration feature is out of scope for this RFC. + +## IntoStream / FromStream traits, mirroring iterators + +* currently blocked on async fn in traits +* The exact bounds are unclear. +* the same as combinators +* These would be needed to provide + +## Async iteration syntax + +We may wish to introduce some dedicated syntax, analogous to `for` + +## Generator syntax +[generator syntax]: #generator-syntax + +```rust +gen async fn foo() -> X { + yield value; +} +``` + +## "Attached" streams -This is also a good place to "dump ideas", if they are out of scope for the -RFC you are writing but otherwise related. +Just as with iterators, there is a -If you have tried and cannot think of any future possibilities, -you may simply state that you cannot think of anything. -Note that having something written down in the future-possibilities section -is not a reason to accept the current or a future RFC; such notes should be -in the section on motivation or rationale in this or subsequent RFCs. -The section merely provides additional information. From a85362591a0d9131a3758fbeffc98b7084f49e5c Mon Sep 17 00:00:00 2001 From: HackMD Date: Fri, 22 May 2020 15:55:15 +0000 Subject: [PATCH 3/4] examples and more --- rfc-drafts/stream.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/rfc-drafts/stream.md b/rfc-drafts/stream.md index 7a2bac89..9ade40d1 100644 --- a/rfc-drafts/stream.md +++ b/rfc-drafts/stream.md @@ -182,7 +182,10 @@ Designing such a migration feature is out of scope for this RFC. * currently blocked on async fn in traits * The exact bounds are unclear. * the same as combinators -* These would be needed to provide +* These would be needed to provide similar iteration semantics as Iterator: + * `for x in iter` uses `impl IntoIterator for T` + * `for x in &iter` uses `impl IntoIterator for &T` + * `for x in &mut iter` uses `impl IntoIterator for &mut T` ## Async iteration syntax From 77b4840544bdbd264f64e3643fad9cec4844f147 Mon Sep 17 00:00:00 2001 From: HackMD Date: Fri, 22 May 2020 16:07:55 +0000 Subject: [PATCH 4/4] mention modules --- rfc-drafts/stream.md | 50 +++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 47 insertions(+), 3 deletions(-) diff --git a/rfc-drafts/stream.md b/rfc-drafts/stream.md index 9ade40d1..b1dba8f8 100644 --- a/rfc-drafts/stream.md +++ b/rfc-drafts/stream.md @@ -47,6 +47,7 @@ the current task to be re-awoken when the data is ready. [`Poll::pending`]: https://doc.rust-lang.org/std/task/enum.Poll.html#variant.Pending ```rust +// Defined in std::stream module pub trait Stream { type Item; @@ -74,12 +75,51 @@ The arguments to `poll_next` match that of the [`Future::poll`] method: [context]: https://doc.rust-lang.org/std/task/struct.Context.html [Waker]: https://doc.rust-lang.org/std/task/struct.Waker.html +## Initial impls + +There are a number of simple "bridge" impls that are also provided: + +```rust +impl Stream for Box +where + S: Stream + Unpin + ?Sized, +{ + type Item = ::Item +} + +impl Stream for &mut S +where + S: Stream + Unpin + ?Sized, +{ + type Item = ::Item; +} + +impl Stream for Pin

+where + P: DerefMut + Unpin, + T::Target: Stream, +{ + type Item = ::Item; +} + +impl Stream for AssertUnwindSafe +where + S: Stream, +{ + type Item = ::Item; +} +``` + # Reference-level explanation [reference-level-explanation]: #reference-level-explanation This section goes into details about various aspects of the design and why they ended up the way they did. +## Where does `Stream` live in the std lib? + +`Stream` will live in the `core::stream` module and be re-exported as `std::stream`. + ## Why use a `poll` method? An alternative design for the stream trait would be to have a trait @@ -114,9 +154,10 @@ Why should we *not* do this? # Rationale and alternatives [rationale-and-alternatives]: #rationale-and-alternatives -- Why is this design the best in the space of possible designs? -- What other designs have been considered and what is the rationale for not choosing them? -- What is the impact of not doing this? +## Where should stream live? + +* core::stream is analogous to core::future +* but do we want to find some other naming scheme that can scale up to other future additions, such as io traits or channels? # Prior art [prior-art]: #prior-art @@ -164,6 +205,9 @@ and to come back to the problem of extending it with combinators. [outstanding design issues]: https://rust-lang.github.io/wg-async-foundations/design_notes/async_closures.html +Another reason to defer adding combinators is because of the possibility +that some combinators may work best + This path does carry some risk. Adding combinator methods can cause existing code to stop compiling due to the ambiguities in method resolution. We have had problems in the past with attempting to migrate