Skip to content

Commit 8ce4da6

Browse files
authored
Clarify documentation about use of tokio tasks (#13474)
1 parent 963e8af commit 8ce4da6

File tree

1 file changed

+8
-4
lines changed

1 file changed

+8
-4
lines changed

datafusion/core/src/lib.rs

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -475,10 +475,12 @@
475475
//!
476476
//! The number of cores used is determined by the `target_partitions`
477477
//! configuration setting, which defaults to the number of CPU cores.
478-
//! During execution, DataFusion creates this many distinct `async` [`Stream`]s and
479-
//! this many distinct [Tokio] [`task`]s, which drive the `Stream`s
480-
//! using threads managed by the `Runtime`. Many DataFusion `Stream`s perform
481-
//! CPU intensive processing.
478+
//! While preparing for execution, DataFusion tries to create this many distinct
479+
//! `async` [`Stream`]s for each `ExecutionPlan`.
480+
//! The `Stream`s for certain `ExecutionPlans`, such as as [`RepartitionExec`]
481+
//! and [`CoalescePartitionsExec`], spawn [Tokio] [`task`]s, that are run by
482+
//! threads managed by the `Runtime`.
483+
//! Many DataFusion `Stream`s perform CPU intensive processing.
482484
//!
483485
//! Using `async` for CPU intensive tasks makes it easy for [`TableProvider`]s
484486
//! to perform network I/O using standard Rust `async` during execution.
@@ -582,6 +584,8 @@
582584
//! [`Runtime`]: tokio::runtime::Runtime
583585
//! [`task`]: tokio::task
584586
//! [Using Rustlang’s Async Tokio Runtime for CPU-Bound Tasks]: https://thenewstack.io/using-rustlangs-async-tokio-runtime-for-cpu-bound-tasks/
587+
//! [`RepartitionExec`]: physical_plan::repartition::RepartitionExec
588+
//! [`CoalescePartitionsExec`]: physical_plan::coalesce_partitions::CoalescePartitionsExec
585589
//!
586590
//! ## State Management and Configuration
587591
//!

0 commit comments

Comments
 (0)