Skip to content

Improve the documentation. #509

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Dec 4, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion docs/source/explanations/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@ systems in general as well as its design.
maxdepth: 1
---
why_pytask
interfaces_for_dependencies_products
comparison_to_other_tools
pluggy
```
30 changes: 0 additions & 30 deletions docs/source/explanations/interfaces_for_dependencies_products.md

This file was deleted.

2 changes: 1 addition & 1 deletion docs/source/how_to_guides/functional_interface.ipynb

Large diffs are not rendered by default.

31 changes: 15 additions & 16 deletions docs/source/how_to_guides/how_to_write_a_plugin.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,41 +32,40 @@ This section explains some steps which are required for all plugins.

pytask discovers plugins via `setuptools` entry-points. Following the approach advocated
for by [setuptools_scm](https://github.com/pypa/setuptools_scm), the entry-point is
specified in `setup.cfg`.
specified in `pyproject.toml`.

```cfg
# Content of setup.cfg
```toml
[project]
name = "pytask-plugin"

[metadata]
name = pytask-plugin
[tool.setuptools.package-dir]
"" = "src"

[options.packages.find]
where = src
[tool.setuptools.packages.find]
where = ["src"]
namespaces = false

[options.entry_points]
pytask =
pytask_plugin = pytask_plugin.plugin
[project.entry-points.pytask]
pytask_plugin = "pytask_plugin.plugin"
```

For `setuptools_scm` you also need a `pyproject.toml` with the following content.
For `setuptools_scm` you also need the following additions in `pyproject.toml`.

```toml
# Content of pyproject.toml

[build-system]
requires = ["setuptools>=45", "wheel", "setuptools_scm[toml]>=6.0"]

[tool.setuptools_scm]
write_to = "src/pytask_plugin/_version.py"
```

For a complete example with `setuptools_scm` and `setup.cfg` see the
[pytask-parallel repo](https://github.com/pytask-dev/pytask-parallel/blob/main/setup.cfg).
For a complete example with `setuptools_scm` and `pyproject.toml` see the
[pytask-parallel repo](https://github.com/pytask-dev/pytask-parallel/blob/main/pyproject.toml).

The entry-point for pytask is called `"pytask"` and points to a module called
`pytask_plugin.plugin`.

### plugin.py
### `plugin.py`

`plugin.py` is the entrypoint for pytask to your package. You can put all of your hook
implementations in this module, but it is recommended to imitate the structure of pytask
Expand Down
1 change: 1 addition & 0 deletions docs/source/how_to_guides/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ specific tasks with pytask.
maxdepth: 1
---
migrating_from_scripts_to_pytask
interfaces_for_dependencies_products
functional_interface
capture_warnings
how_to_influence_build_order
Expand Down
114 changes: 114 additions & 0 deletions docs/source/how_to_guides/interfaces_for_dependencies_products.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
# Interfaces for dependencies and products

Different interfaces exist for dependencies and products, and it might not be obvious
when to use what. This guide gives you an overview of the different strengths of each
approach.

## Legend

- ✅ = True
- ❌ = False
- ➖ = Does not apply

## Dependencies

In general, pytask regards everything as a task dependency if it is not marked as a
product. Thus, you can also think of the following examples as how to inject values into
a task. When we talk about products later, the same interfaces will be used.

| | `def task(arg: ... = ...)` | `Annotated[..., value]` | `@task(kwargs=...)` | `@pytask.mark.depends_on(...)` |
| --------------------------------------- | :------------------------: | :---------------------: | :-----------------: | :----------------------------: |
| Not deprecated | ✅ | ✅ | ✅ | ❌ |
| No type annotations required | ✅ | ❌ | ✅ | ✅ |
| Flexible choice of argument name | ✅ | ✅ | ✅ | ❌ |
| Supports third-party functions as tasks | ❌ | ❌ | ✅ | ❌ |

(default-argument)=

### Default argument

You can pass a value to a task as a default argument.

```{literalinclude} ../../../docs_src/how_to_guides/interfaces/dependencies_default.py
```

(annotation)=

### Annotation with value

It is possible to include the value in the type annotation.

It is especially helpful if you pass a {class}`~pytask.PNode` to the task. If you passed
a node as the default argument, type checkers like mypy would expect the node to enter
the task, but the value injected into the task depends on the nodes
{meth}`~pytask.PNode.load` method. For a {class}`~pytask.PathNode`

```{literalinclude} ../../../docs_src/how_to_guides/interfaces/dependencies_annotation.py
```

(task-kwargs)=

### `@task(kwargs=...)`

You can use the `kwargs` argument of the {func}`@task <pytask.task>` decorator to pass a
dictionary. It applies to dependencies and products alike.

```{literalinclude} ../../../docs_src/how_to_guides/interfaces/dependencies_task_kwargs.py
```

## Products

| | `def task(arg: Annotated[..., Product] = ...)` | `Annotated[..., value, Product]` | `produces` | `@task(produces=...)` | `def task() -> Annotated[..., value]` | `@pytask.mark.produces(...)` |
| --------------------------------------------------------- | :--------------------------------------------: | :------------------------------: | :--------: | :-------------------: | :-----------------------------------: | :--------------------------: |
| Not deprecated | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ |
| No type annotations required | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
| Flexible choice of argument name | ✅ | ✅ | ❌ | ✅ | ➖ | ❌ |
| Supports third-party functions as tasks | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ |
| Allows to pass custom node while preserving type of value | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ |

### `Product` annotation

The syntax is the same as {ref}`default-argument`, but the {class}`~pytask.Product`
annotation turns the argument into a task product.

```{literalinclude} ../../../docs_src/how_to_guides/interfaces/products_annotation.py
```

### `Product` annotation with value

The syntax is the same as {ref}`annotation`, but the {class}`~pytask.Product` annotation
turns the argument into a task product.

```{literalinclude} ../../../docs_src/how_to_guides/interfaces/products_annotation_with_pnode.py
```

### `produces`

Without using any type annotation, you can use `produces` as a magical argument name to
treat every value passed to it as a task product.

```{literalinclude} ../../../docs_src/how_to_guides/interfaces/products_produces.py
```

(return-annotation)=

### Return annotation

You can also add a node or a value that will be parsed to a node to the annotation of
the return type. It allows us to treat the returns of the task function as products.

```{literalinclude} ../../../docs_src/how_to_guides/interfaces/products_return_annotation.py
```

(task-produces)=

### `@task(produces=...)`

In situations where the task return is the product like {ref}`return-annotation`, but
you cannot modify the type annotation of the return, use the argument `produces` of the
{func}`@task <pytask.task>` decorator.

Pass the node or value you otherwise include in the type annotation to `produces`.

```{literalinclude} ../../../docs_src/how_to_guides/interfaces/products_task_produces.py
```
8 changes: 4 additions & 4 deletions docs/source/how_to_guides/writing_custom_nodes.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,18 +9,18 @@ your own to improve your workflows.

## Use-case

A typical task operation is to load data like a {class}`pandas.DataFrame` from a pickle
A typical task operation is to load data like a {class}`~pandas.DataFrame` from a pickle
file, transform it, and store it on disk. The usual way would be to use paths to point
to inputs and outputs and call {func}`pandas.read_pickle` and
{meth}`pandas.DataFrame.to_pickle`.
to inputs and outputs and call {func}`~pandas.read_pickle` and
{meth}`~pandas.DataFrame.to_pickle`.

```{literalinclude} ../../../docs_src/how_to_guides/writing_custom_nodes_example_1.py
```

To remove IO operations from the task and delegate them to pytask, we will write a
`PickleNode` that automatically loads and stores Python objects.

And we pass the value to `df` via {obj}`Annotated` to preserve the type hint.
And we pass the value to `df` via {obj}`~typing.Annotated` to preserve the type hint.

The result will be the following task.

Expand Down
Loading