Merge branch 'main' into pre-commit-ci-update-config

Author: tobiasraabe

.gitignore

@@ -14,7 +14,7 @@ _generated
 *.egg-info
 .eggs
 
-.pytask.sqlite3
+.pytask
 
 build
 dist

.pre-commit-config.yaml

@@ -51,6 +51,7 @@ repos:
     rev: v0.1.3
     hooks:
       - id: ruff
+        args: [--unsafe-fixes]
 -   repo: https://github.com/dosisod/refurb
     rev: v1.22.1
     hooks:
@@ -114,6 +115,7 @@ repos:
                 docs/source/tutorials/repeating_tasks_with_different_inputs.md|
                 docs/source/tutorials/selecting_tasks.md|
                 docs/source/tutorials/set_up_a_project.md|
+                docs/source/tutorials/using_a_data_catalog.md|
                 docs/source/tutorials/write_a_task.md
             )$
 -   repo: https://github.com/nbQA-dev/nbQA

docs/source/_static/md/defining-dependencies-products.md

@@ -0,0 +1,26 @@
+<div class="termy">
+
+```console
+
+$ pytask
+──────────────────────────── Start pytask session ────────────────────────────
+Platform: win32 -- Python <span style="color: var(--termynal-blue)">3.10.0</span>, pytask <span style="color: var(--termynal-blue)">0.4.0</span>, pluggy <span style="color: var(--termynal-blue)">1.0.0</span>
+Root: C:\Users\pytask-dev\git\my_project
+Collected <span style="color: var(--termynal-blue)">2</span> task.
+
+┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┓
+┃ Task                                              ┃ Outcome ┃
+┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━┩
+│ <span class="termynal-dim">task_data_preparation.py::</span>task_create_random_data │ <span class="termynal-success">.</span>       │
+│ <span class="termynal-dim">task_plot_data.py::</span>task_plot_data                 │ <span class="termynal-success">.</span>       │
+└───────────────────────────────────────────────────┴─────────┘
+
+<span class="termynal-dim">──────────────────────────────────────────────────────────────────────────────</span>
+<span class="termynal-success">╭───────────</span> <span style="font-weight: bold;">Summary</span> <span class="termynal-success">────────────╮</span>
+<span class="termynal-success">│</span> <span style="font-weight: bold;"> 2  Collected tasks </span>           <span class="termynal-success">│</span>
+<span class="termynal-success">│</span> <span class="termynal-success-textonly"> 2  Succeeded        (100.0%) </span> <span class="termynal-success">│</span>
+<span class="termynal-success">╰────────────────────────────────╯</span>
+<span class="termynal-success">───────────────────────── Succeeded in 0.06 seconds ──────────────────────────</span>
+```
+
+</div>

docs/source/changes.md

@@ -19,6 +19,9 @@ releases are available on [PyPI](https://pypi.org/project/pytask) and
 - {pull}`463` raise error when a task function is not defined inside the loop body.
 - {pull}`464` improves pinned dependencies.
 - {pull}`465` adds test to ensure internal tracebacks are removed by reports.
+- {pull}`466` implements hashing for files instead of modification timestamps.
+- {pull}`470` moves `.pytask.sqlite3` to `.pytask`.
+- {pull}`472` adds `is_product` to {meth}`PNode.load`.
 
 ## 0.4.1 - 2023-10-11
 

docs/source/conf.py

@@ -82,6 +82,7 @@
 
 intersphinx_mapping = {
     "click": ("https://click.palletsprojects.com/en/8.0.x/", None),
+    "deepdiff": ("https://zepworks.com/deepdiff/current/", None),
     "networkx": ("https://networkx.org/documentation/stable", None),
     "pandas": ("https://pandas.pydata.org/docs", None),
     "pluggy": ("https://pluggy.readthedocs.io/en/latest", None),

docs/source/how_to_guides/bp_scaling_tasks.md

@@ -42,7 +42,8 @@ my_project
 │
 ├───setup.py
 │
-├───.pytask.sqlite3
+├───.pytask
+│   └────...
 │
 └───bld
 ```

docs/source/how_to_guides/functional_interface.ipynb

@@ -62,10 +62,10 @@ from interpreter session to interpreter session for security reasons (see
 ```
 
 {class}`list` and {class}`dict` are not hashable by default. Luckily, there are
-libraries who provide this functionality like `deepdiff`. We can use them to pass a
+libraries who provide this functionality like {mod}`deepdiff`. We can use them to pass a
 function to the {class}`~pytask.PythonNode` that generates a stable hash.
 
-First, install `deepdiff`.
+First, install {mod}`deepdiff`.
 
 ```console
 $ pip install deepdiff

docs/source/how_to_guides/hashing_inputs_of_tasks.md

@@ -19,6 +19,7 @@ hashing_inputs_of_tasks
 using_task_returns
 writing_custom_nodes
 how_to_write_a_plugin
+the_data_catalog
 ```
 
 ## Best Practice Guides

docs/source/how_to_guides/index.md

@@ -0,0 +1,73 @@
+# The `DataCatalog` - Revisited
+
+An introduction to the data catalog can be found in the
+[tutorial](../tutorials/using_a_data_catalog.md).
+
+This guide explains some details that were left out of the tutorial.
+
+## Changing the default node
+
+The data catalog uses the {class}`~pytask.PickleNode` by default to serialize any kind
+of Python object. You can use any other node that follows the {protocol}`~pytask.PNode`
+protocol and register it when creating the data catalog.
+
+For example, use the {class}`~pytask.PythonNode` as the default.
+
+```python
+from pytask import PythonNode
+
+
+data_catalog = DataCatalog(default_node=PythonNode)
+```
+
+Or, learn to write your own node by reading {doc}`writing_custom_nodes`.
+
+Here, is an example for a `PickleNode` that uses cloudpickle instead of the normal
+`pickle` module.
+
+```{literalinclude} ../../../docs_src/how_to_guides/the_data_catalog.py
+```
+
+## Changing the name and the default path
+
+By default, the data catalogs store their data in a directory `.pytask/data_catalogs`.
+If you use a `pyproject.toml` with a `[tool.pytask.ini_options]` section, then the
+`.pytask` folder is in the same folder as the configuration file.
+
+The default name for a catalog is `"default"` and so you will find its data in
+`.pytask/data_catalogs/default`. If you assign a different name like
+`"data_management"`, you will find the data in `.pytask/data_catalogs/data_management`.
+
+```python
+data_catalog = DataCatalog(name="data_management")
+```
+
+You can also change the path where the data catalogs will be stored by changing the
+`path` attribute. Here, we store the data catalog's data next to the module where the
+data catalog is defined in `.data`.
+
+```python
+from pathlib import Path
+
+
+data_catalog = DataCatalog(path=Path(__file__).parent / ".data")
+```
+
+## Multiple data catalogs
+
+You can use multiple data catalogs when you want to separate your datasets across
+multiple catalogs or when you want to use the same names multiple times (although it is
+not recommended!).
+
+Make sure you assign different names to the data catalogs so that their data is stored
+in different directories.
+
+```python
+# Stored in .pytask/data_catalog/a
+data_catalog_a = DataCatalog(name="a")
+
+# Stored in .pytask/data_catalog/b
+data_catalog_b = DataCatalog(name="b")
+```
+
+Or, use different paths as explained above.

docs/source/how_to_guides/the_data_catalog.md

@@ -20,9 +20,6 @@ to inputs and outputs and call {func}`pandas.read_pickle` and
 To remove IO operations from the task and delegate them to pytask, we will write a
 `PickleNode` that automatically loads and stores Python objects.
 
-We will also use the feature explained in {doc}`using_task_returns` to define products
-of the task function via the function's return value.
-
 And we pass the value to `df` via {obj}`Annotated` to preserve the type hint.
 
 The result will be the following task.
@@ -37,12 +34,28 @@ The result will be the following task.
 
 :::
 
+:::{tab-item} Python 3.10+ & Return
+:sync: python310plus
+
+```{literalinclude} ../../../docs_src/how_to_guides/writing_custom_nodes_example_2_py310_return.py
+```
+
+:::
+
 :::{tab-item} Python 3.8+
 :sync: python38plus
 
 ```{literalinclude} ../../../docs_src/how_to_guides/writing_custom_nodes_example_2_py38.py
 ```
 
+:::
+
+:::{tab-item} Python 3.8+ & Return
+:sync: python38plus
+
+```{literalinclude} ../../../docs_src/how_to_guides/writing_custom_nodes_example_2_py38_return.py
+```
+
 :::
 ::::
 
@@ -97,7 +110,12 @@ Here are some explanations.
   the value changes, pytask knows it needs to regenerate the workflow. We can use
   the timestamp of when the node was last modified.
 - pytask calls {meth}`PickleNode.load` when it collects the values of function arguments
-  to run the function. In our example, we read the file and unpickle the data.
+  to run the function. The argument `is_product` signals that the node is loaded as a
+  product with a {class}`~pytask.Product` annotation or via `produces`.
+
+  When the node is loaded as a dependency, we want to inject the value of the pickle
+  file. In the other case, the node returns itself so users can call
+  {meth}`PickleNode.save` themselves.
 - {meth}`PickleNode.save` is called when a task function returns and allows to save the
   return values.
 

docs/source/how_to_guides/writing_custom_nodes.md

@@ -33,7 +33,7 @@ To write to the terminal, use pytask's console.
 pytask uses marks to attach additional information to task functions which is processed
 by the host or by plugins. The following marks are available by default.
 
-### Marks
+### Built-in marks
 
 ```{eval-rst}
 .. function:: pytask.mark.depends_on(objects: Any | Iterable[Any] | dict[Any, Any])
@@ -236,7 +236,8 @@ The remaining exceptions convey specific errors.
 
 ```{eval-rst}
 .. autoclass:: pytask.Session
-
+.. autoclass:: pytask.DataCatalog
+   :members:
 ```
 
 ## Protocols
@@ -262,7 +263,11 @@ Nodes are the interface for different kinds of dependencies or products.
 
 ```{eval-rst}
 .. autoclass:: pytask.PathNode
+   :members: load, save
+.. autoclass:: pytask.PickleNode
+   :members: load, save
 .. autoclass:: pytask.PythonNode
+   :members: load, save
 ```
 
 To parse dependencies and products from nodes, use the following functions.
@@ -338,6 +343,13 @@ outcome.
 .. autofunction:: pytask.count_outcomes
 ```
 
+## Path utilities
+
+```{eval-rst}
+.. autofunction:: pytask.path.import_path
+.. autofunction:: pytask.path.hash_path
+```
+
 ## Programmatic Interfaces
 
 ```{eval-rst}
@@ -355,6 +367,17 @@ There are some classes to handle different kinds of reports.
 .. autoclass:: pytask.DagReport
 ```
 
+## Tree utilities
+
+```{eval-rst}
+.. autofunction:: pytask.tree_util.PyTree
+.. autofunction:: pytask.tree_util.tree_flatten_with_path
+.. autofunction:: pytask.tree_util.tree_leaves
+.. autofunction:: pytask.tree_util.tree_map
+.. autofunction:: pytask.tree_util.tree_map_with_path
+.. autofunction:: pytask.tree_util.tree_structure
+```
+
 ## Typing
 
 ```{eval-rst}

docs/source/reference_guides/api.md

@@ -46,12 +46,12 @@ are welcome to also support macOS.
 
 pytask uses a database to keep track of tasks, products, and dependencies over runs. By
 default, it will create an SQLite database in the project's root directory called
-`.pytask.sqlite3`. If you want to use a different name or a different dialect
+`.pytask/pytask.sqlite3`. If you want to use a different name or a different dialect
 [supported by sqlalchemy](https://docs.sqlalchemy.org/en/latest/core/engines.html#backend-specific-urls),
 use either {option}`pytask build --database-url` or `database_url` in the config.
 
 ```toml
-database_url = "sqlite:///.pytask.sqlite3"
+database_url = "sqlite:///.pytask/pytask.sqlite3"
 ```
 
 Relative paths for SQLite databases are interpreted as either relative to the

docs/source/reference_guides/configuration.md

@@ -4,7 +4,7 @@ pytask can be configured via the command-line interface or permanently with a
 `pyproject.toml` file.
 
 The file also indicates the root of your project where pytask stores information in a
-`.pytask.sqlite3` database.
+`.pytask` folder.
 
 ## The configuration file