Skip to content

Add more figures to the documentation. #193

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jan 11, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -66,16 +66,16 @@ projects. Its features include:
<https://pytask-dev.readthedocs.io/en/latest/tutorials/how_to_select_tasks.html>`_
known from pytest.

- **Easily extensible with plugins**. pytask's architecture is based on `pluggy
<https://pluggy.readthedocs.io/en/latest/>`_, a plugin management framework, so that
you can adjust pytask to your needs. Plugins are available for `parallelization
- **Easily extensible with plugins**. pytask is built on top of `pluggy
<https://pluggy.readthedocs.io/en/latest/>`_, a plugin management framework, which
allows you to adjust pytask to your needs. Plugins are available for `parallelization
<https://github.com/pytask-dev/pytask-parallel>`_, `LaTeX
<https://github.com/pytask-dev/pytask-latex>`_, `R
<https://github.com/pytask-dev/pytask-r>`_, and `Stata
<https://github.com/pytask-dev/pytask-stata>`_ and `many more
<https://github.com/topics/pytask>`_. Read `here
<https://github.com/pytask-dev/pytask-stata>`_ and more can be found `here
<https://github.com/topics/pytask>`_. Read in `this tutorial
<https://pytask-dev.readthedocs.io/en/latest/tutorials/how_to_use_plugins.html>`_ how
you can use plugins.
to use and create plugins.

.. end-features

Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/source/_static/images/how-to-debug-trace.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/source/_static/images/persist-executed.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/source/_static/images/persist-persisted.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/source/_static/images/pytask-collect.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/source/_static/images/pytask-profile.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions docs/source/changes.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ all releases are available on `PyPI <https://pypi.org/project/pytask>`_ and
------------------

- :gh:`191` adds a guide on how to profile pytask to the developer's guide.
- :gh:`193` adds more figures to the documentation.


0.1.5 - 2022-01-10
Expand Down
45 changes: 29 additions & 16 deletions docs/source/how_to_guides/how_to_write_a_plugin.rst
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,8 @@ steps.
- Check whether there exist plugins which offer similar functionality. For example, many
plugins provide convenient interfaces to run another program with inputs via the
command line. Naturally, there is a lot of overlap in the structure of the program and
even the the test battery. Finding the right plugin as a template may save you a lot
of time.
even the test battery. Finding the right plugin as a template may save you a lot of
time.

- Make a list of hooks you want to implement. Think about how this plugin relates to
functionality defined in pytask and other plugins. Maybe skim the documentation on
Expand All @@ -38,25 +38,38 @@ This section explains some steps which are required for all plugins.
Set up the setuptools entry-point
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

pytask discovers plugins via ``setuptools`` entry-points. This is specified in
``setup.py``. See the following example.
pytask discovers plugins via ``setuptools`` entry-points. Following the approach
advocated for by `setuptools_scm <https://github.com/pypa/setuptools_scm>`_, the
entry-point is specified in ``setup.cfg``.

.. code-block:: python
.. code-block:: cfg

# Content of setup.cfg

[metadata]
name = pytask-plugin

[options.packages.find]
where = src

[options.entry_points]
pytask =
pytask_plugin = pytask_plugin.plugin

For ``setuptools_scm`` you also need a ``pyproject.toml`` with the following content.

.. code-block:: toml

# Content of setup.py
# Content of pyproject.toml

from setuptools import find_packages
from setuptools import setup
[build-system]
requires = ["setuptools>=45", "wheel", "setuptools_scm[toml]>=6.0"]

setup(
name="pytask-plugin",
version="0.0.1",
entry_points={"pytask": ["pytask_plugin = pytask_plugin.plugin"]},
# PyPI classifier for pytask plugins
classifiers=["Framework :: pytask"],
)
[tool.setuptools_scm]
write_to = "src/pytask_plugin/_version.py"

For an example with ``setuptools_scm`` and ``setup.cfg`` see the `pytask-parallel repo
For a complete example with ``setuptools_scm`` and ``setup.cfg`` see the
`pytask-parallel repo
<https://github.com/pytask-dev/pytask-parallel/blob/main/setup.cfg>`_.

The entry-point for pytask is called ``"pytask"`` and points to a module called
Expand Down
32 changes: 6 additions & 26 deletions docs/source/tutorials/how_to_capture_output.rst
Original file line number Diff line number Diff line change
Expand Up @@ -46,11 +46,11 @@ You can influence output capturing mechanisms from the command line:

.. code-block:: console

pytask -s # disable all capturing
pytask --capture=sys # replace sys.stdout/stderr with in-mem files
pytask --capture=fd # also point filedescriptors 1 and 2 to temp file
pytask --capture=tee-sys # combines 'sys' and '-s', capturing sys.stdout/stderr
# and passing it along to the actual sys.stdout/stderr
$ pytask -s # disable all capturing
$ pytask --capture=sys # replace sys.stdout/stderr with in-mem files
$ pytask --capture=fd # also point filedescriptors 1 and 2 to temp file
$ pytask --capture=tee-sys # combines 'sys' and '-s', capturing sys.stdout/stderr
# and passing it along to the actual sys.stdout/stderr


Using print statements for debugging
Expand All @@ -75,24 +75,4 @@ print statements for debugging:
and running this module will show you precisely the output of the failing function and
hide the other one:

.. code-block:: console

$ pytask -s
========================= Start pytask session =========================
Platform: win32 -- Python 3.x.x, pytask 0.x.x, pluggy 0.13.x
Root: .
Collected 2 task(s).

F.
=============================== Failures ===============================
_________________ Task task_capture.py::task_func2 failed ______________

Traceback (most recent call last):
File "task_capture.py", line 7, in task_func2
assert False
AssertionError

---------------------- Captured stdout during call ---------------------
Debug statement.

==================== 1 succeeded, 1 failed in 0.01s ====================
.. image:: /_static/images/how-to-capture-output.png
33 changes: 6 additions & 27 deletions docs/source/tutorials/how_to_collect_tasks.rst
Original file line number Diff line number Diff line change
Expand Up @@ -21,36 +21,12 @@ For example, let us take the following task

Now, running :program:`pytask collect` will produce the following output.

.. code-block:: console

$ pytask collect
========================= Start pytask session =========================
Platform: linux -- Python 3.x.y, pytask 0.x.y, pluggy 0.x.y
Root: xxx
Collected 1 task(s).

<Module /.../task_module.py>
<Function task_write_file>

========================================================================
.. image:: /_static/images/pytask-collect.png

If you want to have more information regarding dependencies and products of the task,
append the ``--nodes`` flag.

.. code-block:: console

$ pytask collect
========================= Start pytask session =========================
Platform: linux -- Python 3.x.y, pytask 0.x.y, pluggy 0.x.y
Root: xxx
Collected 1 task(s).

<Module /.../task_module.py>
<Function task_write_file>
<Dependency /.../in.txt>
<Product /.../out.txt>

========================================================================
.. image:: /_static/images/pytask-collect-nodes.png

To restrict the set of tasks you are looking at, use markers, expression and ignore
patterns as usual.
Expand All @@ -59,4 +35,7 @@ patterns as usual.
Further reading
---------------

- :program:`pytask collect` in :doc:`../reference_guides/command_line_interface`.
- The documentation on the command line interface of :program:`pytask collect` can be
found :doc:`here <../reference_guides/command_line_interface>`.
- Read :doc:`here <how_to_select_tasks>` about selecting tasks.
- Paths can be ignored with :confval:`ignore`.
23 changes: 8 additions & 15 deletions docs/source/tutorials/how_to_debug.rst
Original file line number Diff line number Diff line change
Expand Up @@ -12,24 +12,19 @@ Tracebacks
----------

You can enrich the display of tracebacks by showing local variables in each stack frame.
Just execute pytask with
Just execute pytask with :confval:`show_locals`, meaning ``pytask --show-locals``.

.. code-block:: console

$ pytask --show-locals
.. image:: /_static/images/how-to-debug-show-locals.png


Debugging
---------

Running

.. code-block:: console
Using :confval:`pdb` enables the post-mortem debugger. Whenever an exception is raised
inside a task, the prompt will enter the debugger enabling you to find out the cause of
the exception.

$ pytask --pdb

enables the post-mortem debugger. Whenever an exception is raised inside a task, the
prompt will enter the debugger enabling you to discover the source of the exception.
.. image:: /_static/images/how-to-debug-pdb.png

.. seealso::

Expand All @@ -46,11 +41,9 @@ prompt will enter the debugger enabling you to discover the source of the except
Tracing
-------

If you want to enter the debugger at the start of every task, use

.. code-block:: console
If you want to enter the debugger at the start of every task, use :confval:`trace`.

$ pytask --trace
.. image:: /_static/images/how-to-debug-trace.png


Custom debugger
Expand Down
70 changes: 28 additions & 42 deletions docs/source/tutorials/how_to_make_tasks_persist.rst
Original file line number Diff line number Diff line change
@@ -1,16 +1,23 @@
How to make tasks persist
=========================

Sometimes you want to skip the execution. It means that if all dependencies
and products exist, the task will not be executed even though a dependency, the task's
source file or a product has changed. Instead, the state of the dependencies, the source
file and the products is updated in the database such that the next execution will skip
the task successfully.
Sometimes you want to skip the execution of a task and pretend like nothing has changed.

A common scenario is that you have a long running task which will be executed again if
you would format the task's source file with `black <https://github.com/psf/black>`_.

In this case, you can apply the ``@pytask.mark.persist`` decorator to the task which
will skip its execution as long as all products exist.

Internally, the state of the dependencies, the source file and the products is updated
in the database such that the next execution will skip the task successfully.


When is this useful?
--------------------

- You ran a formatter like Black on the files in your project.
- You ran a formatter like Black on the files in your project and want to prevent the
longest running tasks from being rerun.

- You extend a parametrization, but do not want to rerun all tasks.

Expand All @@ -30,12 +37,11 @@ How to do it?

To create a persisting task, apply the correct decorator and, et voilà, it is done.

Let us take the second scenario as an example. First, we define the tasks, the
dependency and the product and save everything in the same folder.
To see the whole process, first, we create some task and its dependency.

.. code-block:: python

# Content of task_file.py
# Content of task_module.py

import pytask

Expand All @@ -47,45 +53,25 @@ dependency and the product and save everything in the same folder.
produces.write_text("**" + depends_on.read_text() + "**")


.. code-block::
.. code-block:: md

# Content of input.md. Do not copy this line.
<!-- Content of input.md. -->

Here is the text.

If you execute the task with pytask, the task will be executed since the product is
missing.

.. code-block::

# Content of output.md. Do not copy this line.

**Here is the text.**


If you run pytask in this folder, you get the following output.

.. code-block:: console

$ pytask demo
========================= Start pytask session =========================
Platform: win32 -- Python 3.8.5, pytask 0.0.6, pluggy 0.13.1
Root: xxx/demo
Collected 1 task(s).

p
====================== 1 succeeded in 1 second(s) ======================

The green p signals that the task persisted. Another execution will show the following.
.. image:: /_static/images/persist-executed.png

.. code-block:: console
After that, we change the source file of the task accidentally by formatting the file
with black. Without the ``@pytask.mark.persist`` decorator the task would run again
since it has changed. With the decorator, the execution is skipped which is signaled by
a green p.

$ pytask demo
========================= Start pytask session =========================
Platform: win32 -- Python 3.8.5, pytask 0.0.6, pluggy 0.13.1
Root: xxx/demo
Collected 1 task(s).
.. image:: /_static/images/persist-persisted.png

s
====================== 1 succeeded in 1 second(s) ======================
If we now run the task again, it is skipped because nothing has changed and not because
it is marked with ``@pytask.mark.persist``.

Now, the task is skipped successfully because nothing has changed compared to the
previous run.
.. image:: /_static/images/persist-skipped-successfully.png
4 changes: 4 additions & 0 deletions docs/source/tutorials/how_to_profile_tasks.rst
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,7 @@ display the information, enter
.. code-block:: console

$ pytask profile

Here is an example

.. image:: /_static/images/pytask-profile.png
4 changes: 2 additions & 2 deletions docs/source/tutorials/how_to_use_plugins.rst
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@ Plugins can be found in many places.
How to use plugins
------------------

To use a plugin, simply follow the installation instructions. A plugin will enable
itself by using pytask's entry-point.
To use a plugin, simply follow the installation instructions and the accompanying
documentation. A plugin will usually enable itself by using pytask's entry-point.


How to implement your own plugin
Expand Down
Loading