Support for autosummary :recursive: documentation of entire API

[Yoshanuikabundi]

Hi! Thanks for this package, I love the results it produces!

I've been experimenting with ways to fully automate the generation of API reference documentation from source code and docstrings, a la rustdoc. I know you're aware of this, but there's no easy way to include autodoc_pydantic output in a fully automated API reference.

The following works, of course:

.. autosummary::
   :toctree: _autosummary

   module_with_lots_of_stuff.AutoSummaryModel
   module_with_lots_of_stuff.AutoSummarySettings
   module_with_lots_of_stuff.another_module

But I would like the following:

.. autosummary::
   :toctree: _autosummary
   :recursive:

   module_with_lots_of_stuff

I understand that the reason this doesn't work is that Sphinx doesn't let you expose new variables to the autosummary templates (as you discuss in #11). I've worked around this to some extent in the project I'm working on by adding an autosummary table with all the members not documented elsewhere in the template to the end of the module template, but it's not very good and we probably won't end up using it.

My suggestion would be to workaround the bug in Sphinx by adding a Jinja2 filter along the lines of "keep_pydantic_models" that takes an iterator of names of objects and filters out those that aren't models. So a block like the following could be added to an Autosummary template:

{% set models = members | keep_pydantic_models(module=fullname) | list %}

{% if models %}
Models
---------

{% for item in models %}
.. autopydantic_model:: item
{% endfor %}

I think you should be able to add such a filter to all templates, but I'm not super familiar. Sorry if this isn't helpful.

Thanks!

[StephenHogg]

I'd like to add my voice to this, have just run into this exact problem today and would love it if this would fit into the recursive command along with everything else. For now I'm going to just ignore the pydantic files as I don't quite understand the workaround, unfortunately. If anyone (@Yoshanuikabundi ?) could explain it, I'd be very grateful though.

[mansenfranzen]

@Yoshanuikabundi Thanks for your detailed report and your thoughts on this issue. I know it's a pity that a fully automated API documentation does not yet work properly with autodoc_pydantic.

Unfortunately I haven't had enough time to take a deep dive on this issue, yet. But judging from a high level perspective, I assume effort is best invested in extending sphinx.ext.autosummary instead of finding a workaround in autodoc_pydantic. We might find a workaround but it is going to be hacky I guess (but please prove me wrong otherwise - I'm happy to take a look at any implementation or PR).

I hopefully will come back to this soon. Providing a PR to sphinx with the required functionality also shouldn't be too difficult.

[StephenHogg]

I think I've narrowed it down. Try this with v1.3.2-a.1, you should see an error:

conf.py

import os
import sys

sys.path.insert(0, os.path.abspath('.'))

project = 'Test'
copyright = 'Test'
author = 'Test'

extensions = [
    'sphinx.ext.autodoc',
    'sphinxcontrib.autodoc_pydantic',
    'sphinx.ext.napoleon',
    'sphinx.ext.autosummary'
]

autosummary_generate = True  # Turn on sphinx.ext.autosummary
html_theme = "sphinx_rtd_theme"

index.rst

.. Test documentation master file, created by
   sphinx-quickstart on Sat Aug  7 16:28:18 2021.
   You can adapt this file completely to your liking, but it should at least
   contain the root `toctree` directive.

Welcome to Test's documentation!
================================

.. autosummary:: test
    :toctree: _autosummary
    :recursive:


Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

test.py

from pydantic import BaseModel


class TestClass:
    """Test

    Attributes:
        model (TestModel): Model
    """

    def __init__(self):
        self.model = TestModel()


class TestModel(BaseModel):
    pass

Thanks for making this package, by the way!

[StephenHogg]

The error it generates for me is as follows:

shogg@DESKTOP:~/git/test$ make html
Running Sphinx v4.1.2
making output directory... done
[autosummary] generating autosummary for: index.rst
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 1 source files that are out of date
updating environment: [new config] 1 added, 0 changed, 0 removed
reading sources... [100%] index                                                                                                         
/home/shogg/git/test/index.rst.rst:9: WARNING: autosummary: stub file not found 'test'. Check your autosummary_generate setting.
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] index                                                                                                          
generating indices... genindex done
writing additional pages... search done
copying static files... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded, 1 warning.

The HTML pages are in _build/html.

[mansenfranzen]

@StephenHogg I think your index.rst is not correct. Try:

.. autosummary::
   :toctree: _autosummary
   :recursive:

   test

[mansenfranzen]

@Yoshanuikabundi I took a closer look at your proposal.

It is a good idea to extend the standard autosummary template to call the custom function keep_pydantic_models from within the jinja template that filters only pydantic models and then the jinja template loops through all models. The function itself is not a big deal.

However, providing the function to the jinja template namespace is currently not possible because the jinja template environment (which needs to be accessed to register a function) is created here:

https://github.com/sphinx-doc/sphinx/blob/6ac326e019db949c2c8d58f523c2534be36d4e62/sphinx/ext/autosummary/generate.py#L117-L136

There is no proper way to access the jinja environment without monkeypatching this class :-(.

A dirty fix to allow autosummary to pick up pydantic models is to change the following lines:

https://github.com/sphinx-doc/sphinx/blob/6ac326e019db949c2c8d58f523c2534be36d4e62/sphinx/ext/autosummary/generate.py#L325-L326

to:

        ns['classes'], ns['all_classes'] = \
            get_members(obj, {'class', 'pydantic_model', 'pydantic_settings'}, imported=imported_members)

This change will make autosummary respect pydantic models/settings autodocumenters to be added under the classes section in the stub files.

Of course, both solutions are just workarounds. I'm going to provide a PR upstream to sphinx to address sphinx-doc/sphinx#6264 as this should not be too complicated.

The best solution should make the standard templates extensible without overwriting them completely. Instead, it should be possible to add custom sections to them template while also modifying the jinja namespace template as you've proposed. But this is more complicated.

[Yoshanuikabundi]

@mansenfranzen Thanks for the update! I think I completely agree with you. I thought the Jinja2 environment was extensible because autoapi provides a very clean API to modify it, but on closer inspection it looks like they can do this because they use their own templates rather than extending those of Autosummary. Sorry about that!

I definitely think fixing that upstream issue is a much better solution and I'm very grateful that you're taking a shot at it! Let me know if you need another set of eyes on it.

@StephenHogg Sorry for the confusion! My workaround was a proposal, not something I had working.

[utf]

Hi, I'm just wondering if there's any update on this issue?

[mansenfranzen]

Unfortunately not - I haven't gotten any response in the upstream issue sphinx-doc/sphinx#6264. I just added a friendly reminder.