autosummary generates classes on python <=3.8 but stops working on python >3.8

[FirefoxMetzger]

Describe the bug

When adding lazy importing of members to a module, sphinx fails to build the docs correctly in python 3.9 and 3.10, but works as expected up until python <=3.8.

How to Reproduce

$ git clone -b add-lazy-importing https://github.com/FirefoxMetzger/sphinx-39-bug.git
$ cd sphinx-39-bug
$ pip install sphinx
$ sphinx-build -W ./doc/source ./doc/build

Output on python >3.8:

Running Sphinx v4.4.0
making output directory... done
[autosummary] generating autosummary for: index.rst
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 1 source files that are out of date
updating environment: [new config] 1 added, 0 changed, 0 removed
reading sources... [100%] index                                                                                                                                                                                                                                      

Warning, treated as error:
docstring of demp_project:5:autosummary: stub file not found 'demp_project.Foo'. Check your autosummary_generate setting.

Alternatively, you can check the logs of the CI that I attached to the repo to build the docs on major python versions: FirefoxMetzger/sphinx-39-bug#3

Expected behavior

The docs should build cleanly, no matter which python version is being used.

Your project

https://github.com/FirefoxMetzger/sphinx-39-bug

Screenshots

No response

OS

Tested on Windows 11 and ubuntu 20.04

Python version

3.7 - 3.10

Sphinx version

4.4.0

Sphinx extensions

No response

Extra tools

No response

Additional context

I also posted this as a question on StackOverflow: https://stackoverflow.com/questions/70978404/why-does-sphinx-autosummary-not-create-a-toctree-when-used-inside-a-class

I got some help from a user called mzjn; however, he couldn't reproduce the faulty behavior locally, which makes me wonder if I might just be going about this the wrong way.

[tk0miya]

It seems pydoc.getdoc() does not work with your LazyImporter. It returns an empty string for the instance. I'm not sure why the behavior was changed since 3.9.

sphinx/sphinx/ext/autosummary/generate.py

Lines 511 to 512 in a32d609

	real_name, obj, parent, modname = import_by_name(name, grouped_exception=True)
	lines = pydoc.getdoc(obj).splitlines()

[tk0miya]

$  python
Python 3.8.12 (default, Nov 17 2021, 17:08:21)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import demp_project
>>> import sys
>>> sys.modules['demp_project']
<demp_project.LazyImporter object at 0x7f1744bbf520>
>>> import pydoc
>>> pydoc.getdoc(sys.modules['demp_project'])
'The demp_project package\n\nThe demp_project consists of the following classes:\n\n.. autosummary::\n    :toctree: _autosummary\n\n    demp_project.Foo\n    demp_project.Bar'

$ python
Python 3.9.9 (main, Nov 17 2021, 16:30:15)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import demp_project
>>> import sys
>>> sys.modules['demp_project']
<demp_project.LazyImporter object at 0x7fbba25c92e0>
>>> import pydoc
>>> pydoc.getdoc(sys.modules['demp_project'])
''

[paradox-lab]

The behavior returns an empty string for the instance was changed in #20073.

pydoc.getdoc no longer returns docstring inherited from the type of the object or from parent class if it is a class if it is not defined in the object itself.

It works on python >3.8 like below:

demp_project.__init__.py

import sys

class Foo:
    """ Summary of class Foo
    
    Here comes extensive documentation.
    
    """


class Bar:
    """ Summary of class Bar

    even more extensive documentation.
    
    """


class LazyImporter:

    __name__ = __name__

    def __getattr__(self, name):
        """Lazy-Import Awesome Things

        Note: this doesn't actually lazy import here, because everything is in
        the same file for demo purposes. In the real world, this calls
        importlib.import_module to make the magic happen.

        """

        if name == "Foo":
            return Foo
        elif name == "Bar":
            return Bar
        else:
            raise AttributeError(f"module '{__name__}' has no attribute '{name}'")


lazy_importer = LazyImporter()
lazy_importer.__doc__ = """ The demp_project package

    The demp_project consists of the following classes:

    .. autosummary::
        :toctree: _autosummary

        demp_project.Foo
        demp_project.Bar

"""

sys.modules[__name__] = lazy_importer

sphinx-build -W ./doc/source ./doc/build

output

Running Sphinx v4.4.0
[autosummary] generating autosummary for: index.rst
[autosummary] generating autosummary for: C:\lzx\sphinx-39-bug\doc\source\_autosummary\demp_project.Bar.rst, C:\lzx\sphinx-39-bug\doc\source\_autosummary\demp_project.Foo.rst
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 1 source files that are out of date
updating environment: [new config] 3 added, 0 changed, 0 removed
reading sources... [100%] index
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] index
generating indices... genindex py-modindex done
writing additional pages... search done
copying static files... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded.

The HTML pages are in doc\build.

[FirefoxMetzger]

@paradox-lab that does the trick 🚀 . Defining the docstring on the class instance instead of the class itself does indeed fix the issue.

Since this gets resolved by your suggestion, I will close this issue.