NewType and TypeVar don't resolve correctly

[anselor]

Describe the bug
Autodoc generates warnings and doesn't resolve NewType or TypeVar correctly.

To Reproduce
Steps to reproduce the behavior:

$ git clone https://github.com/anselor/sphinx-issues-examples/
$ cd sphinx-issues-examples/typevar-namespace
$ python -m sphinx -b html docs/ doc-out/ -nvWT

Expected behavior
Documentation generated with TypeVar and NewType names that link to the documented declaration

Your project
https://github.com/anselor/sphinx-issues-examples/tree/main/typevar-namespace

Screenshots
If applicable, add screenshots to help explain your problem.

Environment info

• OS: Ubuntu 18.04, 20.04
• Python version: 3.6.9, 3.8.2
• Sphinx version: 3.2.1
• Sphinx extensions: sphinx.ext.autodoc
• Extra tools:

Additional context
Add any other context about the problem here.

• [e.g. URL or Ticket]

[anselor]

@tk0miya FYI - this is the new issue I created as you suggested.

[tk0miya]

@anselor I can't reproduce any warnings from your example.

This is a Dockerfile to reproduce the warning:

FROM python:3.8-slim

RUN apt update; apt install -y build-essential curl git make unzip vim
RUN git clone https://github.com/anselor/sphinx-issues-examples/
RUN pip install Sphinx
WORKDIR /sphinx-issues-examples/typevar-namespace/src
RUN pip install -e .
WORKDIR /sphinx-issues-examples/typevar-namespace/docs
RUN sphinx-build . _build/html

And its result:

Running Sphinx v3.3.1
making output directory... done
loading intersphinx inventory from https://docs.python.org/3/objects.inv...
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 1 source files that are out of date
updating environment: [new config] 1 added, 0 changed, 0 removed
reading sources... [100%] index

looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] index

generating indices... genindex py-modindex done
writing additional pages... search done
copying static files... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded.

The HTML pages are in _build/html.

[anselor]

I was using 3.2.1 - just updated to sphinx 3.3.1 to make sure. The steps in your docker container do not match the same steps/options that I documented in the ticket.

I tried running it the way you have in your docker container and can confirm that sphinx doesn't report any errors but silently failed to resolve symbols or link them. Note that a locally declared TypeVar resolves correctly but a TypeVar imported from another module fails. I've underlined them in green and red in the output here:

For NewType it doesn't recognize the NewType name and generates this instead:

I would expect it to say:
special() -> SpecialType and link to the decleration of SpecialType

[tk0miya]

Sorry, I did not pass -n option to sphinx-build command. Now I reproduced the error. Okay, I'll take a look again.

FROM python:3.8-slim

RUN apt update; apt install -y build-essential curl git make unzip vim
RUN git clone https://github.com/anselor/sphinx-issues-examples/
RUN pip install Sphinx
WORKDIR /sphinx-issues-examples/typevar-namespace/src
RUN pip install -e .
WORKDIR /sphinx-issues-examples/typevar-namespace/docs
RUN sphinx-build -nvWT .  _build/html

Running Sphinx v3.3.1
making output directory... done
loading intersphinx inventory from https://docs.python.org/3/objects.inv...
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 1 source files that are out of date
updating environment: [new config] 1 added, 0 changed, 0 removed
reading sources... [100%] index

looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] index


Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/sphinx/cmd/build.py", line 280, in build_main
    app.build(args.force_all, filenames)
  File "/usr/local/lib/python3.8/site-packages/sphinx/application.py", line 352, in build
    self.builder.build_update()
  File "/usr/local/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 297, in build_update
    self.build(to_build,
  File "/usr/local/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 361, in build
    self.write(docnames, list(updated_docnames), method)
  File "/usr/local/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 535, in write
    self._write_serial(sorted(docnames))
  File "/usr/local/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 545, in _write_serial
    self.write_doc(docname, doctree)
  File "/usr/local/lib/python3.8/contextlib.py", line 120, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.8/site-packages/sphinx/util/logging.py", line 213, in pending_warnings
    memhandler.flushTo(logger)
  File "/usr/local/lib/python3.8/site-packages/sphinx/util/logging.py", line 178, in flushTo
    logger.handle(record)
  File "/usr/local/lib/python3.8/logging/__init__.py", line 1587, in handle
    self.callHandlers(record)
  File "/usr/local/lib/python3.8/logging/__init__.py", line 1649, in callHandlers
    hdlr.handle(record)
  File "/usr/local/lib/python3.8/logging/__init__.py", line 946, in handle
    rv = self.filter(record)
  File "/usr/local/lib/python3.8/logging/__init__.py", line 807, in filter
    result = f.filter(record)
  File "/usr/local/lib/python3.8/site-packages/sphinx/util/logging.py", line 421, in filter
    raise exc
sphinx.errors.SphinxWarning: /sphinx-issues-examples/typevar-namespace/src/issue/typevar/support.py:docstring of issue.typevar.support.MyClass.special::py:class reference target not found: NewType.<locals>.new_type

Warning, treated as error:
/sphinx-issues-examples/typevar-namespace/src/issue/typevar/support.py:docstring of issue.typevar.support.MyClass.special::py:class reference target not found: NewType.<locals>.new_type
The command '/bin/sh -c sphinx-build -nvWT  .  _build/html' returned a non-zero code: 2

[tk0miya]

I posted #8426 to fix the TypeVar warnings.

On the other hand, it is difficult to fix the case of NewType. It is hard to get the original name of the type in runtime. So please use autodoc_type_hints to resolve the name. It will resolve your case.
https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html#confval-autodoc_type_aliases

[anselor]

Can you not access the __name__ property?

https://github.com/python/cpython/blob/2837241f22be33a5597707b2aa723cb2cf6f3967/Lib/typing.py#L1942

[tk0miya]

Do you know how to detect is the type a kind of NewType? When should we access __name__?

[anselor]

They way it's implemented, NewType() returns a function that has 2 additional attributes: __name__ and __supertype__.
In my code that does some runtime type validation I check for these things:

Does __qualname__ equal NewType.<locals>.new_type ?
Do __name__ and __supertype__ exist as attributes?

[tk0miya]

Hmm... I feel it's a bit fragile. But, I understand it is better to support NewType as possible by default. Additionally, I need to know how to get the module of the new type. In this case, we need to get issue.typevar.support.SpecialInt from the NewType object. I'll take a look later.