autodoc and autosummary failing when class and module have the same name

[larsoner]

Describe the bug

We have mne/dipole.py in which there is a Dipole class defined. We import this class (and .dipole) in mne/__init__.py, and document with autosummary/autodoc the mne.Dipole class in the root namespace. On Windows (case-insensitive), we have the following problem, which just started show up on April 17th-ish in our builds:

C:\Users\larsoner\python\mne-python\mne\dipole.py:docstring of mne.dipole.Dipole:98: WARNING: autosummary: failed to import crop.
Possible hints:
* ValueError: not enough values to unpack (expected 2, got 1)
* ModuleNotFoundError: No module named 'crop'
* ModuleNotFoundError: No module named 'mne.Dipole.crop'; 'mne.Dipole' is not a package
* KeyError: 'crop'
* AttributeError: module 'mne' has no attribute 'crop'
* ModuleNotFoundError: No module named 'mne.crop'
* AttributeError: module 'mne.Dipole' has no attribute 'crop'

mne.Dipole.crop is a valid method and is documented properly on Linux. I think it's driven by the following behavior on Windows -- namely, that if you try import mne.Dipole Python imports mne/dipole.py module as mne.Dipole:

>>> import mne
>>> mne.Dipole   # correct!
<class 'mne.dipole.Dipole'>
>>> import mne.Dipole
>>> mne.Dipole  # bad!
<module 'mne.Dipole' from 'C:\\Users\\larsoner\\python\\mne-python\\mne\\Dipole.py'>
>>> from mne import Dipole  # bad enough, but continue...
>>> Dipole  # also bad!
<module 'mne.Dipole' from 'C:\\Users\\larsoner\\python\\mne-python\\mne\\Dipole.py'>

I think autosummary is doing some module importing under the hood here that is having this effect. I'm not sure if it's possible to directly try mne.dipole.Dipole or mne.Dipole before trying to import mne.Dipole, as that would fix the problem...?

How to Reproduce

I can try to come up with a MWE if it's not obvious that there is some existing workaround or what the problem is. I'm not sure it will be easy, though, as which errors we get for which classes appears to change run-to-run :(

Environment Information

At least sphinx 5.3 and the latest release (tested both).

Sphinx extensions

autodoc, autosummary

Additional context

• I've looked at autodoc&autosummary fails for uppercase/lowercase class/function #4874 and changing the stub name does not help.
• I've considered documenting mne.Dipole at mne.dipole.Dipole in the docs instead, but we really want this (and other classes with the same problem) in the root namespace. We also want mne.dipole to be public because some dipole-related functions live there and shouldn't live in the root mne namespace.

[picnixz]

The way autosummary is importing objects is using

sphinx/sphinx/ext/autosummary/__init__.py

Lines 663 to 707 in 3c4e78e

	def _import_by_name(name: str, grouped_exception: bool = True) -> tuple[Any, Any, str]:
	"""Import a Python object given its full name."""
	errors: list[BaseException] = []
	try:
	name_parts = name.split('.')
	# try first interpret `name` as MODNAME.OBJ
	modname = '.'.join(name_parts[:-1])
	if modname:
	try:
	mod = import_module(modname)
	return getattr(mod, name_parts[-1]), mod, modname
	except (ImportError, IndexError, AttributeError) as exc:
	errors.append(exc.__cause__ or exc)
	# ... then as MODNAME, MODNAME.OBJ1, MODNAME.OBJ1.OBJ2, ...
	last_j = 0
	modname = None
	for j in reversed(range(1, len(name_parts) + 1)):
	last_j = j
	modname = '.'.join(name_parts[:j])
	try:
	import_module(modname)
	except ImportError as exc:
	errors.append(exc.__cause__ or exc)
	if modname in sys.modules:
	break
	if last_j < len(name_parts):
	parent = None
	obj = sys.modules[modname]
	for obj_name in name_parts[last_j:]:
	parent = obj
	obj = getattr(obj, obj_name)
	return obj, parent, modname
	else:
	return sys.modules[modname], None, modname
	except (ValueError, ImportError, AttributeError, KeyError) as exc:
	errors.append(exc)
	if grouped_exception:
	raise ImportExceptionGroup('', errors)
	else:
	raise ImportError(*exc.args) from exc

Try to look inside the codesource directly

sphinx/sphinx/ext/autosummary/__init__.py

Lines 316 to 322 in 3c4e78e

	try:
	real_name, obj, parent, modname = self.import_by_name(name, prefixes=prefixes)
	except ImportExceptionGroup as exc:
	errors = list({f"* {type(e).__name__}: {e}" for e in exc.exceptions})
	logger.warning(__('autosummary: failed to import %s.\nPossible hints:\n%s'),
	name, '\n'.join(errors), location=self.get_location())
	continue

putting print here and there (I'm on Linux so it's kind of hard to reproduce the issue). First, it'd be good to know where the ValueError occurs because it's a bit weird. Also, it'd be good to know the original real target name that was tried.

What I understood:

• The names that will be imported and checked are actually prefixed with some strings. This is done in

  sphinx/sphinx/ext/autosummary/__init__.py

  Line 629 in 3c4e78e

  def import_by_name(

• The prefixes that are used are constructed in

  sphinx/sphinx/ext/autosummary/__init__.py

  Line 608 in 3c4e78e

  def get_import_prefixes_from_env(env: BuildEnvironment) -> list[str | None]:

  . So it is worth taking a look at what is actually imported and in which order.

If you can tell me in the exact order of what was imported (and also why there was a ValueError), I can think of a workaround. By the way, the reason why autosummary_filename_map does not work is because it is used after the items are imported, so it will never be considered. The only idea I have is to actually bypass the import system and make it so that everything is imported in a case-sensitive way but 1) I don't have any testing environment 2) it might still be fragile

[larsoner]

1. I don't have any testing environment

FWIW nowadays you can get a Windows VM running for testing stuff. It would probably take ~1h to set it up with Python and everything, but it's at least doable if you're interested!

2. it might still be fragile

Indeed in thinking about the issue it's hard to see a solution that wouldn't be fragile given Python's own behavior of import mne.Dipole creating a mne.Dipole module :(

I'll try to look again on my Windows boot at some point but probably won't get to it for a bit.