AttributeError: Can't get attribute '_check_file' on <module '__main__' (built-in)>

[jaraco]

In the cssutils project, the sphinxlint command has started failing on my macOS Python 3.11.6 environment.

The execution emits a lot of errors and then hangs until I interrupt it.

The issue isn't currently happening in CI, so I'm suspicious the Python 3.11.6 update may be implicated.

[jaraco]

I've started to notice the same issue in other projects (distutils).

[jaraco]

But yet other projects (jaraco.functools) are unaffected. Could the difference be how many files are in the docs?

[jaraco]

The issue doesn't occur if I invoke sphinx-lint (the command runs to completion) instead of python -m sphinxlint. The issue also doesn't occur if I remove docs/distutils (the bulk of the docs).

[jaraco]

The issue emerges between 0.6.7 and 0.6.8.

[hugovk]

I can reproduce with cssutils when running via python -m sphinxlint docs (as used in the cssutils tox.ini), but not when running as sphinx-lint docs:

❯ sphinx-lint docs
No problems found.

0.6.8 is when sphinxlint/__main__.py was added with all the multiprocessing handling.

[jaraco]

The offending commit appears to be 42f1b98.

[hugovk]

Looks like it's this:

• https://stackoverflow.com/a/42383397/724176
• Possible race condition in Pool python/cpython#69240

And the fix is to move _check_file out of __main__.py and import it instead.

Here's a quick demo fix:

diff --git a/sphinxlint/__init__.py b/sphinxlint/__init__.py
index 91df074..55d9bdb 100644
--- a/sphinxlint/__init__.py
+++ b/sphinxlint/__init__.py
@@ -6,4 +6,12 @@ from sphinxlint.sphinxlint import check_file, check_text
 
 __version__ = importlib.metadata.version("sphinx_lint")
 
-__all__ = ["check_text", "check_file"]
+__all__ = ["check_text", "check_file", "_check_file"]
+
+
+def _check_file(todo):
+    """Wrapper to call check_file with arguments given by
+    multiprocessing.imap_unordered."""
+    return check_file(*todo)
+
+
diff --git a/sphinxlint/__main__.py b/sphinxlint/__main__.py
index 5a1290b..4835501 100644
--- a/sphinxlint/__main__.py
+++ b/sphinxlint/__main__.py
@@ -5,7 +5,7 @@ import os
 import sys
 from itertools import chain, starmap
 
-from sphinxlint import check_file, __version__
+from sphinxlint import check_file, __version__, _check_file
 from sphinxlint.checkers import all_checkers
 from sphinxlint.sphinxlint import CheckersOptions
 
@@ -166,12 +166,6 @@ def walk(path, ignore_list):
             yield file if file[:2] != "./" else file[2:]
 
 
-def _check_file(todo):
-    """Wrapper to call check_file with arguments given by
-    multiprocessing.imap_unordered."""
-    return check_file(*todo)
-
-
 def sort_errors(results, sorted_by):
     """Flattens and potentially sorts errors based on user prefernces"""
     if not sorted_by:

❯ sphinx-lint docs
No problems found.

❯ python -m sphinxlint docs
No problems found.

[jaraco]

I suspect the issue is that when multiprocessing is involved, _check_file is passed into the pool workers, but when invoked by runpy, __main__ is found at the top level and not in sphinxlint.__main__, so fails to be reimported in the worker.

[jaraco]

My recommendation would be to move nearly all logic out of sphinxlint.__main__ into another module (sphinxlint.cli maybe, or just sphinxlint). Doing so would avoid the temptation to include behavior that may need to be imported.

[jaraco]

I've also determined that by setting --jobs 1, it bypasses the multiprocessing, matching the behavior observed in CI where the issue doesn't occur.

Let me know if you'd like me to present the PR (especially if you're interested in the sphinxlint.cli approach).

[hugovk]

I've already started on doing sphinxlint.cli :)

[hugovk]

Please see PR #99, thanks for the investigation and suggestions!