Clean sys.modules if TYPE_CHECKING=True import fails

[godlygeek]

autodoc now attempts to import with typing.TYPE_CHECKING = True first, and then falls back to typing.TYPE_CHECKING = False if that fails. Unfortunately, the first import can leave behind some partially-imported modules in sys.modules such that the retry fails.

Attempt to work around this by detecting what modules were added to sys.modules by the first attempt and removing them before retrying.

Feature or Bugfix

• Bugfix

Purpose

autodoc now attempts to importing with typing.TYPE_CHECKING = True first, and then falls back to typing.TYPE_CHECKING = False if that fails. Unfortunately, the first import can leave behind some partially-imported modules in sys.modules such that the retry fails.

Attempt to work around this by detecting what modules were added to sys.modules by the first attempt and removing them before retrying.

Detail / Relates

See #11642

[godlygeek]

The 1 failing test run seems unrelated to my changes, AFAICS

[picnixz]

Can you add an explicit test checking the state of sys.modules please?

[picnixz]

Use a frozenset instead. Same in the except when you are running over the modules to remove them. At least it shows that the variable is meant to be readonly.

[godlygeek]

Can you add an explicit test checking the state of sys.modules please?

Can you elaborate on what you'd like this test to look like? I'm not sure I follow. At what point do you want sys.modules to be checked? Do you want it to be checked by the test body, or by the code inside the test root?

Oh, maybe I see: are you just suggesting that tests/roots/test-ext-autodoc/circular_import/c.py should have something like assert hasattr(circular_import, "a") to make it more explicit exactly what bug is being fixed? I've done that.

[picnixz]

Actually the test root is only some fixture directory so it should be tested in the test body. For instance, you check what sys.modules is before and after building (I think?).

[AA-Turner]

Tentatively targeting 7.2.5 for this, as it is a regression from 7.1.

a

[godlygeek]

Actually the test root is only some fixture directory so it should be tested in the test body. For instance, you check what sys.modules is before and after building (I think?).

OK, I've added a test that explicitly looks at sys.modules. I'm not sure if it's what you were picturing or not, so please take a look.

[picnixz]

Thank you. I was thinking about a more explicit check where you check that some module does not exist in sys.modules.

[AA-Turner]

Implementation looks good, please ping when the test suggested by @picnixz has been added for merging. Please also add an entry to CHANGES describing the bug fixed as I intend this to be included in 7.2.5.

A

[godlygeek]

Thank you. I was thinking about a more explicit check where you check that some module does not exist in sys.modules.

I don't understand...

The regression that this PR is fixing is that autodoc fails to import some modules. The buggy behavior that the new test would exhibit without my fix is that autodoc fails to import circular_import, and the corrected behavior, as well as the 7.1 behavior, is that it imports it successfully. I don't understand what sort of test you're picturing for this behavior that's based on a module being absent from sys.modules, when the whole goal of the change is to cause more modules to be imported than would be without the change.

[AA-Turner]

I'd understood it to mean checking that a half-imported module isn't left over, even if the entire import fails, but that might be difficult to set up a test-case for.

A

[godlygeek]

I'd understood it to mean checking that a half-imported module isn't left over, even if the entire import fails

This PR won't do that, though. It prevents a failed import done with typing.TYPE_CHECKING = True from polluting sys.modules in a way that causes a future import of the same module with typing.TYPE_CHECKING = False to fail. It stops failures in the first pass from bleeding into the second and causing failures, but doesn't prevent failures in the second pass from affecting sys.modules - if the module really can't be imported, even with typing.TYPE_CHECKING = False, then half-imported modules will still be present in sys.modules (just like they would have been in 7.1)

[AA-Turner]

Ahh! Then let's leave things as they are, just the CHANGES entry needed please if possible!

A

[picnixz]

Ah, my bad then, I understood wrongly.

[godlygeek]

@AA-Turner I've added that CHANGES entry

[AA-Turner]

Thanks -- I've credited you as Matt, just checking on Matt vs Matthew?

[godlygeek]

I prefer "Matt" - thanks for checking!

[AA-Turner]

Thanks @godlygeek!

A