NotImplementedError raised by annotationlib (?) when importing classes derived from NamedTuple

[befeleme]

Bug report

Bug description:

With 3.14.0a3 there is an issue which affects all of the building block packages: setuptools, pip, wheel, causing failure to build of over 2800 packages in Fedora Linux total.

In cases I explored, the issue appears when importing classes derived from NamedTuple.
To reproduce in the Python 3.14.0a3 interpreter:

>>> import setuptools.package_index
Traceback (most recent call last):
  File "<python-input-18>", line 1, in <module>
    import setuptools.package_index
  File "/usr/lib/python3.14/site-packages/setuptools/package_index.py", line 1005, in <module>
    class Credential(NamedTuple):
    ...<12 lines>...
            return f'{self.username}:{self.password}'
  File "/usr/lib64/python3.14/typing.py", line 2971, in __new__
    types = annotationlib.call_annotate_function(original_annotate, annotationlib.Format.FORWARDREF)
  File "/usr/lib64/python3.14/annotationlib.py", line 613, in call_annotate_function
    result = func(Format.VALUE_WITH_FAKE_GLOBALS)
  File "/usr/lib/python3.14/site-packages/setuptools/package_index.py", line 1005, in __annotate__
    class Credential(NamedTuple):
    
NotImplementedError

The class in question looks like this: https://github.com/pypa/setuptools/blob/main/setuptools/package_index.py#L1003

CPython versions tested on:

3.14

Operating systems tested on:

Linux

Linked PRs

• gh-128089: Bump the PYC magic number #128097

[tomasr8]

cc @JelleZijlstra

[vstinner]

What is your setuptools version? I cannot reproduce the issue on the main branch with setuptools 75.6.0.

[vstinner]

I failed to reproduce the issue on Python 3.14 alpha4 with setuptools 75.6.0:

wget https://www.python.org/ftp/python/3.14.0/Python-3.14.0a3.tar.xz
tar -xf Python-3.14.0a3.tar.xz 
cd Python-3.14.0a3/
./configure --prefix /opt/py314 && make && make install
/opt/py314/bin/python3 -m pip install setuptools
/opt/py314/bin/python3  -c 'import setuptools.package_index'

[DavidCEllis]

I can't reproduce this either on 3.14.0a3. It looks like something is out of sync or hasn't rebuilt?

The values of the Format Enum changed from 3.14.0a2 to 3.14.0a3. In a2, the value '2' meant Format.FORWARDREF, in a3 this is now Format.VALUE_WITH_FAKE_GLOBALS. (FORWARDREF is now 3).

Calling 'func' with FORWARDREF would give this NotImplementedError, and so calling it with Format.VALUE_WITH_FAKE_GLOBALS would also do so if something hasn't updated (for example you would see this if you used annotationlib from 3.14.0a3 on a build of 3.14.0a2).

[frenzymadness]

Is it possible that when the module is cached into pyc file (during our RPM builds) the numeric value under annotationlib.Format.* is stored as the respective integer somewhere in the code object of __annotate__ function? And then, when we run the code from the pyc file with newer Python, it expects the new value there but the old one is stored from the build time, right?

I've tried to find the value stored somewhere but did not succeed so I'm asking whether the approach was correct or the problem is completely different.

[DavidCEllis]

I think it's the other way around. Your annotationlib is up to date, but something about the runtime is outdated?

Can you try creating a class and calling its __annotate__ function with the value 2 and seeing what the output is?

EG:

class X:
    a: int

X.__annotate__(2)

See if you get the same NotImplementedError. In 3.14.0a3 this should return the annotation dict, in 3.14.0a2 you'll get the error.

---

3.14.0a2

>>> class X:
...     a: int
...
>>> X.__annotate__(2)
Traceback (most recent call last):
  File "<python-input-1>", line 1, in <module>
    X.__annotate__(2)
    ~~~~~~~~~~~~~~^^^
  File "<python-input-0>", line 1, in __annotate__
    class X:

NotImplementedError

3.14.0a3

>>> class X:
...     a: int
...
>>> X.__annotate__(2)
{'a': <class 'int'>}

This is due to the enum change:

The enum values:

3.14.0a2

>>> import annotationlib
>>> list(annotationlib.Format)
[<Format.VALUE: 1>, <Format.FORWARDREF: 2>, <Format.STRING: 3>]

3.14.0a3

>>> import annotationlib
>>> list(annotationlib.Format)
[<Format.VALUE: 1>, <Format.VALUE_WITH_FAKE_GLOBALS: 2>, <Format.FORWARDREF: 3>, <Format.STRING: 4>]

[befeleme]

It looks there was something out of sync: I rebootstrapped the whole environment and don't see the issue in the newly built packages. It only appears if anything in the test environment runs the NamedTuple code path in packages that were built with alpha2. Our Python behaves correctly as alpha3 according to the provided code snippets.

[vstinner]

Ok, I reproduced the issue. IMO we should bump PYC serial number

Code bug.py:

from typing import NamedTuple

class Credential(NamedTuple):
    username: str
    password: str
    def __str__(self) -> str:
        return f'{self.username}:{self.password}'

Steps to reproduce:

• Install Python 3.14 alpha2 (ex: install in /opt/py314a2)
• Install Python 3.14 alpha3 (ex: install in /opt/py314)
• Create the PYC file using alpha2: /opt/py314a2/bin/python3 -c 'import bug'
• Run the script using alpha3, using alpha2 PYC: /opt/py314/bin/python3 -c 'import bug'

Output:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import bug
  File "/home/vstinner/bug/bug.py", line 3, in <module>
    class Credential(NamedTuple):
    ...<3 lines>...
            return f'{self.username}:{self.password}'
  File "/opt/py314/lib/python3.14/typing.py", line 2971, in __new__
    types = annotationlib.call_annotate_function(original_annotate, annotationlib.Format.FORWARDREF)
  File "/opt/py314/lib/python3.14/annotationlib.py", line 613, in call_annotate_function
    result = func(Format.VALUE_WITH_FAKE_GLOBALS)
  File "/home/vstinner/bug/bug.py", line 3, in __annotate__
    class Credential(NamedTuple):
    
NotImplementedError

[vstinner]

IMO we should bump PYC serial number

The problem is that Python 3.14 alpha3 is already released. I'm not sure if it's still relevant to bump the serial number afterwards?

[JelleZijlstra]

Sorry for this, I should indeed have bumped the magic number. I sort of assumed it was harmless to not bump the magic number for every individual change in the alpha phase. Maybe it would be easier to just bump the number at every alpha release and not worry about it for individual changes.

[DavidCEllis]

Might be worth bumping it before 3.14a4 - to avoid anyone running into this if they've skipped a3?

I didn't realise the generated __annotate__ function was stored in the .pyc but this makes sense now.

[hroncok]

At least for Fedora 3.14 efforts, if you bump the magic number now, we can backport the change and force the pyc files generated with a2 to be ignored.

[Paciupa]

I think I faced with this bug. I've just upgraded py3.14 from a2 to a3. And when I run py -3.14 -m pip, I've got

exception

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\Jan\AppData\Local\Programs\Python\Python314\Lib\site-packages\pip\__main__.py", line 22, in <module>
    from pip._internal.cli.main import main as _main
  File "C:\Users\Jan\AppData\Local\Programs\Python\Python314\Lib\site-packages\pip\_internal\cli\main.py", line 11, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "C:\Users\Jan\AppData\Local\Programs\Python\Python314\Lib\site-packages\pip\_internal\cli\autocompletion.py", line 10, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "C:\Users\Jan\AppData\Local\Programs\Python\Python314\Lib\site-packages\pip\_internal\cli\main_parser.py", line 9, in <module>
    from pip._internal.build_env import get_runnable_pip
  File "C:\Users\Jan\AppData\Local\Programs\Python\Python314\Lib\site-packages\pip\_internal\build_env.py", line 18, in <module>
    from pip._internal.cli.spinners import open_spinner
  File "C:\Users\Jan\AppData\Local\Programs\Python\Python314\Lib\site-packages\pip\_internal\cli\spinners.py", line 9, in <module>
    from pip._internal.utils.logging import get_indentation
  File "C:\Users\Jan\AppData\Local\Programs\Python\Python314\Lib\site-packages\pip\_internal\utils\logging.py", line 13, in <module>
    from pip._vendor.rich.console import (
    ...<6 lines>...
    )
  File "C:\Users\Jan\AppData\Local\Programs\Python\Python314\Lib\site-packages\pip\_vendor\rich\console.py", line 47, in <module>
    from . import errors, themes
  File "C:\Users\Jan\AppData\Local\Programs\Python\Python314\Lib\site-packages\pip\_vendor\rich\themes.py", line 1, in <module>
    from .default_styles import DEFAULT_STYLES
  File "C:\Users\Jan\AppData\Local\Programs\Python\Python314\Lib\site-packages\pip\_vendor\rich\default_styles.py", line 3, in <module>
    from .style import Style
  File "C:\Users\Jan\AppData\Local\Programs\Python\Python314\Lib\site-packages\pip\_vendor\rich\style.py", line 8, in <module>
    from .color import Color, ColorParseError, ColorSystem, blend_rgb
  File "C:\Users\Jan\AppData\Local\Programs\Python\Python314\Lib\site-packages\pip\_vendor\rich\color.py", line 8, in <module>
    from ._palettes import EIGHT_BIT_PALETTE, STANDARD_PALETTE, WINDOWS_PALETTE
  File "C:\Users\Jan\AppData\Local\Programs\Python\Python314\Lib\site-packages\pip\_vendor\rich\_palettes.py", line 1, in <module>
    from .palette import Palette
  File "C:\Users\Jan\AppData\Local\Programs\Python\Python314\Lib\site-packages\pip\_vendor\rich\palette.py", line 5, in <module>
    from .color_triplet import ColorTriplet
  File "C:\Users\Jan\AppData\Local\Programs\Python\Python314\Lib\site-packages\pip\_vendor\rich\color_triplet.py", line 4, in <module>
    class ColorTriplet(NamedTuple):
    ...<33 lines>...
            return red / 255.0, green / 255.0, blue / 255.0
  File "C:\Users\Jan\AppData\Local\Programs\Python\Python314\Lib\typing.py", line 2971, in __new__
    types = annotationlib.call_annotate_function(original_annotate, annotationlib.Format.FORWARDREF)
  File "C:\Users\Jan\AppData\Local\Programs\Python\Python314\Lib\annotationlib.py", line 613, in call_annotate_function
    result = func(Format.VALUE_WITH_FAKE_GLOBALS)
  File "C:\Users\Jan\AppData\Local\Programs\Python\Python314\Lib\site-packages\pip\_vendor\rich\color_triplet.py", line 4, in __annotate__
    class ColorTriplet(NamedTuple):

but when I call pip directly `. 'C:\Users\Jan\AppData\Local\Programs\Python\Python313\Scripts\pip.exe' list` (PowerShell), everything is fine:

Package Version
------- -------
pip     24.3.1

[vstinner]

Python 3.14.0 alpha 4 scheduled next week (Tuesday, 2025-01-14) will fix the issue by changing the PYC magic number.

In the meanwhile, the workaround is to manually remove all Python 3.14 PYC files.

The magic number 3610 is now assigned to annotationlib VALUE_WITH_FAKE_GLOBALS.

I close the issue.