tokenize.generate_tokens no longer reports lineno for indentation errors

[nedbat]

tokenize.generate_tokens raises an exception for indentation errors. It used to populate the .lineno attribute of the exception, but as of yesterday, .lineno is None.

Test file:

# parsebug.py
import io
import sys
import tokenize

text = """\
0 spaces
  2
 1
"""

print(sys.version)
readline = io.StringIO(text).readline
try:
    list(tokenize.generate_tokens(readline))
except Exception as exc:
    print(f"{exc.lineno = }")
    raise

With 3.12.0a7:

% .tox/py312/bin/python parsebug.py
3.12.0a7 (main, Apr  5 2023, 05:51:58) [Clang 14.0.3 (clang-1403.0.22.14.1)]
exc.lineno = 3
Traceback (most recent call last):
  File "/Users/nedbatchelder/coverage/trunk/parsebug.py", line 15, in <module>
    list(tokenize.generate_tokens(readline))
  File "/usr/local/pyenv/pyenv/versions/3.12.0a7/lib/python3.12/tokenize.py", line 516, in _tokenize
    raise IndentationError(
  File "<tokenize>", line 3
    1
IndentationError: unindent does not match any outer indentation level

With a nightly 3.12 build:

% .tox/anypy/bin/python parsebug.py
3.12.0a7+ (heads/main:9bc80dac47, May 22 2023, 05:34:19) [Clang 14.0.3 (clang-1403.0.22.14.1)]
exc.lineno = None
Traceback (most recent call last):
  File "/Users/nedbatchelder/coverage/trunk/parsebug.py", line 15, in <module>
    list(tokenize.generate_tokens(readline))
  File "/usr/local/cpython/lib/python3.12/tokenize.py", line 451, in _tokenize
    for token in _generate_tokens_from_c_tokenizer(source, extra_tokens=True):
  File "/usr/local/cpython/lib/python3.12/tokenize.py", line 542, in _generate_tokens_from_c_tokenizer
    for info in c_tokenizer.TokenizerIter(source, extra_tokens=extra_tokens):
IndentationError: unindent does not match any outer indentation level (<tokenize>, line 3)

The line number is clearly known, since it's reported. Is there another way I should be getting the line number from the exception?

/cc @mgmacias95

Linked PRs

• gh-104741: Add line number attribute to indentation error exception #104743

[pablogsal]

Is there another way I should be getting the line number from the exception?

Nop, this is something we need to fix on our side. Thanks for raising it with us!

@mgmacias95 will make a PR shortly!

[nedbat]

I can confirm that #104743 fixes this.