Bacause the tokenizer only returns pointers to the beginning and the end of the token, calculating line numbers/column offsets is more complicated than needed.
Feature or enhancement
Instead of the tokenizer returning a token type and settings pointers, we wanna return the token type (remains as is) and then set a struct token that has the following information:
- Pointers to beginning and end
- Location information (
lineno, col_offset, etc.)
- Level (the level in the
parenstack)
This way the parser will have a much easier job of setting line numbers & column offsets in the generated AST numbers and will make some of our work on the f-strings parsing easier.