Improve syntax error messages for keywords with typos

[pablogsal]

Currently, when users make typos in Python keywords, they receive generic "invalid syntax" error messages without any helpful suggestions about what might be wrong. This creates a frustrating experience, especially for beginners who might not immediately recognize that they've misspelled a keyword.

For example, typing whille True: instead of while True: would currently result in a generic syntax error without any hint that the problem is a misspelled keyword.

I propose to start raising errors like these:

==================================================

Traceback (most recent call last):
  File "/home/pablogsal/github/python/main/check.py", line 18, in test_mistyped_program
    exec(code_example)
    ~~~~^^^^^^^^^^^^^^
  File "<string>", line 2
    asynch def fetch_data():
    ^^^^^^
SyntaxError: invalid syntax. Did you mean 'async'?

==================================================

Traceback (most recent call last):
  File "/home/pablogsal/github/python/main/check.py", line 18, in test_mistyped_program
    exec(code_example)
    ~~~~^^^^^^^^^^^^^^
  File "<string>", line 6
    result = awaid fetch_data()
             ^^^^^
SyntaxError: invalid syntax. Did you mean 'await'?

==================================================

finally:
Traceback (most recent call last):
  File "/home/pablogsal/github/python/main/check.py", line 18, in test_mistyped_program
    exec(code_example)
    ~~~~^^^^^^^^^^^^^^
  File "<string>", line 6
    finaly:
    ^^^^^^
SyntaxError: invalid syntax. Did you mean 'finally'?

==================================================

Traceback (most recent call last):
  File "/home/pablogsal/github/python/main/check.py", line 18, in test_mistyped_program
    exec(code_example)
    ~~~~^^^^^^^^^^^^^^
  File "<string>", line 4
    raisee ValueError("Value cannot be negative")
    ^^^^^^
SyntaxError: invalid syntax. Did you mean 'raise'?

==================================================

Linked PRs

• gh-132449: Improve syntax error messages for keywords with typos #132450
• gh-132449: Improve the algorithm to detect typos in keywords #132837
• gh-132449: Add whatsnew entry for typos in keywords #132838

[pablogsal]

The implementation of my draft PR adds a keyword typo detection and suggestion mechanism to Python's syntax error handling:

1. Track statement locations in the parser: Store the location information of the last successfully parsed statement to provide critical context for error detection.

   • Added a location struct and last_stmt_location to the parser state
   • Register statement locations during parsing through new functions _PyPegen_register_stmt and _PyPegen_register_stmts
   • The parser also includes optionally the source if the parser was executed from a string as the string will be lost after the parser finishes.

   This tracking is crucial because after the parser fails, the original source context is effectively lost. Without embedding this information in the exception, we cannot accurately determine where the valid code ends and the problematic section begins.

2. Add metadata to SyntaxError objects: Extend SyntaxError to include additional metadata needed for improved error detection.

   • Added a metadata field to PySyntaxErrorObject to store statement location information
   • Pass this metadata to the exception when a syntax error occurs
   • This metadata indicates the last statement that worked so we can use this location later to know where is the range of problematic code (the range from the end of the last working statement to the failure line).

3. Implement keyword typo detection algorithm:

   • Added _find_keyword_typos() method to analyze syntax errors
   • The method extracts and tokenizes the code fragment between the last valid statement and the error point
   • For each NAME token, checks if it might be a typo of a Python keyword using string similarity
   • If a likely match is found, it attempts to compile the code with the suggested keyword
   • If successful, updates the error message with the suggestion
   • The code bails out if the source is too long or there are too many possible substitutions to try.

   This approach correctly handles the parsing context because it works with the preserved statement locations rather than trying to re-analyze the entire source after the error occurs

4. Enhance error reporting: Modify the displayed error message to include suggestions when keyword typos are detected.

[pablogsal]

@lysnikolaou what do you think of this approach?