Skip to content

Fix parser when using Python 3.9 #8716

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Apr 25, 2020
Merged

Fix parser when using Python 3.9 #8716

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
43cb4eb
Fix parser when using Python 3.9
gvanrossum Apr 23, 2020
8ac365b
Reject things like Dict[x:y] or Dict[x, y:z]
gvanrossum Apr 23, 2020
2d5a144
Attempt to fix some trivial test failures under Python 3.9
gvanrossum Apr 24, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 21 additions & 8 deletions mypy/fastparse.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,8 @@
import ast as ast3
assert 'kind' in ast3.Constant._fields, \
"This 3.8.0 alpha (%s) is too old; 3.8.0a3 required" % sys.version.split()[0]
# TODO: Num, Str, Bytes, NameConstant, Ellipsis are deprecated in 3.8.
# TODO: Index, ExtSlice are deprecated in 3.9.
from ast import (
AST,
Call,
Expand Down Expand Up @@ -1503,19 +1505,30 @@ def visit_Bytes(self, n: Bytes) -> Type:
contents = bytes_to_human_readable_repr(n.s)
return RawExpressionType(contents, 'builtins.bytes', self.line, column=n.col_offset)

# Subscript(expr value, slice slice, expr_context ctx)
# Subscript(expr value, slice slice, expr_context ctx) # Python 3.8 and before
# Subscript(expr value, expr slice, expr_context ctx) # Python 3.9 and later
def visit_Subscript(self, n: ast3.Subscript) -> Type:
if not isinstance(n.slice, Index):
self.fail(TYPE_COMMENT_SYNTAX_ERROR, self.line, getattr(n, 'col_offset', -1))
return AnyType(TypeOfAny.from_error)
if sys.version_info >= (3, 9): # Really 3.9a5 or later
sliceval = n.slice # type: Any
if (isinstance(sliceval, ast3.Slice) or
(isinstance(sliceval, ast3.Tuple) and
any(isinstance(x, ast3.Slice) for x in sliceval.elts))):
self.fail(TYPE_COMMENT_SYNTAX_ERROR, self.line, getattr(n, 'col_offset', -1))
return AnyType(TypeOfAny.from_error)
else:
# Python 3.8 or earlier use a different AST structure for subscripts
if not isinstance(n.slice, Index):
self.fail(TYPE_COMMENT_SYNTAX_ERROR, self.line, getattr(n, 'col_offset', -1))
return AnyType(TypeOfAny.from_error)
sliceval = n.slice.value

empty_tuple_index = False
if isinstance(n.slice.value, ast3.Tuple):
params = self.translate_expr_list(n.slice.value.elts)
if len(n.slice.value.elts) == 0:
if isinstance(sliceval, ast3.Tuple):
params = self.translate_expr_list(sliceval.elts)
if len(sliceval.elts) == 0:
empty_tuple_index = True
else:
params = [self.visit(n.slice.value)]
params = [self.visit(sliceval)]

value = self.visit(n.value)
if isinstance(value, UnboundType) and not value.args:
Expand Down
4 changes: 4 additions & 0 deletions mypy/test/helpers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -233,6 +233,8 @@ def clean_up(a: List[str]) -> List[str]:
remove trailing carriage returns.
"""
res = []
pwd = os.getcwd()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What was going on here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Many tests suddenly showed the full pathname in the traceback instead of simply File "driver.py", line NNN. I don't actually know why, but I could imagine that somehow an absolute path was computed by Python where it previously believed the pathname on the command line? It's always about "driver.py" which I presume is the synthetic script created by the test harness.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Playing around, this just seems new behavior in Python 3.9, so I think my fix is the best we can do.

driver = pwd + '/driver.py'
for s in a:
prefix = os.sep
ss = s
Expand All @@ -241,6 +243,8 @@ def clean_up(a: List[str]) -> List[str]:
ss = ss.replace(p, '')
# Ignore spaces at end of line.
ss = re.sub(' +$', '', ss)
# Remove pwd from driver.py's path
ss = ss.replace(driver, 'driver.py')
res.append(re.sub('\\r$', '', ss))
return res

Expand Down
5 changes: 4 additions & 1 deletion mypyc/test-data/run.test
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4352,7 +4352,10 @@ import sys

# We lie about the version we are running in tests if it is 3.5, so
# that hits a crash case.
if sys.version_info[:2] == (3, 8):
if sys.version_info[:2] == (3, 9):
def version() -> int:
return 9
elif sys.version_info[:2] == (3, 8):
def version() -> int:
return 8
elif sys.version_info[:2] == (3, 7):
Expand Down