Implement PEP 526 Variable Annotations Syntax #2131

ilevkivskyi · 2016-09-14T12:54:47Z

Here is an accompanying change for python/typed_ast#16

Github says it cannot automatically merge, probably because I updated type of ast35.Assign.type_comment to Union[AST, Optional[str]] in typeshed. I don't know how to fix this.
I have made a PR to typeshed with this commit.

All mypy tests pass locally with both versions (current and PEP 526) of typed_ast.

gvanrossum · 2016-09-14T15:02:18Z

Maybe the merge conflict is due to the typeshed update. Try leaving that out.

ilevkivskyi · 2016-09-14T16:01:27Z

@gvanrossum
You are right, I have leaved typeshed update, and now everything is OK.

But don't forget that actual implementation is in python/typed_ast#16

gvanrossum · 2016-09-14T21:49:06Z

So this works... A little too well! It now also accepts the new syntax in Python 3.3--3.5.

Also, maybe 'ast35' is now a misnomer? (That's mostly a Q. for @ddfisher who's out this week.)

gvanrossum · 2016-09-14T21:53:34Z

Could you add some unit tests too?

ilevkivskyi · 2016-09-14T21:57:36Z

@gvanrossum

So this works... A little too well! It now also accepts the new syntax in Python 3.3--3.5.

The simplest way to fix this is to make two modules ast35 and ast36. ast36 will accept PEP 526 (possibly also async generators and async comprehensions). ast35 will need to be changed only slightly to have the same type Str for type_comment, (and some changes due to how None is treated).

Could you add some unit tests too?

OK, will do this. I just need to figure out first how to skip tests depending on version of typed_ast.

gvanrossum · 2016-09-14T22:03:43Z

I'd rather not have the ast modules multiplying like rabbits... The plan has always been to accept the most recent syntax at the grammar level and to complain later in the pipeline. If you try `async def` with `--python-version 3.4` you should get an example of this.

ilevkivskyi · 2016-09-14T22:07:26Z

@gvanrossum

I'd rather not have the ast modules multiplying like rabbits...

I agree, I will add version dependent checks.

I have not yet finished the new semantics of prohibiting x: int = None. For some reason it is still not flagged inside functions.

…date

…'s on the RHS

…ional mode

ddfisher · 2016-09-21T16:46:12Z

mypy/checker.py

@@ -1076,7 +1076,8 @@ def check_assignment(self, lvalue: Node, rvalue: Node, infer_lvalue_type: bool =
                elif (is_literal_none(rvalue) and
                        isinstance(lvalue, NameExpr) and
                        isinstance(lvalue.node, Var) and
-                        lvalue.node.is_initialized_in_class):
+                        lvalue.node.is_initialized_in_class and
+                        not experiments.STRICT_OPTIONAL):


If we don't want this behavior, the check should just be deleted because Nones are subtypes of all other types when not in strict Optional mode, so this check is unnecessary.

That said, I think what we really want is to update this check to be more permissive rather than doing AST replacement at the parser stage.

The idea here is to prohibit things like x: int = None. Probably you are right that this could be done by just removing the check. I will try this and if everything will work, I just remove this check in the next commit.

Concerning the AST replacement in the parser, see my last comment below.

ddfisher · 2016-09-21T16:53:00Z

mypy/types.py

+def replace_none(n: Node) -> Node:
+    """
+    Helper for parse and fastparse.
+    Replace all Nones on the right of protected assignment with Any-like.


I don't see the term "protected assignment" in PEP 484. Could you define it here?

@ddfisher
Currently mypy complains about assignments like this

x = None # type: int

with --strict-optional. Guido wants this to be changed. I use the term "protected" here with meaning "protected against complains". Since when someone writes the code as above the reason is not to actually assign None, this is just a limitation of the old syntax.

I will add this explanation to the docstring in the next commit.

ddfisher · 2016-09-21T16:55:30Z

mypy/fastparse.py

+            rvalue = self.visit(n.value)
+        lvalues = self.visit_list(n.targets)
+        # protect 'x, y = None, None # type: int, str' etc.
+        if not n_synt and typ is not None and experiments.STRICT_OPTIONAL:


This understanding should be part of a later stage rather than in the parser. It's better if (as much as possible) the parser faithfully reports the AST of the actual code.

@ddfisher
The idea here is to synchronize the outcome of the old syntax and the new syntax. Guido wanted (and I agree with him) the outcome of these two to be identical:

x: int x = None # type: int

The easiest way is to make them equivalent at this stage (if someone wants a more precise information it is still provided by typed_ast). Otherwise there will be checks all the way down whether a particular None is a "real" one or just a limitation of the old syntax.

The None is a real one, though, and you can't be sure that it's just being used due to limitations of the old syntax -- it maybe intended to be used as an actual assignment. Also, I don't think you'll need check in a bunch of different places. I think modifying the if statement I commented on in checker.py should be sufficient.

@ddfisher @gvanrossum
I am not 100% sure about the interpretation of the old syntax. However, as I understood, Guido is sure. And he wants this

x = None # type: int

to be interpreted as a not real None.

Concerning implementation I will just try to move it out of here and if everything works then OK.

ddfisher · 2016-09-21T17:19:47Z

I'd rather not have the ast modules multiplying like rabbits... The plan
has always been to accept the most recent syntax at the grammar level and
to complain later in the pipeline.

As I recall, the plan was for the ast modules to multiply. That's why ast35 was named ast35 instead of ast3. New Python versions aren't released that often, and I'm a bit reluctant to e.g. include beta code in the ast35 parser. I also think it's valuable for clients to be able to parse specific versions of Python, though I'm open to being convinced that's not the case.

gvanrossum · 2016-09-21T18:22:37Z

David and I just chatted about this and I think he's convinced me not to change mypy's behavior for type comments with None assignment -- at least not as part of this PR. Maybe you can file a separate issue about the discrepancy between mypy and PEP 484. But "support PEP 526" and "change mypy to conform more strictly to PEP 484 in this area" are clearly different tasks and shouldn't be done in a single PR.

ilevkivskyi · 2016-09-21T18:26:28Z

@gvanrossum @ddfisher

at least not as part of this PR

OK, I agree, let us consider this separately.
I will then only leave the necessary changes in the PR in the next commit.

ilevkivskyi · 2016-09-21T19:13:44Z

@ddfisher
OK, I pushed a new commit. Now only the PEP 526 related stuff is in. There is only one line added apart from fastparse.py and tests: The special-casing of class bodies in checker.py will not affect the new syntax, so that x: int = None will be an error everywhere.

If you don't like this, then I could just remove this special-casing altogether. Please take a look.

gvanrossum · 2016-09-21T19:17:57Z

That is correct. The PEP 526 syntax doesn't have this exception.

…

--Guido (mobile)

ddfisher

Looks good -- just a couple small things left to fix. Thanks!

ddfisher · 2016-09-26T18:50:07Z

mypy/checker.py

@@ -1076,7 +1076,8 @@ def check_assignment(self, lvalue: Node, rvalue: Node, infer_lvalue_type: bool =
                elif (is_literal_none(rvalue) and
                        isinstance(lvalue, NameExpr) and
                        isinstance(lvalue.node, Var) and
-                        lvalue.node.is_initialized_in_class):
+                        lvalue.node.is_initialized_in_class and
+                        not self.options.fast_parser):


This isn't correct -- it disallows the following (when run with the fast parser):

class C: x = None # type: int

Instead, I think you'll need to add a flag for the new assignment syntax to AssignmentStmt and check for that here.

ddfisher · 2016-09-26T18:50:51Z

mypy/test/testcheck.py

@@ -67,6 +68,11 @@
    'check-newtype.test',
 ]

+if hasattr(typed_ast, '__version__'):
+    version_info = tuple(map(int, typed_ast.__version__.split('.')))  # type: ignore
+    if version_info >= (0, 5, 7):


python/typed_ast#16 contains an interface change, so it's actually going to be version 0.6.0 -- could you bump that here?

ilevkivskyi · 2016-09-26T18:54:00Z

@ddfisher
Thank you for review! Will fix this soon (I am on a train right now).

ddfisher

Thanks for fixing this up! Just a few small nits remain.

ddfisher · 2016-09-27T22:13:42Z

mypy/fastparse.py

-            else:
-                typ = TypeConverter(line=n.lineno).visit(n.type_comment)
+                                        'comment instead', n.lineno, n.col_offset)
+        if n.type_comment is not None:


We should display an error if there's a type comment and a type annotation.

ddfisher · 2016-09-27T22:52:48Z

mypy/nodes.py

@@ -761,12 +761,14 @@ class AssignmentStmt(Statement):
    rvalue = None  # type: Expression
    # Declared type in a comment, may be None.
    type = None  # type: mypy.types.Type
+    n_synt = False  # type: bool


Could you add a comment about what this means? Also, I think new_syntax would be slightly clearer than the abbreviated n_synt.

ddfisher · 2016-09-28T00:00:23Z

mypy/test/testcheck.py

@@ -67,6 +68,11 @@
    'check-newtype.test',
 ]

+if hasattr(typed_ast, '__version__'):
+    version_info = tuple(map(int, typed_ast.__version__.split('.')))  # type: ignore
+    if version_info >= (0, 6, 0):


It'd be better to do a functionality check instead of this version check (especially now that typed_ast no longer exports a version). You could do so like this:

import typed_ast.ast35 if 'annotation' in typed_ast.ast35.Assign._fields: ...

ddfisher · 2016-09-28T00:01:39Z

mypy/fastparse.py

+            raise TypeCommentParseError('Variable annotation syntax is only '
+                                        'suppoted in Python 3.6, use type '
+                                        'comment instead', n.lineno, n.col_offset)
+        if n.type_comment is not None:


Could you add a comment noting that typed_ast prevents there from being both a type comment and an annotation?

ilevkivskyi · 2016-09-28T12:59:53Z

@ddfisher
I implemented the latest comments here and in typeshed PR. Concerning both type comment and type annotation, typed_ast prohibits this with

SyntaxError: misplaced type annotation

so that I just added a comment in fastparse.py

ddfisher · 2016-09-28T17:28:44Z

Ah, yep. I meant to delete the comment about it when I noticed it wasn't possible, but I guess I forgot.

Thanks for responding to all the feedback! This looks good now!

ilevkivskyi added 2 commits September 13, 2016 22:31

Start PEP 526 implementation

32295da

Backward compatibility for typed_ast

be2ba3a

ilevkivskyi force-pushed the master branch from be2ba3a to 915e9cc Compare September 14, 2016 15:44

ilevkivskyi added 4 commits September 15, 2016 00:21

Implement new semantics for None

1fb5787

sync parse with fastparse; prohibit var annotations before 3.6

22ac4e8

Fix failing tests; some polishing

b55b9ce

Temporarily silence mypy complain; could be removed after typeshed up…

59ffb2c

…date

ilevkivskyi force-pushed the master branch from 915e9cc to 59ffb2c Compare September 15, 2016 14:18

ilevkivskyi added 7 commits September 15, 2016 16:20

Restore testing option for mypy

f02c9e7

Merge to get newest typeshed submodule rev

2a16bc8

Add unit tests for PEP 526 syntax

0e5bf09

Add one more unit test

0fca70f

Imply strict-optional also in classes; fix problem with multiple None…

4a0a440

…'s on the RHS

Be very picky about what is protected against complains in strict opt…

fe94105

…ional mode

Restore mypy's testing option

0f76f23

ddfisher suggested changes Sep 21, 2016

View reviewed changes

Leave only PEP 526 stuff in

aa6c5ba

ddfisher mentioned this pull request Sep 26, 2016

Implement PEP 526 Variable Annotations Syntax python/typed_ast#16

Merged

ddfisher reviewed Sep 26, 2016

View reviewed changes

ilevkivskyi added 3 commits September 27, 2016 21:36

Backward compatible PEP 526 implementation + response to comments

d2868cc

update for TypeCommentParseError

4807cc0

Merge remote-tracking branch 'upstream/master'

11951d2

ddfisher suggested changes Sep 28, 2016

View reviewed changes

ilevkivskyi added 2 commits September 28, 2016 14:38

Updates and response to comments

7b05b49

Remove redundant if

7b46161

ddfisher merged commit 5f0d02c into python:master Sep 28, 2016

Uh oh!

Implement PEP 526 Variable Annotations Syntax #2131

Implement PEP 526 Variable Annotations Syntax #2131

Uh oh!

Conversation

ilevkivskyi commented Sep 14, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gvanrossum commented Sep 14, 2016

Uh oh!

ilevkivskyi commented Sep 14, 2016

Uh oh!

gvanrossum commented Sep 14, 2016

Uh oh!

gvanrossum commented Sep 14, 2016

Uh oh!

ilevkivskyi commented Sep 14, 2016

Uh oh!

gvanrossum commented Sep 14, 2016 via email

Uh oh!

ilevkivskyi commented Sep 14, 2016

Uh oh!

ddfisher Sep 21, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ddfisher Sep 21, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ddfisher commented Sep 21, 2016

Uh oh!

gvanrossum commented Sep 21, 2016 via email

Uh oh!

ilevkivskyi commented Sep 21, 2016

Uh oh!

ilevkivskyi commented Sep 21, 2016

Uh oh!

gvanrossum commented Sep 21, 2016 via email

Uh oh!

ddfisher left a comment

Choose a reason for hiding this comment

Uh oh!

ddfisher Sep 26, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ilevkivskyi commented Sep 26, 2016

Uh oh!

ddfisher left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ilevkivskyi commented Sep 28, 2016

Uh oh!

ddfisher commented Sep 28, 2016

Uh oh!

Uh oh!

ilevkivskyi commented Sep 14, 2016 •

edited

Loading

ddfisher Sep 21, 2016 •

edited

Loading

ddfisher Sep 21, 2016 •

edited

Loading

ddfisher Sep 26, 2016 •

edited

Loading