Improve "Name already defined" error message #5067

sid-kap · 2018-05-17T03:04:39Z

In semanal.py, name_already_defined takes an optional original_ctx: Optional[SymbolTableNode] argument which is used to print the line number of the original context where a name was defined already.

In many cases, the code doesn't pass anything for this argument, even though there is an original context that makes sense. This PR does this for the name_already_defined call in add_symbol, as discussed in #4722.

A few questions if you guys think this is a good idea:

Should it also be done in add_symbol in semanal_pass1.py and semanal_pass3.py?
I only made this change to cover the example that @glenjamin brought up in "Name already defined" error is confusing #4722, but there are other places in semanal.py where adding this additional argument could be helpful:

mypy/mypy/semanal.py

Line 3177 in fae1f25

self.name_already_defined(name, context)

mypy/mypy/semanal.py

Line 3201 in fae1f25

self.name_already_defined(name, ctx)

The function is also called with just two parameters in _visit_func_def, _visit_overloaded_func_def, and analyze_lvalue, but I'm not sure if there is a original_ctx that makes sense to pass there.
Should I add tests?

JelleZijlstra · 2018-05-17T03:11:34Z

I think anywhere this message appears it's useful to give the original definition, if possible.

Tests would be good to add. You'll probably just have to change the existing tests though.

sid-kap · 2018-05-17T04:40:09Z

Thanks for taking a look at this! I updated the tests to what I think should be the correct output, but in many cases the line numbers it outputs aren't what I expected (so the tests are currently failing). I'll have to figure out why these are wrong.

sid-kap · 2018-05-17T21:23:15Z

So my code is failing on two tests because it's reporting the wrong line numbers. For imports, node.line always seems to be 1, rather than the line where import was called. So it always says Name 'x' already defined on line 1 if x was originally defined using an import.

Is anyone familiar with this issue? It seems like I either need to detect that the original definition was an import, and don't report a line number, or maybe somehow figure out on which line the import happened.

sid-kap · 2018-05-21T03:58:51Z

It looks like when SymbolTableNode is created in semanal.py for an import, it stores the node of the referenced module, not the node of the import statement:
https://github.com/python/mypy/blob/master/mypy/semanal.py#L1310-L1314
Therefore, symbol_table_node.node.line is the line of the imported module, which will always be 1.

So it seems that the options are either:

add a field to SymbolTableNode to store the import line numbers (to only be used if kind is MODULE_REF), or
before raising a "Name already defined" error, check if the original type is a module reference (i.e. check if existing.kind == MODULE_REF) and don't report the line number in that case.

Which would be the right thing to do? cc @JelleZijlstra

ilevkivskyi · 2018-05-25T00:50:49Z

Which would be the right thing to do?

I would just not show the additional note in this case. Anyway, conditional imports will require special support which will probably add later. I will now make a review, sorry for waiting so long with this.

sid-kap · 2018-05-25T00:57:15Z

No worries! I can make that change, the tests should pass after I fix that.

ilevkivskyi

Thank you for working on this! The more detailed errors look definitely better.

ilevkivskyi · 2018-05-25T00:52:41Z

mypy/semanal.py

@@ -406,7 +407,8 @@ def _visit_func_def(self, defn: FuncDef) -> None:
                        # Redefinition. Conditional redefinition is okay.
                        n = self.locals[-1][defn.name()].node
                        if not self.set_original_def(n, defn):
-                            self.name_already_defined(defn.name(), defn)
+                            self.name_already_defined(defn.name(), defn,
+                                                      self.locals[-1][defn.name()])


It looks like you missed one more case: global function definition, see else: # Top-level function below. You will likely need to patch also self.check_no_global in the same way you do above for self.name_already defined

I think self.check_no_global already does the right thing?

mypy/mypy/semanal.py

Line 3225 in 7eebb8f

self.name_already_defined(n, ctx, self.globals[n])

OK, if the previous context is always correct in check_no_global, then no action is required here.

ilevkivskyi · 2018-05-25T00:54:11Z

mypy/semanal.py

@@ -1848,7 +1850,12 @@ def analyze_lvalue(self, lval: Lvalue, nested: bool = False,
                self.type.names[lval.name] = SymbolTableNode(MDEF, v)
            elif explicit_type:
                # Don't re-bind types
-                self.name_already_defined(lval.name, lval)
+                global_def = self.globals.get(lval.name)
+                local_def = cast(SymbolTable, self.locals[-1]).get(lval.name) if self.locals and self.locals[-1] else None


If you rewrite it using an if statement, then the cast will be unnecessary.

ilevkivskyi · 2018-05-25T00:58:58Z

mypy/semanal.py

@@ -3230,10 +3237,16 @@ def name_not_defined(self, name: str, ctx: Context) -> None:
                self.add_fixture_note(fullname, ctx)

    def name_already_defined(self, name: str, ctx: Context,
-                             original_ctx: Optional[SymbolTableNode] = None) -> None:
+                             original_ctx: Optional[Union[SymbolTableNode, SymbolNode]] = None) -> None:


In general, it is not a good idea to use a union here, but it seems to me the old signature was "wrong", so it is fine in this case. Maybe you can add a short comment about this?

The old signature was actually correct, the function used to only be called with SymbolTableNodes. I added the option for passing a SymbolNode instead because in _visit_overloaded_func_def I only have a SymbolNode to pass it, not a SymbolTableNode.

So I guess to fix this, the options are

take Optional[SymbolNode] instead

take Optiona[SymbolTableNode] only, and in _visit_overloaded_func_def create a SymbolTableNode to pass to the function

Which do you think makes more sense?

I mean that SymbolTableNode is logically "wrong" because it is just an entry in the symbol table wrapper, while SymbolNode is an actual error context (well to be more precise, SymbolNodes are semantic objects, not syntactic, but at least they are closer, and should ideally point to the line of syntax node defining them). So I vote for SymbolNode.

Actually you might need .kind == MODULE_REF which is only available on SymbolTableNode. Potentially, this can be replaced by isinstance(node, (MypyFile, ImportedName)), but I think it is better to just leave it as is.

Okay, I'll leave it as-is then.

ilevkivskyi · 2018-05-25T00:59:57Z

mypy/semanal.py

+            else:
+                node = None
+            if node and node.line != -1:
+                extra_msg = ' on line {}'.format(node.line)
            else:
                extra_msg = ' (possibly by an import)'


Do we have enough tests for this case? If not you can add one more.

I'm not sure when this would happen (when node is None or node.line is -1). There don't seem to be any tests covering it, so maybe this never happens?

Actually, looks like there was one test where this happened. Do you know in what situations this happens? If so, I could add tests for those cases.

I think this might happen in case of an error or an unresolved circular import. If you can't find a simple additional test case then ignore this.

Hmm, looks like there's already a test case for that situation (testDuplicateDefFromImport in semanal-errors.test). So I guess we can leave this as-is.

ilevkivskyi · 2018-05-25T01:00:37Z

test-data/unit/semanal-errors.test

@@ -1347,7 +1347,7 @@ def dec(x: Any) -> Any:
 @dec
 def f() -> None:
    pass
-@dec  # E: Name 'f' already defined
+@dec  # E: Name 'f' already defined on line 4


Wow, good job updating all these error messages! 👍

ilevkivskyi

This is almost ready, I left few more suggestions.

ilevkivskyi · 2018-05-28T00:10:15Z

mypy/semanal.py

-                self.name_already_defined(lval.name, lval)
+                global_def = self.globals.get(lval.name)
+                if self.locals:
+                    locals_last = self.locals[-1]


@gvanrossum probably will be happy to see another case in mypy for PEP 572 :-)

@sid-kap You can ignore this comment.

ilevkivskyi · 2018-05-28T00:20:59Z

OK, it looks like the comments I added in this round are optional, so this can be merged as soon as you fix the lint error reported by Travis.

ilevkivskyi

LGTM now!

Print more context in add_symbol if name already defined

33c673c

Add line number logging in more places, update test outputs

f0c91b3

sid-kap added 4 commits May 16, 2018 23:51

Fix one more test

d99eb04

Add line numbers for all "Name already defined" errors, update tests

5cfb4dc

Fix more tests

880d6ba

Fix two more tests

12d5111

ilevkivskyi reviewed May 25, 2018

View reviewed changes

sid-kap added 2 commits May 26, 2018 11:12

All tests passing now, address @ilevkivskyi's feedback

cfa3408

Add '(possibly by an import)' back to error message

4a67f14

ilevkivskyi reviewed May 28, 2018

View reviewed changes

Fix lint warnings

7d4ef29

ilevkivskyi approved these changes May 28, 2018

View reviewed changes

ilevkivskyi merged commit 0447473 into python:master May 28, 2018

Uh oh!

Improve "Name already defined" error message #5067

Improve "Name already defined" error message #5067

Uh oh!

Conversation

sid-kap commented May 17, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JelleZijlstra commented May 17, 2018

Uh oh!

sid-kap commented May 17, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sid-kap commented May 17, 2018

Uh oh!

sid-kap commented May 21, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ilevkivskyi commented May 25, 2018

Uh oh!

sid-kap commented May 25, 2018

Uh oh!

ilevkivskyi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sid-kap May 26, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sid-kap May 26, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ilevkivskyi May 28, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ilevkivskyi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ilevkivskyi commented May 28, 2018

Uh oh!

ilevkivskyi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sid-kap commented May 17, 2018 •

edited

Loading

sid-kap commented May 17, 2018 •

edited

Loading

sid-kap commented May 21, 2018 •

edited

Loading

sid-kap May 26, 2018 •

edited

Loading

sid-kap May 26, 2018 •

edited

Loading

ilevkivskyi May 28, 2018 •

edited

Loading