Skip to content

Treat ASCII-only literals as str when using unicode_literals #3648

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 4 commits into from
Closed

Treat ASCII-only literals as str when using unicode_literals #3648

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
8c343a1
Treat ASCII-only literals as bytes even with unicode_literals
ilevkivskyi Jul 2, 2017
ff699d9
Relax check for future import; try to fix Windows encoding problem
ilevkivskyi Jul 2, 2017
6ece9f9
Add more tests as per CR
ilevkivskyi Jul 2, 2017
144041f
Merge branch 'master' into unicode_literals
ilevkivskyi Aug 30, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions mypy/fastparse2.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,7 @@ def __init__(self,
self.options = options
self.is_stub = is_stub
self.errors = errors
self.unicode_literals = False

def fail(self, msg: str, line: int, column: int) -> None:
self.errors.report(line, column, msg)
Expand Down Expand Up @@ -653,6 +654,8 @@ def visit_ImportFrom(self, n: ast27.ImportFrom) -> ImportBase:
n.level,
[(a.name, a.asname) for a in n.names])
self.imports.append(i)
if n.module == '__future__' and any(nm.name == 'unicode_literals' for nm in n.names):
self.unicode_literals = True
return i

# Global(identifier* names)
Expand Down Expand Up @@ -880,6 +883,8 @@ def visit_Str(self, s: ast27.Str) -> Expression:
contents = str(n)[2:-1]
return StrExpr(contents)
else:
if self.unicode_literals and all(ord(c) < 128 for c in s.s): # ASCII only
return StrExpr(s.s)
return UnicodeExpr(s.s)

# Ellipsis
Expand Down
30 changes: 27 additions & 3 deletions test-data/unit/python2eval.test
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -151,15 +151,39 @@ f(**params)
[out]

[case testFromFutureImportUnicodeLiterals2_python2]
# coding=utf-8
from __future__ import unicode_literals
def f(x): # type: (str) -> None
pass
f(b'')
f(u'')
f('')
f(u'щось')
f('щось інше')
[out]
_program.py:5: error: Argument 1 to "f" has incompatible type "unicode"; expected "str"
_program.py:6: error: Argument 1 to "f" has incompatible type "unicode"; expected "str"
_program.py:7: error: Argument 1 to "f" has incompatible type "unicode"; expected "str"

[case testUnicodeLiteralsFutureImportAllASCII_python2]
# coding=utf-8
from __future__ import unicode_literals
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add tests for some of the other syntactic variations, like from __future__ import absolute_import, unicode_literals and from __future__ import unicode_literals as something_else?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, will do.

x = None # type: str
x = 'не латиниця' # Error
x = 'only ascii' # This is however OK
reveal_type('another ascii')
[out]
_program.py:4: error: Incompatible types in assignment (expression has type "unicode", variable has type "str")
_program.py:6: error: Revealed type is 'builtins.str'

[case testUnicodeLiteralsFutureImportAllASCIIvariant1_python2]
from __future__ import absolute_import, unicode_literals
x = None # type: str
x = 'only ascii'
[out]

[case testUnicodeLiteralsFutureImportAllASCIIvariant2_python2]
from __future__ import unicode_literals as something_else
x = None # type: str
x = 'only ascii'
[out]

[case testStrUnicodeCompatibility_python2]
import typing
Expand Down