Text, on Python 2, includes bytes?

[matthiaskramm]

• typing.Text, on Python 2, maps to unicode.
• unicode is automatically promoted to Union[str, unicode], see special rules in typeshed pyi typeshed#270

The combination of these two rules means that an annotation Text now includes bytes, on Python 2.

And indeed, the below type-checks, with both pytype and mypy --python-version=2.7:

import typing
def f(x):
  # (typing.Text) -> None
  pass
f(b"foo")

This seems a bit odd to me. It also means that we don't have a cross-version way to express "unicode string". Should Text not do the auto-promotion, so that it only stands for unicode itself, on Python 2?

[vlasovskikh]

@matthiaskramm There was #208 with lots of ideas, none of them were found satisfactory.

[matthiaskramm]

Right, so Jukka's proposal #208 (comment) mentions something along the same lines:

If a function accepts unicode arguments but no str arguments in Python 2, the Python 2 argument type should probably be Text.

This reads to me as if he was going by the assumption that that's what Text currently means, which makes the current behavior of both mypy and pytype a bug.

[ilevkivskyi]

By the way, at some point I modified the Python 2 parser in typed_ast to distinguish between "ascii chars" and b"ascii chars". @gvanrossum was trying to use it somehow, but I don't remember what was the outcome.

[gvanrossum]

By the way, at some point I modified the Python 2 parser in typed_ast to distinguish between "ascii chars" and b"ascii chars". @gvanrossum was trying to use it somehow, but I don't remember what was the outcome.

IIUC that code is still there, but we ended up not using it because things got too complicated.

The situation is still imperfect, but none of the alternatives seems to really solve everything, so we're keeping the status quo. It's perhaps unfortunate that (in mypy's PY2 mode) you can't distinguish between a function that takes only unicode and one that takes either unicode or bytes (== str), because regardless of whether you spell it Text or unicode it'll accept str/bytes instances too. But the situation is pretty rare. More important IMO is that you can express that something doesn't take unicode, by writing bytes (or str, but bytes makes the intent more clear).

[JukkaL]

This reads to me as if he was going by the assumption that that's what Text currently means, which makes the current behavior of both mypy and pytype a bug.

No, I was aware of mypy treats unicode/Text. This was just a proposal that never went anywhere.

[matthiaskramm]

We just encountered an issue where code like

import io
x = io.StringIO()
x.write("")

silently passed type-checking, even though at runtime, this happens:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unicode argument expected, got 'str'

StringIO.write() is defined as (https://github.com/python/typeshed/blob/master/stdlib/2/_io.pyi#L135):

  def write(self, pbuf: unicode) -> int: ...

and unicode is automatically changed to Union[unicode, bytes].

Should we delete "unicode" from the PEP sentence "For parameters typed as unicode or Text, arguments of type str should be acceptable", to allow us to declare parameters as "pure" unicode?

[vlasovskikh]

@matthiaskramm I believe it's a minor class of false negative errors we could ignore. In #208 we have discussed several ideas of making str / bytes / unicode checking stricter in Python 2. None of them turned out to be satisfactory.

[matthiaskramm]

Right, I remember #208. But that one discusses a much broader topic.
I'm not interested in gradual byting, or introducing new string types. But I do want to be able to say "only unicode", in pyi files.
I'm not convinced that this class of (EDIT) false negatives is as small as we think it is. It already came up for us, and we've annotated less than 1% of our source base.

[gvanrossum]

Yes, it's come up for us too occasionally. I'm not sure there's an easy solution -- I suppose the best approach may be to separate Text from unicode and make it so that Text is compatible with both str and unicode while unicode is more constrained, but that would still be a complex change to mypy, and we'd probably need to revise lots of typeshed stubs to declare their arguments and return types properly as one or the other (since currently everyone assumes Text and unicode are interchangeable, and quite a few places assume str is compatible with both). (One problem IIUC is that mypy "forgets" aliases so you can't have a type that's an alias for another type but also endow it with some special meaning.)

[vlasovskikh]

@matthiaskramm Ok, got it. You mean "false negatives", don't you? A checker doesn't find the error even though it's present at run-time.

[matthiaskramm]

Right, I meant "false negative".

I don't like kicking cans down the road, but how about we change the PEP sentence to not mention unicode, and instead document the conversion unicode -> Union[str, unicode] in typeshed and in the mypy docs? That way, we could at least leave it up to the other type checkers whether they want to be more rigid about unicode or not.

[gvanrossum]

OK, that sounds like a reasonable compromise. Can you propose a PR for PEP 484?

[ilevkivskyi]

@gvanrossum

One problem IIUC is that mypy "forgets" aliases so you can't have a type that's an alias for another type but also endow it with some special meaning.

Maybe it was like this but now mypy is quite flexible with aliases, so this can be implemented if needed.

[srittau]

I think this has been resolved with additions to PEP 484.