Support non-ascii identifiers

[JukkaL]

We should support non-ascii (unicode) identifiers as variable names, etc.

[o11c]

I'm actually not sure how to support this since it is defined in terms of unicode classes, which are not supported by python's regex engine, and there is no efficient way to enumerate the characters in each category (let alone DTRT for the particular version of unicode supported by any given python3 release).

Really seems like more of a misfeature than a feature to me.

[JukkaL]

Hmm. I haven't actually looked into this in much detail. We should still be able to implement this in an efficient manner for the 99% of code that doesn't use unicode identifiers. It's fine if using unicode identifiers slows down parsing, but it's not okay if this feature slows down parsing everything else.

Maybe something like this:

Have a separate case in the lexer for all non-ascii characters outside string literals and comments (character code >= 128). Fall back to a possibly slow implementation that can do whatever needed to get this working when we hit this, but skip this code path altogether for code that doesn't use any unicode identifiers.

Also, the unicodedata module could be useful.

[gvanrossum]

I'm going to close this as a won't fix because the new parser will solve this and we're not going to put any work in the old parser/lexer.

[vedgar]

May I ask what's the status of that New Parser? Maybe I can even work on it, if it's Python-only. I was extremely surprised to learn that mypy still doesn't understand Python naming rules. PEP 3131 is almost a decade old. :-o

[gvanrossum]

The new parser will be the default on the next mypy release (and is already on the GitHub master).

…

--Guido (mobile)

On Feb 1, 2017 7:48 AM, "Vedran Čačić" ***@***.***> wrote: May I ask what's the status of that New Parser? Maybe I can even work on it, if it's Python-only. I was extremely surprised to learn that mypy still doesn't understand Python naming rules. PEP 3131 is almost a decade old. :-o — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#586 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACwrMljOZFb_ocMQ5bjaqq7r9vxMImQ2ks5rYKlGgaJpZM4Dj2E-> .