Python 3.8

[carlwgeorge]

The test suite fails when run with Python 3.8.

=================================== FAILURES ===================================
___________________ TestMessageDecoder.test_not_even_unicode ___________________

self = <test.test_frame_protocol.TestMessageDecoder object at 0x7f758e817bb0>

    def test_not_even_unicode(self):
        payload = u"fñörd"
        payload = payload.encode("iso-8859-1")
    
        decoder = fp.MessageDecoder()
        frame = fp.Frame(
            opcode=fp.Opcode.TEXT,
            payload=payload,
            frame_finished=True,
            message_finished=False,
        )
    
        with pytest.raises(fp.ParseFailed) as excinfo:
>           decoder.process_frame(frame)
E           Failed: DID NOT RAISE <class 'wsproto.frame_protocol.ParseFailed'>

decoder    = <wsproto.frame_protocol.MessageDecoder object at 0x7f758e8179d0>
excinfo    = <ExceptionInfo for raises contextmanager>
frame      = Frame(opcode=<Opcode.TEXT: 1>, payload=b'f\xf1\xf6rd', frame_finished=True, message_finished=False)
payload    = b'f\xf1\xf6rd'
self       = <test.test_frame_protocol.TestMessageDecoder object at 0x7f758e817bb0>

test/test_frame_protocol.py:263: Failed
=============================== warnings summary ===============================
test/test_frame_protocol.py:932
  /builddir/build/BUILD/wsproto-0.14.1/test/test_frame_protocol.py:932: SyntaxWarning: invalid escape sequence \_
    payload = u"¯\_(ツ)_/¯"

test/test_frame_protocol.py:1097
  /builddir/build/BUILD/wsproto-0.14.1/test/test_frame_protocol.py:1097: SyntaxWarning: invalid escape sequence \_
    reason = u"¯\_(ツ)_/¯"

test/test_frame_protocol.py:1106
  /builddir/build/BUILD/wsproto-0.14.1/test/test_frame_protocol.py:1106: SyntaxWarning: invalid escape sequence \_
    reason = u"¯\_(ツ)_/¯" * 10

test/test_frame_protocol.py:1129
  /builddir/build/BUILD/wsproto-0.14.1/test/test_frame_protocol.py:1129: SyntaxWarning: invalid escape sequence \_
    payload = u"¯\_(ツ)_/¯".encode("utf8")

test/test_frame_protocol.py:1140
  /builddir/build/BUILD/wsproto-0.14.1/test/test_frame_protocol.py:1140: SyntaxWarning: invalid escape sequence \_
    payload = u"¯\_(ツ)_/¯".encode("utf8")

-- Docs: https://docs.pytest.org/en/latest/warnings.html
=============== 1 failed, 238 passed, 5 warnings in 0.78 seconds ===============

Fedora is preparing for Python 3.8 in Fedora 31, which is what alerted me to this failure with my python-wsproto package.

[njsmith]

That's disturbing... The test is checking that if someone tries to use invalid utf-8, then wsproto will detect that and raise an error. In practice, we just delegate this check to Python's built-in utf-8 decoder. If that's suddenly failing but only under 3.8-beta, then the simplest explanation would be "python 3.8's utf-8 decoder is broken". Though someone would need to confirm that to be sure.

[tirkarthi]

I tried bisecting on CPython master with below code that is used in the test and came across https://bugs.python.org/issue24214. Before this issue was merged UnicodeDecodeError was raised but after this fix it decodes without exception. Not the commit is also merged to 3.7 so I guess it will be released with 3.7.4 too where this test might fail.

# /tmp/foo.py
from codecs import getincrementaldecoder
decoder = getincrementaldecoder("utf-8")()
print(decoder.decode(b'f\xf1\xf6rd', False))

commit python/cpython@7a465cb5ee

➜  cpython git:(7a465cb5ee) ./python.exe /tmp/foo.py
f

Checking out commit before 7a465cb5ee shows UnicodeDecodeError :

➜  cpython git:(38f4e468d4) ./python.exe /tmp/foo.py
Traceback (most recent call last):
  File "/tmp/foo.py", line 3, in <module>
    print(decoder.decode(b'f\xf1\xf6rd', False))
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf1 in position 1: invalid continuation byte

[tirkarthi]

The deprecation warnings should be fixed with 414c287 where unicode literals are replaced with raw string literals that fixes SyntaxWarning.

[tirkarthi]

I guess this should be fixed with this CPython PR : python/cpython#14304

[pgjones]

Just tested with Python 3.8.0b2+ and the tests pass, so I think this is fixed.