Integer overflow in TCP comm

[fjetter]

What happened:

I am receiving integer overflow errors in some large scale computations. Haven't had the chance to construct a MCVE but will try to.

cc @jakirkham I believe you reworked this section recently. Maybe you spot what's going wrong right away

What you expected to happen:

Minimal Complete Verifiable Example:

N/A

Anything else we need to know?:

Traceback (most recent call last):
  File "/mnt/mesos/sandbox/venv/lib/python3.6/site-packages/distributed/core.py", line 554, in handle_stream
    msgs = await comm.read()
  File "/mnt/mesos/sandbox/venv/lib/python3.6/site-packages/distributed/comm/tcp.py", line 199, in read
    n = await stream.read_into(frames)
  File "/mnt/mesos/sandbox/venv/lib/python3.6/site-packages/tornado/iostream.py", line 475, in read_into
    self._try_inline_read()
  File "/mnt/mesos/sandbox/venv/lib/python3.6/site-packages/tornado/iostream.py", line 842, in _try_inline_read
    pos = self._read_to_buffer_loop()
  File "/mnt/mesos/sandbox/venv/lib/python3.6/site-packages/tornado/iostream.py", line 755, in _read_to_buffer_loop
    if self._read_to_buffer() == 0:
  File "/mnt/mesos/sandbox/venv/lib/python3.6/site-packages/tornado/iostream.py", line 867, in _read_to_buffer
    bytes_read = self.read_from_fd(buf)
  File "/mnt/mesos/sandbox/venv/lib/python3.6/site-packages/tornado/iostream.py", line 1592, in read_from_fd
    return self.socket.recv_into(buf, len(buf))
  File "/usr/lib/python3.6/ssl.py", line 1009, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.6/ssl.py", line 871, in read
    return self._sslobj.read(len, buffer)
  File "/usr/lib/python3.6/ssl.py", line 631, in read
    v = self._sslobj.read(len, buffer)
OverflowError: signed integer is greater than maximum (distributed.core)

Environment:

• Dask version: dask/dask@91cee5b
• tornado version: 6.1
• Python version: py3.6
• Operating System: Deb9
• Install method (conda, pip, source): pip

[fjetter]

might've been introduced by #4506

[jakirkham]

Have we confirmed it is due to that change? Does the previous commit work?

[fjetter]

I reverted the commit and didn't run into the issue anymore but I'm also not sure if my computation triggers this reliably.

[jakirkham]

Ok thanks for that data point

It might be worth sticking a print(frames_nbytes) in read to see how large the message is when things fail

[jakirkham]

Does this show up in other Python versions?

[jakirkham]

Asking about other Python versions as 3.6 is the only one with SSLObject. It got removed with PR ( python/cpython#5252 ) in 3.8, which was also backported to 3.7. So would be good to know if this is just an oddity of 3.6

[jakirkham]

Does this show up in other Python versions?

Just checking in here. @fjetter do you have insight into this question?

[fjetter]

As of now, I don't have any more insights. I will try to set up another test run today to see if I can reproduce it another time. I will also try another python version. Since this requires a bit of time on my end, is there anything else worth checking out in these test runs?

• print statement for the frames_nbytes
• python version
  anything else worthy logging/inspecting?

[fjetter]

So far, I was not able to reproduce :/

[jakirkham]

Interesting. What things have been tried thus far?

Yeah those are the main things I can think of. Are there other things that you think would be relevant to test?

[jrbourbeau]

Just checking in here, @fjetter any luck on reproducing? I don't mean to apply unneeded pressure, but I'm wondering if this is a blocker for our upcoming release (xref dask/community#129)

[fjetter]

I'm not actively working on this due to limited time and my last efforts reproducing it was not successful. Considering that the exception is caused by a SSLObject which only exists in py3.6, I'm wondering if it is a reasonable assumption that this will only be an issue on py36?
My gut feeling tells me that a very difficult to reproduce, yet to be confirmed issue which likely only triggers on a python version we're about to drop should not block a release.