interop_test_runner: use Python 3

[Lekensteyn]

'python2' does not exist on macOS, it is called 'python2.7'. Porting to
Python 3 is however more future-proof so do that instead. Rename the
file such that it can be run with pytest.

Do not listen on a fixed host port and use a random port instead.
Otherwise tests could fail if something is listening on those ports.

---

Note, with this modification, I noticed some ResourceWarnings after calling exec_run due to a Docker socket being leaked. These won't affect the test results though so I just left it there. See docker/docker-py#1293 (comment) I removed the wait logic improvement due to the complexity and the fact that it triggers a bug in docker-py.

I upgraded to Xenial to ensure a newer python3-six version to solve a failure importing docker. docker/docker-py#2294 (comment)

With the file rename, I can now do run a subset of the tests during development:

pytest _dev/interop_test_runner.py -k SIDH -v

and add -nauto to run those tests in parallel (requires pytest-xdist). This can complete all tests in 30 seconds (with 12 jobs). Another nice feature is the ability to pause when an assertion fails and inspect the context:

pytest _dev/interop_test_runner.py -k 'test_SIDH and Server'  --pdb

[kriskwiatkowski]

what for?

[Lekensteyn]

Otherwise pytest won't be able to load the script (see initial description for why pytest helps during development).

[kriskwiatkowski]

remove != 0

[kriskwiatkowski]

can we do it better without calling shell at all?

[Lekensteyn]

I couldn't really find a good alternative. I tried to retrieve container IP and try to port scan it from Python. Does not work on Docker for Mac, details.

An alternative is to start a container for the sole purpose of port-scanning a host (it could consist of a single Go binary), but the size of that support code will be slightly larger than the current approach. If you think that this would be better, I can change it though?

Another possibility is to add a HEALTCHECK command, but that would roughly have the same effect as checking ss, except that the logic is moved into the Docker image.

[kriskwiatkowski]

I would double check docker-py. I wonder if it's not done by this wrapper itself.
If not can't we just "import socket" and "socket.connect"? It's hard for me to believe python doesn't have something equivalent to ss

[Lekensteyn]

I have tried the socket approach before, but it won't work:

Does not work on Docker for Mac, details.

Checking for reachable ports is not something that can be done with the high-level docker-py API (docker/docker-py#2229). Usually healthchecks are used to test whether a service is ready (e.g. aside from checking that a TCP socket can be opened, verify that the HTTP service returns a proper response). In our case, it is sufficient if a TCP socket can be opened.

[kriskwiatkowski]

ok, keep digging.

[kriskwiatkowski]

This change is a root cause of python issuing such warn's

/usr/lib/python3.5/http/client.py:410: ResourceWarning: unclosed <socket.socket fd=8, family=AddressFamily.AF_UNIX, type=2049, proto=0, raddr=/var/run/docker.sock>
  self._close_conn()

I'm actually getting a lot of those locally. It will need to be fixed before merging

[Lekensteyn]

I've dropped the whole server start optimization (and kept two versions at https://github.com/cloudflare/tls-tris/commits/pwu/tests-speedup-v0 and https://github.com/cloudflare/tls-tris/commits/pwu/tests-speedup-v1).

Now it only fixes the Python 3 issues, pytest support and some potential resource leaks.