Deployment tips to deal with a lot of connections

[bsergean]

Hi there,

I am thinking about the best way to handle a bunch of connections using the least amount of machines (2 to 4 beefy servers probably), and I wonder if folks here have insights. With http I've been using gunicorn which makes it very easy to use many cores, but I'm a bit lost with asyncio on how to use all my cores. My application will likely just push data into redis and do little CPU work, but if there are a bunch of connected clients cpu might become an issue.

I thought about running multiple servers on multiple ports, and use nginx who's serving on port 80 as a load-balancer + proxy_pass (is that even possible) to go to multiple daemon running the websockets app (something like that?).

And then I'd have multiple services (each one, on a given machine, running under systemd) / each proxy_pass talk to one.

start_server = websockets.serve(hello, 'localhost', 8000) # for 'proxy_pass A'
start_server = websockets.serve(hello, 'localhost', 8001) # for 'proxy_pass B'
start_server = websockets.serve(hello, 'localhost', 8002) # for 'proxy_pass C, etc...'

I know it's a very vague question and not really an issue, but in absence of a better place to discuss this ... here we go !

Thanks,

• Benjamin

[bsergean]

Apparently gunicorn has options to run 'asyncio' workers. Maybe that would work ... see AyncIO Workers section in the doc. I'm trying to have the least complicated nginx setup.

[aaugustin]

gunicorn asyncio workers rely on aiohttp. If you're deploying that way, you may just as well use aiohttp instead of websockets.

I'm aware that websockets doesn't have a great deployment story: http://websockets.readthedocs.io/en/stable/deployment.html.

Running multiple processes behind nginx feels slightly old-school but it should work fine. See https://docs.aiohttp.org/en/stable/deployment.html.

I'm not aware of a asyncio application server able to bind to a socket, then fork workers, allowing multiple processes to handle connections received on the same socket. The create_server API provided by asyncio may make this difficult.

I suppose the new-fangled way to deploy would be to run one websockets process per container, start as many containers as you need, and let the ingress engine route traffic to containers. I would only recommend trying that if you're already comfortable with a container infrastructure.

I would really like to hear how you solve this and improve the documentation :-)

[bsergean]

I have to admit I got lazy and went with aiohttp... however I am still using websockets right now on the client side to try to generate a lot of connections. Not sure whether I can make that work with many process though, or if it's worth it.

I've also found a code example .. that I cannot find anymore where a specific socket option was used so that the same socket could be used by multiple forked process. I'll keep looking and I'll share it here.

[aaugustin]

So you deployed aiohttp with gunicorn?

[bsergean]

Yes they have a bunch of tutorials on how to do so, it's in their doc. And I haven't deployed anything at this point, it's still running locally but it 'works' (I can hit it with a test client).

Here is what it takes apparently to make a 'worker' that is compatible with gunicorn.
https://github.com/aio-libs/aiohttp/blob/master/aiohttp/worker.py

I don't know what interface the 'app' needs to follow/implement so that gunicorn can 'run your code'.

[aaugustin]

In fact, if you're running a relatively recent Linux (>= 3.9), there's a very simple solution: use serve(..., reuse_port=True) and run as many websocket processes as you need.

If you run five copies of this script:

slow_echo.py

#!/usr/bin/env python

# Script for testing parallel deployment (without burning CPU cycles).

import asyncio
import time
import websockets

DELAY = 0.1     # 100ms


async def slow_echo(websocket, path):
    async for message in websocket:
        time.sleep(DELAY)   # block the event loop to simulate CPU load
        await websocket.send(message)

asyncio.get_event_loop().run_until_complete(
    websockets.serve(slow_echo, 'localhost', 8765, reuse_port=True))
asyncio.get_event_loop().run_forever()

Then this script should complete in 2 seconds rather than 10 seconds:

benchmark.py

#!/usr/bin/env python

# WS client example

import asyncio
import websockets

CLIENTS = 10
MESSAGES = 10


async def run_client(client):
    async with websockets.connect('ws://localhost:8765') as websocket:
        for message in range(MESSAGES):
            await websocket.send('{client}:{message}')
            await websocket.recv()


clients = [run_client(client) for client in range(CLIENTS)]
asyncio.get_event_loop().run_until_complete(asyncio.wait(clients))

Apparently this doesn't work on macOS: https://stackoverflow.com/questions/41247790/so-reuseport-on-macos-with-libuv#41479612

[aaugustin]

I tested the reuse_port technique on Linux. It's simple and it works.

It isn't ideal from a latency perspective under load, for reasons explained here: https://blog.cloudflare.com/the-sad-state-of-linux-socket-balancing/

[aaugustin]

In fact the Cloudflare blog post describes three deployment techniques and there's a reasonable way to implement them with websockets:

a. Single listen socket, single worker process => run a Python process. (This is also what you would do inside a container.)
b. Single listen socket, multiple worker processes => deploy nginx or haproxy in front of N Python processes listening on different ports. (This is the model with the most moving parts, but it's also a well known model for running web apps.)
c. Multiple worker processes, each with separate listen socket => run N Python processes listening on the same port with the reuse_port option.

I think I'm going to document that.

[bsergean]

For simplicity, I was considering option 2 myself, with no nginx, but instead gunicorn forking multiple python process. We have a load-balancer that can redirected incoming connection (SSL) to any port, so I would make incoming connections go to port 8765 (one that can be 'created' without being superuser).

Actually gunicorn seems to also have an option to reuse ports.

reuse_port
~~~~~~~~~~

* ``--reuse-port``
* ``False``

Set the ``SO_REUSEPORT`` flag on the listening socket.

.. versionadded:: 19.8

On that Cloudflare blog post, they use an http benchmark and notice odd performance with option C, but I wonder if with persistent connections / websockets that would perform better.

[bsergean]

https://stackoverflow.com/questions/32661091/behavior-of-so-reuseaddr-and-so-reuseport-changed

Also on macOS there are 2 sockets options, maybe they have to be combined to provide that Linux behavior. But I'm not a socket expert at all so that stack-overflow comment is probably right...

[aaugustin]

The "odd performance" is increased variance in the latency for accepting connections. This is likely to be less a problem for websockets than for HTTP, if only because web pages rarely have multiple websocket requests on the critical path for rendering something in the browser.

[aaugustin]

And I'm pretty sure that SO_REUSEADDR doesn't matter here, we want SO_REUSEPORT and it doesn't do what we need on BSD.

[bsergean]

Hello, just to close the loop here I wanted to mention that I am now using docker + kubernete etc... and just simply spinning up more replicas/container instances, as you had mentioned earlier (the new fangled way !). It is a very simple and elegant way to scale I feel. Now I need to read up on the websockets doc a little more to tune how I can make sure my app doesn't run out of memory, when tons of new connections comes in, and instead reject them. I hope there's a way, and if there isn't we will just try to bump our replicas count until things settle for our load.

By browsing the web I've also came across something called ASGI, which is an interface similar to WSGI / some people have already made it possible to run your websockets library through something called uvicorn, whose name sounds a lot like gunicorn... and another thing called starlette, so a bunch of options are available. At some point I built a little class interface myself, mostly ripped of aio-http so that I could use gunicorn.

Thanks a lot for all your hard work on this websockets library, it's great stuff !

[bsergean]

BTW it looks like a good place to start refusing new connections when approaching machine memory limit would be:

class ServerProtocol(websockets.WebSocketServerProtocol):
    async def process_request(self, path, request_headers):
          ... add code here that compute memory usage each time a new request come in, and maybe send an HTTP error (overloaded) when we are at 95% memory capacity ?

https://serverfault.com/questions/233519/what-is-the-proper-http-status-for-too-many-connections

429 / too many requests seems to be a good candidate.