Epic: Get rid of db-sync

[geropl]

Summary

This is the epic for getting rid of db-sync, which was already discussed at length in the context of "Simplified Multi-Meta" in this RFC.

Context

• basically summed up in the RFC
• previous epic: Epic: Simplified Multi-Meta #7435

Value

• enables more Gitpod clusters (e.g., asia) without adding yet another DB
• enables delivering multi-cluster Gitpod to Self-Hosted customers (because it reduces setup complexity)

Acceptance Criteria

• we got rid of db-sync

Measurement

• when there is no db-sync deployment anymore, and we're still able to deploy to prod 🙃

Notion

Discussion related to this epic is in Notion.

Tasks

• [db-sync] Fix potential lost update #14560
• Create a sidecar container to configure Toxiproxy with a latency toxic.
• Run the above sidecar in the Toxiproxy pod.
• Deploy a second slow-server deployment which connects to mysql through Toxiproxy.
• Allow proxy to route websocket connections with the slow-database protocol header to the slow-server deployment.
• Allow setting of the slow-database websocket connection header from websocket connections from dashboard based on the value of a feature flag.
• [db-sync] Configure the correct values for the latency toxic to simulate a connection to a US DB. #14958
• [db-sync] Pre-flight requests to server /api/* endpoints #14960
• [db-sync] Investigate cause of slow workspace starts when working against a higher latency db #15027
• [db-sync] Roll out higher latency dashboard websocket and api/* connections to the webapp team #14962
• [db-sync] Compare the EU and US databases to ensure no significant data differences exist #14963
• [db-sync] Add latency to non-dashboard websocket connections to server JSON RPC-API #14964
• [db-sync] Add latency to additional server endpoints #14965
• [db-sync] Improve API interaction patterns on dashboard #14995
• [db-sync] Install Redis into application cluster (or use GCP managed Redis) #14996
• [db-sync] Set up TypeORM to use Redis #14997
• [db-sync] Gather internal feedback on the usability of a higher latency database connection #14998
• [db-sync] Configure observability for slow-server. #15297
• [db-sync] Define metrics of interest for slow-server. #15298
• [db-sync] Consider alternatives to the auth/workspace-cookie redirect #15328
• [db-sync] Look at slowest queries on Grafana and configure caching for them. (typeorm allows caching on a per-query basis).
• Resolve the TypeORM readiness check race condition when using slow-server #15655

[geropl]

Old workspace cluster table PR: #8539