Add e2e tests

[bayandin]

This PR adds a test for VM migration and checks that the connection doesn't break. It uses kuttl as a test tool.

• Create VM with Postgres
• Start workload
  • Wait for Postgres to be able to accept connection using pg_isready
  • Create a small db using pgbench --initialize
  • Start 10m pgbench workload (the duration of the workload should be longer that test duration)
• Check that VM is running
• Migrate VM
• Check that Migration is finished
• Check that pgbench is still running

How to run it locally:

• Install kuttl: brew install kuttl or go install github.com/kudobuilder/kuttl/cmd/[email protected] or download the binary from their release page https://github.com/kudobuilder/kuttl/releases/tag/v0.15.0
• make local-cluster
• make deploy
• make e2e or, if you want to keep pods running after the test, you can use kubectl kuttl test --config tests/e2e/kuttl-test.yaml --skip-delete (note, that before running the test next time, you'll need to delete namespace to avoid Postgres ports collision: kubectl get namespaces --no-headers | awk '/kuttl-test-/ {print $1}' | xargs --no-run-if-empty -n 1 kubectl delete namespace)

pgbench output for the migration timeframe looks like this:

pgbench (15.2)
starting vacuum...end.
progress: 1676899716.775 s, 57.0 tps, lat 32.092 ms stddev 12.869, 0 failed
...
progress: 1676899732.775 s, 29.0 tps, lat 73.181 ms stddev 59.306, 0 failed
progress: 1676899733.775 s, 22.0 tps, lat 96.649 ms stddev 60.388, 0 failed
progress: 1676899734.775 s, 21.0 tps, lat 66.528 ms stddev 43.079, 0 failed
progress: 1676899735.776 s, 0.0 tps, lat 0.000 ms stddev 0.000, 0 failed
progress: 1676899736.780 s, 0.0 tps, lat 0.000 ms stddev 0.000, 0 failed
progress: 1676899737.776 s, 0.0 tps, lat 0.000 ms stddev 0.000, 0 failed
...
progress: 1676899760.776 s, 0.0 tps, lat 0.000 ms stddev 0.000, 0 failed
progress: 1676899761.776 s, 0.0 tps, lat 0.000 ms stddev 0.000, 0 failed
progress: 1676899762.776 s, 0.0 tps, lat 0.000 ms stddev 0.000, 0 failed
progress: 1676899763.775 s, 54.1 tps, lat 1085.284 ms stddev 5378.556, 0 failed
progress: 1676899764.775 s, 73.0 tps, lat 27.662 ms stddev 3.444, 0 failed
progress: 1676899765.775 s, 66.0 tps, lat 29.834 ms stddev 8.584, 0 failed
progress: 1676899766.775 s, 74.0 tps, lat 27.189 ms stddev 4.092, 0 failed
...

[bayandin]

@cicdteam @sharnoff I'd appreciate your review on this one.
It's still WIP until we have make installed on our runners (and then I can reuse make instead of shell scripts in a workflow, see attached issue). If you know some other cases we want to test, please let me know, and I can try to add them.

UPD switched to make

[sharnoff]

Broadly looks good. I'm not familiar with kuttl, but if it works it works. One curiosity: why is CGO_ENABLED=0 required? (and why even on go fmt?) — It'd be good to have a brief comment explaining as well.

[bayandin]

One curiosity: why is CGO_ENABLED=0 required? (and why even on go fmt?) — It'd be good to have a brief comment explaining as well.

Actually, it's not required for go fmt, but it's required for go vet, and GCC is not installed on our runners by default: https://github.com/neondatabase/neonvm/actions/runs/4294900432/jobs/7484573737#step:8:116
I've added a comment, thanks!