e2e tests wont run locally

[jberkhahn]

Bug Report

the e2e tests will not run locally unless I add these two lines before the call to setup_envs in the script that the Makefile task I'm running is shelling out to. This isn't documented anywhere so it's very confusing if you don't know to do this. According to @joelanford these lines are commented out because they cause the e2e tests in travis to fail(!?). It would be nice if we could have e2e tests that run correctly all the time, or at least document what you have to do to get them to run locallly.

Environment

I did this with both minikube and kind, and got the same error both times, it seems to blow up when attempting to build a controller-manager image with the following error:
msg: “failed to start the controlplane. retried 5 times: fork/exec /tmp/operator-sdk/bin/etcd: no such file or directory”,

[joelanford]

I think I see what the problem might be.

In CI, we always run make setup-k8s which invokes ./hack/ci/setup-k8s.sh before running tests. This file cleans up the staging directory, downloads all the necessary binaries (envtest, kind, kubectl) for the test, then sets up kind/kubectl.

In ./hack/test/e2e-go.sh (and other test scripts), none of this setup occurs.

Therefore running make test-e2e-go locally will always fail because the necessary setup hasn't happened.

The immediate fix is to always run e2e's and integration tests like this: make setup-k8s <test-target>.

However, I don't like this for two reasons:

1. If a test depends on something happening, it should declare that in the Makefile so that callers don't have to explicitly call the right targets in the right order to run tests.
2. The make setup-k8s target changes my personal ~/.kube/config file, leaves the kind cluster running after the test finishes, and leaves files in /tmp/operator-sdk. And that's just what I could find in the last few minutes. All of these are annoying. We should either:
   • NOT require setup-k8s for local tests, OR
   • Require it, but ensure that everything is cleaned up and user configurations are left untouched.

[jmrodri]

I'd like the tests to possibly see if there's a cluster available for it to run. If not, possibly let the user know what they need. On CI, I think it is okay to always run setup-k8s but I don't want something to muck with my environment locally I already have a kind cluster running.

[camilamacedo86]

Hi @jberkhahn,

We only need to have a Kind cluster running and the bins then, we can just run make test-e2e-helm for example.

Regards the error:

msg: “failed to start the controlplane. retried 5 times: fork/exec /tmp/operator-sdk/bin/etcd: no such file or directory”,

It is because the ENV TEST requires binaries. IMO we can solve it by;

• Adding a new target setup-local-e2e-env as suggested by @joelanford
• Then, properly documenting in https://sdk.operatorframework.io/docs/contribution-guidelines/testing/running-the-tests/

[jberkhahn]

Summary of comments from Sep 14 bug triage meeting:

• the tests (e2e or otherwise) need to be platform independent. this is a requirement
• the tests need to be run the same locally as in CI. this will probably help Generate k8s controller/workqueue logic #1
• what make task runs/does should be made clear. currently there's a bunch of different test-related make tasks
• the docs should clearly spell out how to run the tests

[estroz]

#4023 fixes this except ansible-molecule tests, which will be addressed in a separate issue.