🌱 Add e2e clusterctl upgrade tests #1371
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -173,3 +173,6 @@ sshuttle.pid | |
|
|
||
| # Book | ||
| docs/book/book/ | ||
|
|
||
| # venv | ||
| .venv | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -35,6 +35,7 @@ providers: | |
| - name: v1.2.4 | ||
| value: "https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.2.4/core-components.yaml" | ||
| type: url | ||
| contract: v1beta1 | ||
| files: | ||
| - sourcePath: "../data/shared/v1beta1/metadata.yaml" | ||
| replacements: | ||
|
|
@@ -48,6 +49,7 @@ providers: | |
| - name: v1.2.4 | ||
| value: "https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.2.4/bootstrap-components.yaml" | ||
| type: url | ||
| contract: v1beta1 | ||
| files: | ||
| - sourcePath: "../data/shared/v1beta1/metadata.yaml" | ||
| replacements: | ||
|
|
@@ -61,6 +63,7 @@ providers: | |
| - name: v1.2.4 | ||
| value: "https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.2.4/control-plane-components.yaml" | ||
| type: url | ||
| contract: v1beta1 | ||
| files: | ||
| - sourcePath: "../data/shared/v1beta1/metadata.yaml" | ||
| replacements: | ||
|
|
@@ -71,9 +74,26 @@ providers: | |
| - name: openstack | ||
| type: InfrastructureProvider | ||
| versions: | ||
| # This is only for clusterctl upgrade tests | ||
| - name: v0.6.3 | ||
| value: "https://github.com/kubernetes-sigs/cluster-api-provider-openstack/releases/download/v0.6.3/infrastructure-components.yaml" | ||
| type: url | ||
| contract: v1beta1 | ||
| files: | ||
| - sourcePath: "../data/shared/v1beta1_provider/metadata.yaml" | ||
| - sourcePath: "./infrastructure-openstack/cluster-template.yaml" | ||
| replacements: | ||
| - old: "imagePullPolicy: Always" | ||
| new: "imagePullPolicy: IfNotPresent" | ||
| - old: "--v=2" | ||
| new: "--v=4" | ||
| - old: "--leader-elect" | ||
| new: "--leader-elect=false\n - --sync-period=1m" | ||
|
Comment on lines
+77
to
+91
Contributor
There was a problem hiding this comment. In all honesty this remains magic to me. I'll work it out one day when it breaks. @tobiasgiese @jichenjc @seanschneeweiss Are any of you able to give this stanza meaningful review?
Contributor
Author
There was a problem hiding this comment. Anything in particular that you are wondering about? My understanding of this is that it is used to create a repository that clusterctl is then configured to use as the source of truth instead of reaching out to github to check what releases we have. This is mostly useful for not-yet-released versions, but since the same config is also used for upgrade tests, we need to explicitly list the versions that should be "available" to the test. I didn't think much about the additional config here (replacements and files) but I think they make sense and follow what we already have as well as what CAPI is doing. |
||
| - name: v0.6.99 | ||
| value: ../../../config/default | ||
| # This is the upcoming version. | ||
| # Specify no contract so that upgrade tests that start from a specific contract won't pick it up. | ||
| # contract: v1beta1 | ||
| files: | ||
| - sourcePath: "../data/shared/v1beta1_provider/metadata.yaml" | ||
| - sourcePath: "./infrastructure-openstack/cluster-template.yaml" | ||
|
|
@@ -116,6 +136,7 @@ variables: | |
| OPENSTACK_VOLUME_TYPE_ALT: "test-volume-type" | ||
| CONFORMANCE_WORKER_MACHINE_COUNT: "5" | ||
| CONFORMANCE_CONTROL_PLANE_MACHINE_COUNT: "1" | ||
| INIT_WITH_KUBERNETES_VERSION: "v1.25.0" | ||
|
Contributor
There was a problem hiding this comment. Is this used only for the management cluster? We still have the issue that our CI installation image is stuck on 1.18.
Contributor
Author
There was a problem hiding this comment. This is for the secondary management cluster. It will be used when rendering the cluster-template here. |
||
|
|
||
| intervals: | ||
| conformance/wait-control-plane: ["30m", "10s"] | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,3 +4,4 @@ resources: | |
|
|
||
| components: | ||
| - ../common-patches | ||
| - ../upgrade-patches | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| --- | ||
| # Workaround for https://github.com/kubernetes-sigs/cluster-api/issues/7457 | ||
| # There is a small but important difference between these two: | ||
| # path: /a/b/c | ||
| # *creates* the c array, overwriting anything that was there before | ||
| # path: /a/b/c/- | ||
| # *adds* to the c array and does not work if the array is missing | ||
| # | ||
| # We add to the postKubeadmCommands (instead of pre*) since we need the CI artifacts | ||
| # script to run first. Without this, the container images are not imported properly. | ||
| - op: add | ||
| path: /spec/kubeadmConfigSpec/postKubeadmCommands | ||
| value: | ||
| - /usr/local/bin/ci-artifacts-openstack.sh | ||
|
Contributor
There was a problem hiding this comment. Where does this run? Is it on the test runner or on the target? I assume the latter.
Contributor
Author
There was a problem hiding this comment. It runs on the target yes. I'm mimicking what we already use to to get the "CI artifacts". At first I actually tried adding the new script to the ci-artifacts-platform-kustomization.yaml, but that turned out to be harder than expected since it is way too easy to overwrite each others file in those patches. 😬 So I ended up putting it here together with the common patches. |
||
| - op: add | ||
| path: /spec/kubeadmConfigSpec/files/- | ||
| value: | ||
| content: | | ||
| #!/bin/bash | ||
| DOWNLOAD_E2E_IMAGE=${DOWNLOAD_E2E_IMAGE:=false} | ||
| if [ ! "${DOWNLOAD_E2E_IMAGE}" = true ]; then | ||
| echo "Not downloading E2E image, exiting" | ||
| exit 0 | ||
| fi | ||
| # Download the locally built CAPO controller image | ||
| echo "Downloading ${E2E_IMAGE_URL}" | ||
| wget "${E2E_IMAGE_URL}" -O "/tmp/capo-controller-manager.tar" | ||
| sudo ctr -n k8s.io images import "/tmp/capo-controller-manager.tar" || echo "* ignoring expected 'ctr images import' result" | ||
| owner: root:root | ||
| path: /usr/local/bin/ci-artifacts-openstack.sh | ||
| permissions: "0750" | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| --- | ||
| # Workaround for https://github.com/kubernetes-sigs/cluster-api/issues/7457 | ||
| # There is a small but important difference between these two: | ||
| # path: /a/b/c | ||
| # *creates* the c array, overwriting anything that was there before | ||
| # path: /a/b/c/- | ||
| # *adds* to the c array and does not work if the array is missing | ||
| # | ||
| # We add to the postKubeadmCommands (instead of pre*) since we need the CI artifacts | ||
| # script to run first. Without this, the container images are not imported properly. | ||
| - op: add | ||
| path: /spec/template/spec/postKubeadmCommands | ||
| value: | ||
| - /usr/local/bin/ci-artifacts-openstack.sh | ||
| - op: add | ||
| path: /spec/template/spec/files/- | ||
| value: | ||
| content: | | ||
| #!/bin/bash | ||
| DOWNLOAD_E2E_IMAGE=${DOWNLOAD_E2E_IMAGE:=false} | ||
| if [ ! "${DOWNLOAD_E2E_IMAGE}" = true ]; then | ||
| echo "Not downloading E2E image, exiting" | ||
| exit 0 | ||
| fi | ||
| # Download the locally built CAPO controller image | ||
| echo "Downloading ${E2E_IMAGE_URL}" | ||
| wget "${E2E_IMAGE_URL}" -O "/tmp/capo-controller-manager.tar" | ||
| sudo ctr -n k8s.io images import "/tmp/capo-controller-manager.tar" || echo "* ignoring expected 'ctr images import' result" | ||
| owner: root:root | ||
| path: /usr/local/bin/ci-artifacts-openstack.sh | ||
| permissions: "0750" |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| # Modifications to release templates for clusterctl upgrade scenarios | ||
| --- | ||
| apiVersion: kustomize.config.k8s.io/v1alpha1 | ||
| kind: Component | ||
|
|
||
| patches: | ||
| - target: | ||
| kind: KubeadmControlPlane | ||
| name: \${CLUSTER_NAME}-control-plane | ||
| path: ci-hack-kcp.yaml | ||
| - target: | ||
| kind: KubeadmConfigTemplate | ||
| name: \${CLUSTER_NAME}-md-0 | ||
| path: ci-hack-kct.yaml |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| --- | ||
| resources: | ||
| - ../../../../../kustomize/v1alpha5/default | ||
|
|
||
| components: | ||
| - ../common-patches |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,55 @@ | ||
| //go:build e2e | ||
| // +build e2e | ||
|
|
||
| /* | ||
| Copyright 2021 The Kubernetes Authors. | ||
| Licensed under the Apache License, Version 2.0 (the "License"); | ||
| you may not use this file except in compliance with the License. | ||
| You may obtain a copy of the License at | ||
| http://www.apache.org/licenses/LICENSE-2.0 | ||
| Unless required by applicable law or agreed to in writing, software | ||
| distributed under the License is distributed on an "AS IS" BASIS, | ||
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| See the License for the specific language governing permissions and | ||
| limitations under the License. | ||
| */ | ||
|
|
||
| package e2e | ||
|
|
||
| import ( | ||
| "context" | ||
|
|
||
| . "github.com/onsi/ginkgo" | ||
| capi_e2e "sigs.k8s.io/cluster-api/test/e2e" | ||
|
|
||
| "sigs.k8s.io/cluster-api-provider-openstack/test/e2e/shared" | ||
| ) | ||
|
|
||
| // Note: There isn't really anything fixing this to the v0.6 release. The test will | ||
| // simply pick the latest release with the correct contract from the E2EConfig, as seen here | ||
| // https://github.com/kubernetes-sigs/cluster-api/blob/3abb9089485f51d46054096c17ccb8b59f0f7331/test/e2e/clusterctl_upgrade.go#L265 | ||
| // When new minor releases are added (with the same contract) we will need to work on this | ||
| // if we want to continue testing v0.6. | ||
| var _ = Describe("When testing clusterctl upgrades (v0.6=>current) [clusterctl-upgrade]", func() { | ||
| ctx := context.TODO() | ||
| shared.SetEnvVar("USE_CI_ARTIFACTS", "true", false) | ||
| shared.SetEnvVar("DOWNLOAD_E2E_IMAGE", "true", false) | ||
|
Contributor
There was a problem hiding this comment. Isn't this setting an environment variable on the test runner? If so, how does that affect
Contributor
Author
There was a problem hiding this comment. Indeed! It is used when rendering the cluster-template. Both these variables are used in the cluster-template that we use in the tests. You can see the rendered result in
Contributor
There was a problem hiding this comment. I'm going to have to look at the template renderer. I didn't think it would support bash syntax like If that question itself reveals my fundamental misunderstanding, please can you enlighten me?
Contributor
There was a problem hiding this comment. The substitution language is linked from the clusterctl documentation here: https://cluster-api.sigs.k8s.io/clusterctl/commands/generate-yaml.html |
||
| shared.SetEnvVar("E2E_IMAGE_URL", "http://10.0.3.15/capo-e2e-image.tar", false) | ||
|
|
||
| capi_e2e.ClusterctlUpgradeSpec(ctx, func() capi_e2e.ClusterctlUpgradeSpecInput { | ||
| return capi_e2e.ClusterctlUpgradeSpecInput{ | ||
| E2EConfig: e2eCtx.E2EConfig, | ||
| ClusterctlConfigPath: e2eCtx.Environment.ClusterctlConfigPath, | ||
| BootstrapClusterProxy: e2eCtx.Environment.BootstrapClusterProxy, | ||
| ArtifactFolder: e2eCtx.Settings.ArtifactFolder, | ||
| SkipCleanup: false, | ||
| InitWithBinary: "https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.2.4/clusterctl-{OS}-{ARCH}", | ||
| InitWithProvidersContract: "v1beta1", | ||
| MgmtFlavor: shared.FlavorDefault, | ||
| WorkloadFlavor: shared.FlavorV1alpha5, | ||
| } | ||
| }) | ||
| }) | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm going to use this to write v1alpha5 simulator tests!