Merge pull request #5 from jetstack/add-smoke-tests

Add smoke tests

Author: jetstack-bot

Dockerfile

@@ -1,2 +1,28 @@
-# Doc: https://github.com/GoogleCloudPlatform/marketplace-k8s-app-tools/blob/master/docs/building-deployer-helm.md#build-your-deployer-container
+# This Dockerfile allows us to create a "deployer" image. The deployer
+# image, using this "ombuild" base image, will contain our Marketplace
+# application's Helm chart and will be in charge of doing the "helm install"
+# (which is the only thing the deployer image ever does).
+#
+# See:
+# https://github.com/GoogleCloudPlatform/marketplace-k8s-app-tools/blob/4335f9/docs/building-deployer-helm.md#build-your-deployer-container
+
 FROM gcr.io/cloud-marketplace-tools/k8s/deployer_helm/onbuild
+
+# The schema,yaml and chart/ have already been added thanks to the
+# "onbuild" Dockerfile:
+# https://github.com/GoogleCloudPlatform/marketplace-k8s-app-tools/blob/4335f9/marketplace/deployer_helm_base/onbuild/Dockerfile#L4-L12
+#
+# Since the deployer image must only contain compressed charts, i.e.,
+# chart/chart.tar.gz (which is created by the "onbuild" Docker image), and
+# a data-test/chart/chart.tar.gz. That is why we need to tar the chart in
+# data-test.
+COPY data-test/schema.yaml /data-test/schema.yaml
+COPY data-test/chart /tmp/data-test/chart.tmp
+RUN cd /tmp/data-test \
+        && mv chart.tmp/* chart \
+        && tar -czvf /tmp/data-test/chart.tar.gz chart \
+        && mv chart.tar.gz /data-test/chart/
+
+
+# If you wonder what magic is this, take a look at:
+# https://github.com/GoogleCloudPlatform/marketplace-k8s-app-tools/blob/4335f9/marketplace/deployer_helm_base/onbuild/Dockerfile#L16-L20

README.md

@@ -1,12 +1,57 @@
 
 # jsp-gcm
 
+This is the repository that holds the configuration for our Google
+Marketplace solution, [jetstack-secure-for-cert-manager][].
+
 **Content:**
 
+- [Technical considerations](#technical-considerations)
 - [Installing and manually testing the deployer](#installing-and-manually-testing-the-deployer)
 - [Testing and releasing the deployer using Google Cloud Build](#testing-and-releasing-the-deployer-using-google-cloud-build)
+  - [Debugging deployer and smoke-tests when run in Cloud Build](#debugging-deployer-and-smoke-tests-when-run-in-cloud-build)
 - [Updating the upstream cert-manager chart version](#updating-the-upstream-cert-manager-chart-version)
 
+## Technical considerations
+
+**Retagging cert-manager images:**
+
+In order to abide by the [schema.md][], which states:
+
+> When users deploy your app from Google Cloud Marketplace, the final image
+> names may be different, but they will follow **the same release tag** and
+> name prefix rule.
+
+This means we do re-tag all our images (cert-manager, cas-issuer, ubbagent,
+preflight-agent) using a unified tag that is distinct from the cert-manager
+regular version. We call this version the "application version". In the
+following example, the application version is `1.0.0` although the
+cert-manager-controller is `1.1.0`:
+
+```sh
+gcr.io/jetstack-public/jetstack-secure-for-cert-manager:1.0.0
+gcr.io/jetstack-public/jetstack-secure-for-cert-manager/cert-manager-acmesolver:1.0.0
+gcr.io/jetstack-public/jetstack-secure-for-cert-manager/cert-manager-cainjector:1.0.0
+gcr.io/jetstack-public/jetstack-secure-for-cert-manager/cert-manager-google-cas-issuer:1.0.0
+gcr.io/jetstack-public/jetstack-secure-for-cert-manager/cert-manager-webhook:1.0.0
+gcr.io/jetstack-public/jetstack-secure-for-cert-manager/deployer:1.0.0
+gcr.io/jetstack-public/jetstack-secure-for-cert-manager/preflight:1.0.0
+gcr.io/jetstack-public/jetstack-secure-for-cert-manager/ubbagent:1.0.0
+```
+
+**cert-manager-controller is the "primary image":**
+
+The "primary" image is pushed to the "root" of the registry, for example:
+
+```sh
+# The primary image "cert-manager-controller":
+gcr.io/jetstack-public/jetstack-secure-for-cert-manager:1.0.0
+
+# All other images:
+gcr.io/jetstack-public/jetstack-secure-for-cert-manager/deployer:1.0.0
+gcr.io/jetstack-public/jetstack-secure-for-cert-manager/cert-manager-webhook:1.0.0
+```
+
 ## Installing and manually testing the deployer
 
 In order to have the google-cas-issuer working, we need to enable [workload
@@ -18,45 +63,42 @@ gcloud container clusters create foo --region us-east1 --num-nodes=1 --preemptib
   --workload-pool=$(gcloud config get-value project | tr ':' '/').svc.id.goog
 ```
 
-This application re-tags the various images (cert-manager, cas-issuer, ubbagent, preflight-agent) using
-a unified tag that we call "application version". Although it does not appear to be a requirement for
-releasing to the Google Marketplace, we were not able to set "default" tags for each image and thus
-resolved to just having a unified tag; this means that we will have to keep this difference in tags when
-supporting [jetstack-secure-for-cert-manager][].
-
-[jetstack-secure-for-cert-manager]: https://console.cloud.google.com/partner/editor/jetstack-public/jetstack-secure-for-cert-manager?project=jetstack-public
-
-Re-publish the images to the project:
+Now, re-publish the images to the project:
 
 ```sh
 export REGISTRY=gcr.io/$(gcloud config get-value project | tr ':' '/')
-export APP_NAME=jetstack-secure
+export SOLUTION=jetstack-secure-for-cert-manager
+
 docker pull quay.io/jetstack/cert-manager-controller:v1.1.0
+docker pull quay.io/jetstack/cert-manager-acmesolver:v1.1.0
 docker pull quay.io/jetstack/cert-manager-cainjector:v1.1.0
 docker pull quay.io/jetstack/cert-manager-webhook:v1.1.0
 docker pull quay.io/jetstack/cert-manager-google-cas-issuer:0.1.0
 docker pull quay.io/jetstack/preflight:0.1.27
-docker tag quay.io/jetstack/cert-manager-controller:v1.1.0 $REGISTRY/$APP_NAME/cert-manager-controller:1.0.0
-docker tag quay.io/jetstack/cert-manager-cainjector:v1.1.0 $REGISTRY/$APP_NAME/cert-manager-cainjector:1.0.0
-docker tag quay.io/jetstack/cert-manager-webhook:v1.1.0 $REGISTRY/$APP_NAME/cert-manager-webhook:1.0.0
-docker tag quay.io/jetstack/cert-manager-google-cas-issuer:latest $REGISTRY/$APP_NAME/cert-manager-google-cas-issuer:1.0.0
-docker tag quay.io/jetstack/preflight:latest $REGISTRY/$APP_NAME/cert-manager-preflight:1.0.0
-docker push $REGISTRY/$APP_NAME/cert-manager-controller:1.0.0
-docker push $REGISTRY/$APP_NAME/cert-manager-cainjector:1.0.0
-docker push $REGISTRY/$APP_NAME/cert-manager-webhook:1.0.0
-docker push $REGISTRY/$APP_NAME/cert-manager-google-cas-issuer:1.0.0
-docker push $REGISTRY/$APP_NAME/cert-manager-preflight:1.0.0
+docker pull gcr.io/cloud-marketplace-tools/metering/ubbagent:latest
+
+docker tag quay.io/jetstack/cert-manager-controller:v1.1.0 $REGISTRY/$SOLUTION:1.0.0
+docker tag quay.io/jetstack/cert-manager-acmesolver:v1.1.0 $REGISTRY/$SOLUTION/cert-manager-acmesolver:1.0.0
+docker tag quay.io/jetstack/cert-manager-cainjector:v1.1.0 $REGISTRY/$SOLUTION/cert-manager-cainjector:1.0.0
+docker tag quay.io/jetstack/cert-manager-webhook:v1.1.0 $REGISTRY/$SOLUTION/cert-manager-webhook:1.0.0
+docker tag quay.io/jetstack/cert-manager-google-cas-issuer:latest $REGISTRY/$SOLUTION/cert-manager-google-cas-issuer:1.0.0
+docker tag quay.io/jetstack/preflight:latest $REGISTRY/$SOLUTION/preflight:1.0.0
+docker pull gcr.io/cloud-marketplace-tools/metering/ubbagent:latest $REGISTRY/$SOLUTION/ubbagent:1.0.0
+
+docker push $REGISTRY/$SOLUTION:1.0.0
+docker push $REGISTRY/$SOLUTION/cert-manager-acmesolver:1.0.0
+docker push $REGISTRY/$SOLUTION/cert-manager-cainjector:1.0.0
+docker push $REGISTRY/$SOLUTION/cert-manager-webhook:1.0.0
+docker push $REGISTRY/$SOLUTION/cert-manager-google-cas-issuer:1.0.0
+docker push $REGISTRY/$SOLUTION/preflight:1.0.0
+docker push $REGISTRY/$SOLUTION/ubbagent:1.0.0
 ```
 
-> Note: although cert-manager's tags are of the form "v1.1.0", we 
-> use the same JSP version tag for all the Google Marketplace images, 
-> for consistency with other marketplace packages.
-
 Then, build and push the deployer image:
 
 ```sh
-docker build --tag $REGISTRY/$APP_NAME/deployer .
-docker push $REGISTRY/$APP_NAME/deployer
+docker build --tag $REGISTRY/$SOLUTION/deployer:1.0.0 .
+docker push $REGISTRY/$SOLUTION/deployer:1.0.0
 ```
 
 Finally, use `mpdev` to install jetstack-secure to the `test-ns` namespace:
@@ -67,7 +109,7 @@ docker run gcr.io/cloud-marketplace-tools/k8s/dev cat /scripts/dev > /tmp/mpdev
 
 kubectl create ns test-ns
 kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/marketplace-k8s-app-tools/master/crd/app-crd.yaml
-mpdev install --deployer=$REGISTRY/$APP_NAME/deployer --parameters='{"name": "test-ns", "namespace": "test"}'
+mpdev install --deployer=$REGISTRY/$SOLUTION/deployer --parameters='{"name": "test-ns", "namespace": "test"}'
 ```
 
 Now, we need to have access to a CAS root. To create a "root" certificate
@@ -166,7 +208,33 @@ Requirements before running `gcloud builds`:
      --workload-pool=$(gcloud config get-value project | tr ':' '/').svc.id.goog
    ```
 
-2. Go to [IAM and Admin > Permissions for
+2. A Google CAS root and subordinate CA as well as a Google service account
+   that will be "attached" to the Kubernetes service account that will be
+   created by the deployer:
+
+   ```sh
+   gcloud beta privateca roots create my-ca --subject="CN=root,O=my-ca"
+   gcloud beta privateca subordinates create my-sub-ca  --issuer=my-ca --location us-east1 --subject="CN=intermediate,O=my-ca,OU=my-sub-ca"
+   gcloud iam service-accounts create sa-google-cas-issuer
+   gcloud beta privateca subordinates add-iam-policy-binding my-sub-ca \
+     --role=roles/privateca.certificateRequester \
+     --member=serviceAccount:sa-google-cas-issuer@$(gcloud config get-value project | tr ':' '/').iam.gserviceaccount.com
+   gcloud iam service-accounts add-iam-policy-binding sa-google-cas-issuer@$(gcloud config get-value project | tr ':' '/').iam.gserviceaccount.com \
+     --role roles/iam.workloadIdentityUser \
+     --member "serviceAccount:$(gcloud config get-value project | tr ':' '/').svc.id.goog[test-ns/test-google-cas-issuer-serviceaccount-name]"
+   ```
+
+   > Note: the last step which is adding the annotation to the
+   > google-cas-issuer Kubernetes service account is done in
+   > `cloudbuild.yml`. The annotation will look like:
+   >
+   >  ```yaml
+   >  metadata:
+   >    annotations:
+   >      iam.gke.io/gcp-service-account=sa-google-cas-issuer@PROJECT_ID.iam.gserviceaccount.com
+   >  ```
+
+3. Go to [IAM and Admin > Permissions for
    project](https://console.cloud.google.com/iam-admin/iam) and configure
    the `[email protected]` service account with the
    following roles so that it has permission to deploy RBAC configuration
@@ -175,7 +243,7 @@ Requirements before running `gcloud builds`:
    - `Kubernetes Engine Admin`
    - `Storage Object Admin`
 
-3. Create a bucket that has the same name as your project. To create it,
+4. Create a bucket that has the same name as your project. To create it,
    run:
 
    ```sh
@@ -189,7 +257,73 @@ gcloud builds submit --timeout 1800s --config cloudbuild.yaml \
   --substitutions _CLUSTER_NAME=$GKE_CLUSTER_NAME,_CLUSTER_LOCATION=$GKE_CLUSTER_LOCATION
 ```
 
-This will also verify the application using the [Google Cloud Marketplace verification tool](https://github.com/GoogleCloudPlatform/marketplace-k8s-app-tools/blob/c5899a928a2ac8d5022463c82823284a9e63b177/scripts/verify).
+This will run [`mpdev verify`]([Google Cloud Marketplace verification
+tool](https://github.com/GoogleCloudPlatform/marketplace-k8s-app-tools/blob/c5899a928a2ac8d5022463c82823284a9e63b177/scripts/verify)),
+which runs [smoke tests](/smoke-test.yaml).
+
+Note that debugging `mpdev verify` is quite tricky. In order to inspect the
+state of the namespace created by `mpdev verify`, we can artificially pause
+`mpdev verify` when it tries to [delete the application](https://github.com/GoogleCloudPlatform/marketplace-k8s-app-tools/blob/4ecf535/scripts/verify#L301-L304):
+
+### Debugging deployer and smoke-tests when run in Cloud Build
+
+There is no official spec for the `smoke-test.yaml` file, although there is
+the example [suite.yaml](https://github.com/GoogleCloudPlatform/marketplace-testrunner/blob/4245fa9/specs/testdata/suite.yaml):
+
+```yaml
+actions:
+- name: {{ .Env.TEST_NAME }}
+  httpTest:
+    url: http://{{ .Var.MainVmIp }}:9012
+    expect:
+      statusCode:
+        equals: 200
+      statusText:
+        contains: OK
+      bodyText:
+        html:
+          title:
+            contains: Hello World!
+- name: Update success variable
+  gcp:
+    setRuntimeConfigVar:
+      runtimeConfigSelfLink: https://runtimeconfig.googleapis.com/v1beta1/projects/my-project/configs/my-config
+      variablePath: status/success
+      base64Value: c3VjY2Vzcwo=
+- name: Can echo to stdout and stderr
+  bashTest:
+    script: |-
+      echo "Text1"
+      >2& echo "Text2"
+    expect:
+      exitCode:
+        equals: 0
+        notEquals: 1
+      stdout:
+        contains: "Text1"
+        notContains: "Foo"
+        matches: "T.xt1"
+      stderr:
+        contains: "Text2"
+        notContains: "Foo"
+        matches: "T.xt2"
+```
+
+Unfortunately, the `stdout` or `stderr` output won't be shown whenever a
+step fails. Reason: the
+[logic in bash.go](https://github.com/GoogleCloudPlatform/marketplace-testrunner/blob/4245fa9/tests/bash.go#L88-L96)
+first checks the status code and returns if mismatch, then checks the
+stdout and returns if mismatch, and finally checks stderr.
+
+**Workaround:**: add to `smoke-test.yaml` a step that hangs, e.g.:
+
+```yaml
+  - name: hang for debugging purposes
+    bashTest:
+      script: sleep 1200
+```
+
+then you can `exec` into the snoke-test pod and debug around.
 
 ## Updating the upstream cert-manager chart version
 
@@ -201,3 +335,6 @@ bump the version of the cert-manager chart in requirements.yaml. Then:
 helm repo add jetstack https://charts.jetstack.io
 helm dependency build chart/jetstacksecure-mp
 ```
+
+[schema.md]: https://github.com/GoogleCloudPlatform/marketplace-k8s-app-tools/blob/d9d3a6f/docs/schema.md
+[jetstack-secure-for-cert-manager]: https://console.cloud.google.com/partner/editor/jetstack-public/jetstack-secure-for-cert-manager?project=jetstack-public

chart/jetstacksecure-mp/charts/google-cas-issuer/Chart.yaml

@@ -14,4 +14,4 @@ maintainers:
   - name: "Jetstack Ltd"
     email: "[email protected]"
     url: "https://www.jetstack.io/"
-appVersion: "0.1.0"
+appVersion: "0.1.0"

chart/jetstacksecure-mp/charts/google-cas-issuer/templates/crds.yaml

@@ -1,4 +1,4 @@
-{{ if .Values.installCRDs }}
+{{- if .Values.installCRDs }}
 apiVersion: apiextensions.k8s.io/v1beta1
 kind: CustomResourceDefinition
 metadata:
@@ -243,4 +243,4 @@ status:
     plural: ""
   conditions: []
   storedVersions: []
-{{ end }}
+{{- end }}

chart/jetstacksecure-mp/charts/google-cas-issuer/values.yaml

@@ -18,16 +18,18 @@ imagePullSecrets: {}
 securityContext: {}
 podSecurityContext: {}
 
-resources:
-  {}
-  # requests:
-  #   cpu: 10m
-  #   memory: 32Mi
-
 nodeSelector: {}
 affinity: {}
 tolerations: {}
 
 prometheus:
   # Enables the creation of the ClusterIP service.
   enabled: true
+
+resources:
+  limits:
+    cpu: 200m
+    memory: 200Mi
+  requests:
+    cpu: 200m
+    memory: 200Mi

chart/jetstacksecure-mp/charts/preflight/templates/deployment.yaml

@@ -1,3 +1,4 @@
+{{ if not "disabled-for-now-due-to-missing-configmap" }}
 apiVersion: apps/v1
 kind: Deployment
 metadata:
@@ -29,7 +30,7 @@ spec:
           secretName: agent-credentials
       containers:
       - name: {{ .Chart.Name }}
-        image: "{{ .Values.image.repository }}:v{{ .Chart.AppVersion }}"
+        image: "{{ .Values.image.repository }}:{{ .Chart.AppVersion }}"
         args:
         - "agent"
         - "-c"
@@ -47,3 +48,4 @@ spec:
           readOnly: true
         resources:
           {{- toYaml .Values.resources | nindent 10 }}
+{{- end }}

chart/jetstacksecure-mp/templates/application.yaml

@@ -12,7 +12,7 @@ metadata:
 spec:
   descriptor:
     type: Jetstack Secure Platform
-    version: "1.0.0"
+    version: "1.1.0-gcm.1" # Must be the same as in the schema.yaml file.
   selector:
     matchLabels:
       app.kubernetes.io/name: "{{ .Release.Name }}"

chart/jetstacksecure-mp/values.yaml

@@ -31,6 +31,7 @@ preflight:
 
 ubbagent:
   # reportingSecretName: "some-secret-name"
-  image: {}
+  image:
+    {}
     # tag: 1.1.0
     # repository: gcr.io/jetstack-public/ubbagent