Fix the smoke-test tester image

[maelvls]

While reviewing the Jetstack Secure for cert-manager solution, Google bumped into an issue:

LAST SEEN   TYPE      REASON                 OBJECT                          MESSAGE
2s          Warning   BackoffLimitExceeded   job/apptest-4v6dctp1-deployer   Job has reached the specified backoff limit
3m41s       Warning   Failed                 pod/smoke-test-pod              Failed to pull image "gcr.io/jetstack-mael-valais/jetstack-secure-for-cert-manager/smoke-test:1.1.0-gcm.1": rpc error: code = Unknown desc = Error response from daemon: pull access denied for gcr.io/jetstack-mael-valais/jetstack-secure-for-cert-manager/smoke-test, repository does not exist or may require 'docker login': denied: Permission denied for "1.1.0-gcm.1" from request "/v2/jetstack-mael-valais/jetstack-secure-for-cert-manager/smoke-test/manifests/1.1.0-gcm.1".
3m41s       Warning   Failed                 pod/smoke-test-pod              Error: ErrImagePull
3m26s       Warning   Failed                 pod/smoke-test-pod              Error: ImagePullBackOff

The "tester image" mentioned in the above logs is the image used by the "tester pod" as described in verification-integration.md. The tester image is built and pushed as part of our cloud-build.yaml.

In the above error, the tester pod seems to be using the wrong image:

gcr.io/jetstack-mael-valais/jetstack-secure-for-cert-manager/smoke-test:1.1.0-gcm.1

Instead, it should show:

gcr.io/jetstack-public/jetstack-secure-for-cert-manager/smoke-test:1.1.0-gcm.1

It seems like the helm value smokeTestImage is hardcoded into the deployer:1.1.0-gcm.1 image, which is something I did not think about when I used envsubst at cloudbuild.yaml#L176. The idea is to set the correct tester image in data-test/schema.yaml:

# data-test/schema.yaml
properties:
  smokeTestImage:   # used as "{{.Values.smokeTestImage}}" in tester.yaml
    type: string
    default: $IMAGE.  # ← envsubt'd
    x-google-property:
      type: IMAGE

What I did not realize at the time is that the the deployer:1.1.0-gcm.1 image would end up with:

# data-test/schema.yaml
properties:
  smokeTestImage:
    type: string
    default: gcr.io/jetstack-mael-valais/jetstack-secure-for-cert-manager/smoke-test:1.1.0-gcm.1
    x-google-property:
      type: IMAGE

The whole issue is that the deployer:1.1.0-gcm.1 has not been built directly from the jetstack-public project.

Solutions:

1. Use x-google-marketplace.images.smoke-test instead of properties.smokeTestImage.
2. Use the jetstack-public project in order to build the deployer:1.1.0-gcm.1 image. Until today, I wasn't able to use it because of missing permissions, so I ended up using my own project jetstack-mael-valais.

Since (1) does not seem possible as detailed in the below #29 (comment)). We could ask Google if that is intentional or not.

In the meantime, I propose that we go with the solution (2). I will investigate what permissions are missing and report as a comment to this issue.

cc @james-w

[maelvls]

I studied the possibility of using x-google-marketplace.images.smoke-test instead of properties.smokeTestImage.

to;dr: This solution does not seem to be supported by mpdev verify.

In #30, I tried using the x-google-marketplace.images field that I also use in /schema.yaml. With this, the mpdev verify command should be able to fill in the right image:

# data-test/schema.yaml
x-google-marketplace:
  images:
    smoke-test:
      properties:
        smokeTestImage:
          type: FULL

Unfortunately, the image field on the pod stays empty:

% gcloud builds submit  --timeout 1800s --config cloudbuild.yaml \
  --substitutions _CLUSTER_NAME=foobar,_CLUSTER_LOCATION=us-east1-b
Step #28 - "verify": INFO Top level resources: 2
Step #28 - "verify": + tester_manifest=/data/tester.yaml
Step #28 - "verify": + [[ -e /data/tester.yaml ]]
Step #28 - "verify": + cat /data/tester.yaml
Step #28 - "verify": apiVersion: v1
Step #28 - "verify": kind: Pod
Step #28 - "verify": metadata:
Step #28 - "verify":   name: smoke-test-pod
Step #28 - "verify": spec:
Step #28 - "verify":   containers:
Step #28 - "verify":   - image: ''                                  # ❌

I'm not too sure how to fix the "hardcoded image" issue. How are we supposed to be letting mpdev verify set the image at runtime? In mpdev-references.md, it says:

Properties in /data-test/schema.yaml will override properties in /data/schema.yaml. This can also be used to overwrite existing default values for verification.

@vcanaa after looking at this commit (2019), I wonder:

• is mpdev verify only supporting the top-level properties field? e.g.:

  # data-test/schema.yaml
  properties:
    smokeTestImage: # used as "{{.Values.smokeTestImage}}" in tester.yaml
      type: string
      # These variables are envsubt'd in cloudbuild.yml
      default: $IMAGE
      x-google-property:
        type: IMAGE

• Or can we also use it for the x-google-marketplace.images field? e.g.:

  # data-test/schema.yaml
   x-google-marketplace:
    images:
      smoke-test:
        properties:
          smokeTestImage: # used as "{{.Values.smokeTestImage}}" in tester.yaml
            type: FULL

This information is not available in mpdev-references.md unfortunately.

[maelvls]

I am now working on the solution (2), see https://github.com/jetstack/platform-board/issues/331 (private to jetstack)

Update: done! Next steps:

• Disable the re-tagging in Disable retagging in cloudbuild.yaml due to the licenses requirements #33
• Run a gcloud builds so that the deployer image is recreated

  gcloud builds submit --project jetstack-public --timeout 1800s --config cloudbuild.yaml \
    --substitutions _CLUSTER_NAME=smoke-test,_CLUSTER_LOCATION=europe-west2-b

  → Done, see build logs
• Check locally that the data-test/schema.yaml is correct:

  docker pull gcr.io/jetstack-public/jetstack-secure-for-cert-manager/deployer:1.1
  docker run -it --rm --entrypoint=bash gcr.io/jetstack-public/jetstack-secure-for-cert-manager/deployer:1.1 \
    -c "cat /data-test/schema.yaml" | grep smoke-test:

  → Looks nice:

      default: gcr.io/jetstack-public/jetstack-secure-for-cert-manager/smoke-test:1.1.0-gcm.1
  

• Go into the Marketplace UI trackOverviewPanel panel and update the 1.1 version:
  
• Re-submit the solution using the trackOverviewPanel form (done on 5 March 2021 13:30 UTC)

[maelvls]

Problem fixed using solution (2)!

/close

[jetstack-bot]

@maelvls: Closing this issue.

In response to this:

Problem fixed using solution (2)!

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.