Skip to content

Conversation

@maelvls
Copy link
Member

@maelvls maelvls commented Jan 21, 2021

→ PR recreated in #3 (we closed the PR and could not re-open it)

@maelvls
Copy link
Member Author

maelvls commented Jan 26, 2021

I propose to start this repo "from scratch" instead of starting off the existing click-to-deploy (C2D).

Rationale:

We realized that click-to-deploy (C2D) somehow creates a single image instead of the 3 we usually have on quay.io. The image marketplace.gcr.io/google/cert-manager0:0.13 contains the webhook, controller, cainjector and acmesolver in /app. We searched everywhere in click-to-deploy and cannot find where the single-image-for-multiple-binaries Dockerfile is.

Let's skip trying to retro-engineer this piece of spaghetti Makefile totally undocumented!

export CLOUDSDK_CORE_PROJECT=jetstack-mael-valais
gsutil mb gs://$CLOUDSDK_CORE_PROJECT
git clone https://github.com/GoogleCloudPlatform/click-to-deploy
gcloud auth configure-docker
cd k8s/cert-manager
make app/build

And we would get stuck on trying to figure out where the cert-manager0 image comes from:

% make app/build
---- REGISTRY = gcr.io/jetstack-mael-valais
---- GCS_URL = gs://jetstack-mael-valais
---- NAMESPACE = default
---- image-cert-manager = Error response from daemon: manifest for marketplace.gcr.io/google/cert-manager0:1.1 not found: manifest unknown: Failed to fetch "1.1" from request "/v2/google/cert-manager0/manifests/1.1".

@maelvls
Copy link
Member Author

maelvls commented Jan 27, 2021

@wallrj I am stuck with one thing we do in the cert-manager helm chart. The webhook' secret name is configured using the chart's name with:

kind: Role
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: [ '{{ template "webhook.fullname" . }}-ca' ]   # ⚠️
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: {{ template "webhook.fullname" . }}:dynamic-serving

The problem with the schema.yaml that the Google Markeplace expects is that
I cannot use such templating; for example, the following will not work
since the name cert-manager-webhook-ca should be in reality something
like {{.Release}}-webhook-ca:

properties:
  cert-manager.webhook.serviceAccount.name:
    type: string
    x-google-marketplace:
      type: SERVICE_ACCOUNT
      serviceAccount:
        roles:
          - type: Role
            rulesType: CUSTOM
            rules:
              - apiGroups: [""]
                resources: ["secrets"]
                resourceNames: ["cert-manager-webhook-ca"]    #
                verbs: ["get", "list", "watch", "update"]
              - apiGroups: [""]
                resources: ["secrets"]
                verbs: ["create"]

Here is where this secret is used:

kind: CustomResourceDefinition
metadata:
  annotations:
    cert-manager.io/inject-ca-from-secret: '{{ template "webhook.caRef" . }}'
spec:
  conversion:
    webhook:
      clientConfig:
        service:
          name: '{{ template "webhook.fullname" . }}'
---
kind: ValidatingWebhookConfiguration
metadata:
  annotations:
    cert-manager.io/inject-ca-from-secret: "{{ .Release.Namespace }}/{{ template "webhook.fullname" . }}-ca"

@wallrj wallrj closed this Jan 27, 2021
@wallrj
Copy link
Member

wallrj commented Jan 27, 2021

Oops, wrong button!

@wallrj wallrj reopened this Jan 27, 2021
@wallrj
Copy link
Member

wallrj commented Jan 27, 2021

I found this in the click-to-deploy repo:

richard   master  ~  projects  GoogleCloudPlatform  click-to-deploy  git grep webhook-ca
k8s/cert-manager/chart/cert-manager/templates/_helpers.tpl:{{ printf "%s-webhook-ca" $trimmedName }}
k8s/cert-manager/chart/cert-manager/templates/deployment/controller.yaml:          - --webhook-ca-secret={{ include "webhook.rootCACertificate" . }}
 richard   master  ~  projects  GoogleCloudPlatform  click-to-deploy  git grep 'webhook.rootCACertificate'
k8s/cert-manager/chart/cert-manager/templates/_helpers.tpl:{{- define "webhook.rootCACertificate" -}}
k8s/cert-manager/chart/cert-manager/templates/deployment/controller.yaml:          - --webhook-ca-secret={{ include "webhook.rootCACertificate" . }}

@wallrj
Copy link
Member

wallrj commented Jan 27, 2021

@maelvls I added a cloudbuild.yaml file which almost works.
I can see it deploy the deployer and then cert-manager, but it fails in various places due to insufficient permissions of the service account of the test deployment. E.g.

Step #14 - "verify": from server for: "/data/resources.yaml": mutatingwebhookconfigurations.admissionregistration.k8s.io "apptest-515lpdy5-cert-manager-webhook" is forbidden: User "system:serviceaccount:apptest-515lpdy5:apptest-515lpdy5-deployer-sa" cannot get resource "mutatingwebhookconfigurations" in API group "admissionregistration.k8s.io" at the cluster scope
Step #14 - "verify": from server for: "/data/resources.yaml": customresourcedefinitions.apiextensions.k8s.io "issuers.cert-manager.io" is forbidden: User "system:serviceaccount:apptest-515lpdy5:apptest-515lpdy5-deployer-sa" cannot get resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope

I changed all the version numbers in the repo to 1.1.0 because I saw ImagePull errors when it attempted to pull the wrong image tag, but I think I may have gone too far.
I suppose the 0.1.0 version number in application.yaml may represent the version of the JSP marketplace package rather than the version of cert-manager that we're deploying. Change it back if I've broken it for you.

@maelvls
Copy link
Member Author

maelvls commented Jan 27, 2021

it fails in various places due to insufficient permissions of the service account of the test deployment

I'm very much struggling with that at the moment, I need to figure out how to fix all these permission/missing rbac issues...

Do you want to pair together on this tomorrow?

@wallrj
Copy link
Member

wallrj commented Jan 27, 2021

I think we just need to copy all the permissions from the click-to-deploy repo: https://github.com/GoogleCloudPlatform/click-to-deploy/blob/a86db3278f8107309c2faaf06c284bce313b8600/k8s/cert-manager/schema.yaml#L187 and probably some extra since we're deploying a much newer version of CM.

@maelvls
Copy link
Member Author

maelvls commented Jan 27, 2021

Do you think I can try gcloud build? Which command should I type to run a build?

@wallrj
Copy link
Member

wallrj commented Jan 27, 2021

Do you think I can try gcloud build? Which command should I type to run a build?

Yeah, it should work for you too. I documented the command in the README file. You will need to change the cluster name and zone to match your GKE cluster. 🤞

@wallrj wallrj closed this Jan 28, 2021
@wallrj
Copy link
Member

wallrj commented Jan 28, 2021

I keep doing this. Sorry.

@wallrj
Copy link
Member

wallrj commented Jan 28, 2021

Now I can't re-open :-(

@wallrj
Copy link
Member

wallrj commented Jan 28, 2021

The google cloud build now succeeds:

kubectl -n apptest-x131bfcl logs apptest-x131bfcl-deployer-l7tq7  --follow

...
INFO Top level resources: 1
INFO Initialization: Found applications.app.k8s.io/apptest-x131bfcl ready status to be True.
INFO Wait 30 seconds to make sure app stays in healthy state.
INFO Top level resources: 1
INFO Top level resources: 1
INFO Top level resources: 1
INFO Top level resources: 1
INFO Top level resources: 1
INFO Top level resources: 1
+ tester_manifest=/data/tester.yaml
+ [[ -e /data/tester.yaml ]]
+ echo 'SMOKE_TEST No tester manifest found at /data/tester.yaml.'
+ clean_iam_resources.sh
+ [[ -z apptest-x131bfcl ]]
+ [[ -z apptest-x131bfcl ]]
+ kubectl delete --namespace=apptest-x131bfcl ServiceAccount -l app.kubernetes.io/component=deployer.marketplace.cloud.google.com,app.kubernetes.io/name=apptest-x131bfcl --ignore-not-found
SMOKE_TEST No tester manifest found at /data/tester.yaml.
Using /opt/kubectl/1.16/kubectl (server=1.16)
serviceaccount "apptest-x131bfcl-deployer-sa" deleted
+ trap - EXIT

gcloud builds submit  --timeout 1800s --config cloudbuild.yaml
...

41e2d890e349: Layer already exists
6071362d21ac: Layer already exists
965eb7efdb65: Layer already exists
84ff92691f90: Layer already exists
1.1.0: digest: sha256:19b449847233bee2d8df69abf0ff0440c10cbaa468ca5e2274b24b97d48ada1e size: 5550
DONE
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

ID                                    CREATE_TIME                DURATION  SOURCE                                                                                          IMAGES                                                                        STATUS
ff1aa41a-bff5-4465-a11e-b72a2f4ff880  2021-01-28T16:10:27+00:00  3M19S     gs://jetstack-richard_cloudbuild/source/1611850225.728782-97c242f502764c428889f42b1423c8c8.tgz  gcr.io/jetstack-richard/cert-manager/cert-manager-controller:1.1.0 (+3 more)  SUCCESS

@maelvls
Copy link
Member Author

maelvls commented Jan 28, 2021

🎉🎉🎉🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants