system:serviceaccount:cert-manager:csi-snapshotter cannot create resource ...

[tpoxa]

~/csi-driver-host-path/deploy/kubernetes-1.16$ ./deploy-hostpath.sh

Installation failed after 5m tmeout

pod/csi-hostpath-snapshotter-0                0/1     CrashLoopBackOff   7          14m 

Status:"Failure", Message:"customresourcedefinitions.apiextensions.k8s.io is forbidden: User \"system:serviceaccount:cert-manager:csi-snapshotter\" cannot create resource \"customresourcedefinitions\" in API group \"apiextensions.k8s.io\" at the cluster scope", Reason:"Forbidden"

Linux Debian-911-stretch-64-minimal 4.9.0-11-amd64 #1 SMP Debian 4.9.189-3 (2019-09-02) x86_64 GNU/Linux

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:23:11Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

[pohly]

https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/deploy/kubernetes-1.16/hostpath/csi-hostpath-snapshotter.yaml#L43 install v1.2.0 of the external-snapshotter, which means that the deploy script should download the corresponding RBAC rules, which have: https://github.com/kubernetes-csi/external-snapshotter/blob/b3f591d85cce516e431c70e5337d5c67611ae2fe/deploy/kubernetes/rbac.yaml#L50-L52

"Should" - some of the recent changes to the script broke the content of CSI_SNAPSHOTTER_RBAC_YAML, leading to it not just containing the URL but also some extra garbage:

Using non-default RBAC rules for CSI_SNAPSHOTTER. Changes from https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v1.2.0/deploy/kubernetes/rbac.yaml to https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v1.2.0/deploy/kubernetes/rbac.yaml
CSI_RESIZER_RBAC_YAML=https://raw.githubusercontent.com/kubernetes-csi/external-resizer/v0.3.0/deploy/kubernetes/rbac.yaml
: https://raw.githubusercontent.com/kubernetes-csi/external-resizer/v0.3.0/deploy/kubernetes/rbac.yaml

INSTALL_CRD=false

# Some images are not affected by *_REGISTRY/*_TAG and IMAGE_* variables.
# The default is to update unless explicitly excluded.
update_image () {
    case  in socat) return 1;; esac
 are:

Everything between CSI_RESIZER_RBAC_YAML (inclusive) and are (exclusive) shouldn't be there.

The reason is a missing closing bracket, this fixes it:

diff --git a/deploy/util/deploy-hostpath.sh b/deploy/util/deploy-hostpath.sh
index 69d790de..bb5148a0 100755
--- a/deploy/util/deploy-hostpath.sh
+++ b/deploy/util/deploy-hostpath.sh
@@ -121,7 +121,7 @@ CSI_PROVISIONER_RBAC_YAML="https://raw.githubusercontent.com/kubernetes-csi/exte
 CSI_ATTACHER_RBAC_YAML="https://raw.githubusercontent.com/kubernetes-csi/external-attacher/$(rbac_version "${BASE_DIR}/hostpath/csi-hostpath-attacher.yaml" csi-attacher false)/deploy/kubernetes/rbac.yaml"
 : ${CSI_ATTACHER_RBAC:=https://raw.githubusercontent.com/kubernetes-csi/external-attacher/$(rbac_version "${BASE_DIR}/hostpath/csi-hostpath-attacher.yaml" csi-attacher "${UPDATE_RBAC_RULES}")/deploy/kubernetes/rbac.yaml}
 CSI_SNAPSHOTTER_RBAC_YAML="https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/$(rbac_version "${BASE_DIR}/hostpath/csi-hostpath-snapshotter.yaml" csi-snapshotter false)/deploy/kubernetes/${SNAPSHOTTER_RBAC_RELATIVE_PATH}"
-: ${CSI_SNAPSHOTTER_RBAC:=https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/$(rbac_version "${BASE_DIR}/hostpath/csi-hostpath-snapshotter.yaml" csi-snapshotter "${UPDATE_RBAC_RULES}")/deploy/kubernetes/${SNAPSHOTTER_RBAC_RELATIVE_PATH}
+: ${CSI_SNAPSHOTTER_RBAC:=https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/$(rbac_version "${BASE_DIR}/hostpath/csi-hostpath-snapshotter.yaml" csi-snapshotter "${UPDATE_RBAC_RULES}")/deploy/kubernetes/${SNAPSHOTTER_RBAC_RELATIVE_PATH}}
 CSI_RESIZER_RBAC_YAML="https://raw.githubusercontent.com/kubernetes-csi/external-resizer/$(rbac_version "${BASE_DIR}/hostpath/csi-hostpath-resizer.yaml" csi-resizer false)/deploy/kubernetes/rbac.yaml"
 : ${CSI_RESIZER_RBAC:=https://raw.githubusercontent.com/kubernetes-csi/external-resizer/$(rbac_version "${BASE_DIR}/hostpath/csi-hostpath-resizer.yaml" csi-resizer "${UPDATE_RBAC_RULES}")/deploy/kubernetes/rbac.yaml}
 

The more interesting question is: why was this not caught by the CI when merging #98?

The test job shows the same broken output (https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/kubernetes-csi_csi-driver-host-path/98/pull-kubernetes-csi-csi-driver-host-path-1-16-on-kubernetes-1-16/1202409554004938755/build-log.txt).

Does the KinD cluster perhaps not have RBAC enforcement turned on?

/cc @xing-yang @ggriffiths

[pohly]

/cc @msau42

[pohly]

The other problem is that the wrong URL didn't lead to some obvious error and early aborting of the script. For wget, it is the --quiet option which suppresses the error reporting and we don't check the return code during the diff. But that's just for diagnostics.

The actual download happens with kubectl apply -f. My version of kubectl (v1.16.0-rc.1.19+4cb51f0d2d8392) seems to ignore the extra garbage:

$ kubectl apply -f "https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v1.2.0/deploy/kubernetes/rbac.yaml
hello

fjahud72="
serviceaccount/csi-snapshotter unchanged
clusterrole.rbac.authorization.k8s.io/external-snapshotter-runner unchanged
clusterrolebinding.rbac.authorization.k8s.io/csi-snapshotter-role unchanged
role.rbac.authorization.k8s.io/external-snapshotter-leaderelection unchanged
rolebinding.rbac.authorization.k8s.io/external-snapshotter-leaderelection unchanged

@tpoxa: does your version perhaps handle this differently? Which version is that?

I bet there is a difference in kubectl and that also explains why it worked in the CI, because the CI run does install the RBAC rules:

kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v1.2.0/deploy/kubernetes/rbac.yaml
CSI_RESIZER_RBAC_YAML=https://raw.githubusercontent.com/kubernetes-csi/external-resizer/v0.3.0/deploy/kubernetes/rbac.yaml
: https://raw.githubusercontent.com/kubernetes-csi/external-resizer/v0.3.0/deploy/kubernetes/rbac.yaml

INSTALL_CRD=false

# Some images are not affected by *_REGISTRY/*_TAG and IMAGE_* variables.
# The default is to update unless explicitly excluded.
update_image () {
    case  in socat) return 1;; esac

serviceaccount/csi-snapshotter created
clusterrole.rbac.authorization.k8s.io/external-snapshotter-runner created
clusterrolebinding.rbac.authorization.k8s.io/csi-snapshotter-role created
role.rbac.authorization.k8s.io/external-snapshotter-leaderelection created
rolebinding.rbac.authorization.k8s.io/external-snapshotter-leaderelection created

[tpoxa]

This branch did not make big difference for me.

maksym@Debian-911-stretch-64-minimal:~/csi-driver-host-path/deploy/kubernetes-1.16$ ./deploy-hostpath.sh
SNAPSHOTTER_RBAC_RELATIVE_PATH rbac.yaml
applying RBAC rules
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-provisioner/v1.4.0/deploy/kubernetes/rbac.yaml
serviceaccount/csi-provisioner unchanged
clusterrole.rbac.authorization.k8s.io/external-provisioner-runner unchanged
clusterrolebinding.rbac.authorization.k8s.io/csi-provisioner-role unchanged
role.rbac.authorization.k8s.io/external-provisioner-cfg unchanged
rolebinding.rbac.authorization.k8s.io/csi-provisioner-role-cfg unchanged
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-attacher/v2.0.0/deploy/kubernetes/rbac.yaml
serviceaccount/csi-attacher unchanged
clusterrole.rbac.authorization.k8s.io/external-attacher-runner unchanged
clusterrolebinding.rbac.authorization.k8s.io/csi-attacher-role unchanged
role.rbac.authorization.k8s.io/external-attacher-cfg unchanged
rolebinding.rbac.authorization.k8s.io/csi-attacher-role-cfg unchanged
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v1.2.0/deploy/kubernetes/rbac.yaml
serviceaccount/csi-snapshotter created
clusterrole.rbac.authorization.k8s.io/external-snapshotter-runner created
clusterrolebinding.rbac.authorization.k8s.io/csi-snapshotter-role created
role.rbac.authorization.k8s.io/external-snapshotter-leaderelection created
rolebinding.rbac.authorization.k8s.io/external-snapshotter-leaderelection created
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-resizer/v0.3.0/deploy/kubernetes/rbac.yaml
serviceaccount/csi-resizer created
clusterrole.rbac.authorization.k8s.io/external-resizer-runner created
clusterrolebinding.rbac.authorization.k8s.io/csi-resizer-role created
role.rbac.authorization.k8s.io/external-resizer-cfg created
rolebinding.rbac.authorization.k8s.io/csi-resizer-role-cfg created
deploying hostpath components
   ./hostpath/csi-hostpath-attacher.yaml
        using           image: quay.io/k8scsi/csi-attacher:v2.0.0
service/csi-hostpath-attacher created
statefulset.apps/csi-hostpath-attacher created
   ./hostpath/csi-hostpath-driverinfo.yaml
csidriver.storage.k8s.io/hostpath.csi.k8s.io unchanged
   ./hostpath/csi-hostpath-plugin.yaml
        using           image: quay.io/k8scsi/csi-node-driver-registrar:v1.2.0
        using           image: quay.io/k8scsi/hostpathplugin:v1.2.0
        using           image: quay.io/k8scsi/livenessprobe:v1.1.0
service/csi-hostpathplugin created
statefulset.apps/csi-hostpathplugin created
   ./hostpath/csi-hostpath-provisioner.yaml
        using           image: quay.io/k8scsi/csi-provisioner:v1.4.0
service/csi-hostpath-provisioner created
statefulset.apps/csi-hostpath-provisioner created
   ./hostpath/csi-hostpath-resizer.yaml
        using           image: quay.io/k8scsi/csi-resizer:v0.3.0
service/csi-hostpath-resizer created
statefulset.apps/csi-hostpath-resizer created
   ./hostpath/csi-hostpath-snapshotter.yaml
        using           image: quay.io/k8scsi/csi-snapshotter:v1.2.0
service/csi-hostpath-snapshotter created
statefulset.apps/csi-hostpath-snapshotter created
   ./hostpath/csi-hostpath-testing.yaml
        using           image: alpine/socat:1.0.3
service/hostpath-service created
statefulset.apps/csi-hostpath-socat created
09:14:26 waiting for hostpath deployment to complete, attempt #0
09:14:36 waiting for hostpath deployment to complete, attempt #1
09:14:47 waiting for hostpath deployment to complete, attempt #2
09:14:57 waiting for hostpath deployment to complete, attempt #3
09:15:07 waiting for hostpath deployment to complete, attempt #4

kubectl logs pod/csi-hostpath-snapshotter-0
I1209 08:17:39.976602       1 main.go:89] Version: v1.2.0-0-gb3f591d8
W1209 08:17:39.976658       1 main.go:92] --connection-timeout is deprecated and will have no effect
F1209 08:17:39.985822       1 create_crd.go:50] failed to create VolumeSnapshotResource: &v1beta1.CustomResourceDefinition{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1beta1.CustomResourceDefinitionSpec{Group:"", Version:"", Names:v1beta1.CustomResourceDefinitionNames{Plural:"", Singular:"", ShortNames:[]string(nil), Kind:"", ListKind:"", Categories:[]string(nil)}, Scope:"", Validation:(*v1beta1.CustomResourceValidation)(nil), Subresources:(*v1beta1.CustomResourceSubresources)(nil), Versions:[]v1beta1.CustomResourceDefinitionVersion(nil), AdditionalPrinterColumns:[]v1beta1.CustomResourceColumnDefinition(nil), Conversion:(*v1beta1.CustomResourceConversion)(nil)}, Status:v1beta1.CustomResourceDefinitionStatus{Conditions:[]v1beta1.CustomResourceDefinitionCondition(nil), AcceptedNames:v1beta1.CustomResourceDefinitionNames{Plural:"", Singular:"", ShortNames:[]string(nil), Kind:"", ListKind:"", Categories:[]string(nil)}, StoredVersions:[]string(nil)}}, err: &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:""}, Status:"Failure", Message:"customresourcedefinitions.apiextensions.k8s.io is forbidden: User \"system:serviceaccount:cert-manager:csi-snapshotter\" cannot create resource \"customresourcedefinitions\" in API group \"apiextensions.k8s.io\" at the cluster scope", Reason:"Forbidden", Details:(*v1.StatusDetails)(0xc00039c3c0), Code:403}}
User \"system:serviceaccount:cert-manager:csi-snapshotter\" cannot create resource \"customresourcedefinitions\" in API group \"apiextensions.k8s.io\" at the cluster scope", Reason:"Forbidden"

Maybe this is cert-manager related issue?

[pohly]

@tpoxa which version of kubectl are you using?

Can you check that you now get the expected RBAC rules? I have:

$ kubectl get -o yaml pods/csi-hostpath-snapshotter-0 | grep serviceAccount
  serviceAccount: csi-snapshotter
  serviceAccountName: csi-snapshotter

$ kubectl describe serviceaccount/csi-snapshotter
Name:                csi-snapshotter
Namespace:           default
Labels:              <none>
Annotations:         kubectl.kubernetes.io/last-applied-configuration:
                       {"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{},"name":"csi-snapshotter","namespace":"default"}}
Image pull secrets:  <none>
Mountable secrets:   csi-snapshotter-token-8894h
Tokens:              csi-snapshotter-token-8894h
Events:              <none>

$ kubectl describe clusterroles/external-snapshotter-runner
Name:         external-snapshotter-runner
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRole","metadata":{"annotations":{},"name":"external-snapshotter-runner"},"rule...
PolicyRule:
  Resources                                       Non-Resource URLs  Resource Names  Verbs
  ---------                                       -----------------  --------------  -----
  volumesnapshotcontents.snapshot.storage.k8s.io  []                 []              [create get list watch update delete]
  customresourcedefinitions.apiextensions.k8s.io  []                 []              [create list watch delete get update]
  persistentvolumeclaims                          []                 []              [get list watch update]
  volumesnapshots.snapshot.storage.k8s.io         []                 []              [get list watch update]
  persistentvolumes                               []                 []              [get list watch]
  volumesnapshotclasses.snapshot.storage.k8s.io   []                 []              [get list watch]
  storageclasses.storage.k8s.io                   []                 []              [get list watch]
  secrets                                         []                 []              [get list]
  events                                          []                 []              [list watch create update patch]
  volumesnapshots.snapshot.storage.k8s.io/status  []                 []              [update]

$ kubectl describe clusterrolebinding/csi-snapshotter-role
Name:         csi-snapshotter-role
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRoleBinding","metadata":{"annotations":{},"name":"csi-snapshotter-role"},"role...
Role:
  Kind:  ClusterRole
  Name:  external-snapshotter-runner
Subjects:
  Kind            Name             Namespace
  ----            ----             ---------
  ServiceAccount  csi-snapshotter  default

[tpoxa]

kubectl get -o yaml pods/csi-hostpath-snapshotter-0 | grep serviceAccount
  serviceAccount: csi-snapshotter
  serviceAccountName: csi-snapshotter



maksym@Debian-911-stretch-64-minimal:/root$ kubectl describe serviceaccount/csi-snapshotter
Name:                csi-snapshotter
Namespace:           cert-manager
Labels:              <none>
Annotations:         kubectl.kubernetes.io/last-applied-configuration:
                       {"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{},"name":"csi-snapshotter","namespace":"cert-manager"}}
Image pull secrets:  <none>
Mountable secrets:   csi-snapshotter-token-hh2ld
Tokens:              csi-snapshotter-token-hh2ld
Events:              <none>




maksym@Debian-911-stretch-64-minimal:/root$ kubectl describe clusterroles/external-snapshotter-runner
Name:         external-snapshotter-runner
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRole","metadata":{"annotations":{},"name":"external-snapshotter-runner"},"rule...
PolicyRule:
  Resources                                       Non-Resource URLs  Resource Names  Verbs
  ---------                                       -----------------  --------------  -----
  volumesnapshotcontents.snapshot.storage.k8s.io  []                 []              [create get list watch update delete]
  customresourcedefinitions.apiextensions.k8s.io  []                 []              [create list watch delete get update]
  persistentvolumeclaims                          []                 []              [get list watch update]
  volumesnapshots.snapshot.storage.k8s.io         []                 []              [get list watch update]
  persistentvolumes                               []                 []              [get list watch]
  volumesnapshotclasses.snapshot.storage.k8s.io   []                 []              [get list watch]
  storageclasses.storage.k8s.io                   []                 []              [get list watch]
  secrets                                         []                 []              [get list]
  events                                          []                 []              [list watch create update patch]
  volumesnapshots.snapshot.storage.k8s.io/status  []                 []              [update]



maksym@Debian-911-stretch-64-minimal:/root$ kubectl describe clusterrolebinding/csi-snapshotter-role
Name:         csi-snapshotter-role
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRoleBinding","metadata":{"annotations":{},"name":"csi-snapshotter-role"},"role...
Role:
  Kind:  ClusterRole
  Name:  external-snapshotter-runner
Subjects:
  Kind            Name             Namespace
  ----            ----             ---------
  ServiceAccount  csi-snapshotter  default

[pohly]

That looks sane to me. Sorry, I'm out of ideas.

[xing-yang]

@ggriffiths can you please take a look? thanks.

[msau42]

Hold on. Running the 1.16 versions of the hostpath deployment against a 1.12 cluster probably is going to have a lot of problems

[msau42]

Sorry ignore my last comment. I misread the kubectl output. What is the cluster version?

[ggriffiths]

I'm not sure why #98 didn't catch this. I think we tested as far back as
pull-kubernetes-csi-csi-driver-host-path-1-14-on-kubernetes-1-14, so if it's 1.16 hostpath on an older version, that might have issues.

We create the CRDs in prow.sh before the snapshotter is deployed.

Seems like the error is coming from here in the old snapshotter:
https://github.com/kubernetes-csi/external-snapshotter/blob/release-1.2/cmd/csi-snapshotter/create_crd.go#L71

[pohly]

Grant Griffiths <notifications@github.com> writes:

I'm not sure why #98 didn't catch this. I think we tested as far back as `pull-kubernetes-csi-csi-driver-host-path-1-14-on-kubernetes-1-14`, so if it's 1.16 hostpath on an older version, that might have issues.

It seems to depend on the kubectl version. A recent kubectl silently ignored the garbage after the URL.

[tpoxa]

@msau42

maksym@Debian-911-stretch-64-minimal:~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:23:11Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:13:49Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

[tpoxa]

Its clean kubernetes installation. I only have cert-manager installed with Helm.
I see cert-manager mentioned in error message.

kubectl logs pod/csi-hostpath-snapshotter-0
I1209 18:38:03.988445       1 main.go:89] Version: v1.2.0-0-gb3f591d8
W1209 18:38:03.988506       1 main.go:92] --connection-timeout is deprecated and will have no effect
F1209 18:38:03.997233       1 create_crd.go:50] failed to create VolumeSnapshotResource: &v1beta1.CustomResourceDefinition{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1beta1.CustomResourceDefinitionSpec{Group:"", Version:"", Names:v1beta1.CustomResourceDefinitionNames{Plural:"", Singular:"", ShortNames:[]string(nil), Kind:"", ListKind:"", Categories:[]string(nil)}, Scope:"", Validation:(*v1beta1.CustomResourceValidation)(nil), Subresources:(*v1beta1.CustomResourceSubresources)(nil), Versions:[]v1beta1.CustomResourceDefinitionVersion(nil), AdditionalPrinterColumns:[]v1beta1.CustomResourceColumnDefinition(nil), Conversion:(*v1beta1.CustomResourceConversion)(nil)}, Status:v1beta1.CustomResourceDefinitionStatus{Conditions:[]v1beta1.CustomResourceDefinitionCondition(nil), AcceptedNames:v1beta1.CustomResourceDefinitionNames{Plural:"", Singular:"", ShortNames:[]string(nil), Kind:"", ListKind:"", Categories:[]string(nil)}, StoredVersions:[]string(nil)}}, err: &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:""}, Status:"Failure", Message:"customresourcedefinitions.apiextensions.k8s.io is forbidden: User \"system:serviceaccount:cert-manager:csi-snapshotter\" cannot create resource \"customresourcedefinitions\" in API group \"apiextensions.k8s.io\" at the cluster scope", Reason:"Forbidden", Details:(*v1.StatusDetails)(0xc0004261e0), Code:403}}

Maybe its somehow related with Cert-manager finalizers or so... Sorry I am still quite new to k8s.