Kubernetes Roles use incorrect syntax for single resources

[ricky26]

Describe the bug
At the moment the Helm chart produces roles with resource set to kind/resource-name (e.g. secrets/taskcluster-ui). This causes issues when deploying from restricted service accounts where RBAC is enabled.

To Reproduce
Steps to reproduce the behavior:

1. Use helm template on the chart in infrastructure/k8s with a valid values.yml.
2. Look at the generated Roles.
3. Notice that the resource field doesn't match the documentation.

Alternatively:

1. Create a service account with appropriate roles to deploy Taskcluster (in particular all verbs for secrets).
2. Try and deploy Taskcluster.
3. Notice errors during deployment:

roles.rbac.authorization.k8s.io \"taskcluster-web-server-secrets-read\" is forbidden: user \"system:serviceaccount:taskcluster-operator:default\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:taskcluster-operator\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"\"], Resources:[\"secrets/taskcluster-web-server\"], Verbs:[\"get\" \"watch\" \"list\"]}

Expected behavior
Kubernetes Roles should use resourceNames to specify individual resources: https://kubernetes.io/docs/reference/access-authn-authz/rbac/#referring-to-resources.

Taskcluster version
v37.2.0

Kubernetes Version
1.16.13-gke.401

[ricky26]

@imbstack, did you want to ask ops about this one? I'm happy to create a patch if the resolution is valid.

Actually... are these RBAC roles used? I'm pretty sure you don't need them to use envFrom and I don't see any references to the Kubernetes API in the services.

[imbstack]

@sciurus does this make sense with how cloudops deploys tc?

[sciurus]

Based on the docs I think @ricky26 's analysis of resources vs resourceNames is correct. I've asked a teammate who's more familiar with K8S RBAC for advice on the overall usage of it within Taskcluster's helm charts.

[sciurus]

The summary of my conversation with @erkolson is that Taskcluster does not need the roles or rolebindings. We should stop generating those and can delete them from our existing clusters. You only need to use RBAC if your software interacts with the Kubernetes API, which Taskcluster does not.

@ricky26 we noticed you created an operator for running Taskcluster. Your error is likely because your operator is running with the default service account of the namespace and it doesn't have permissions to modify RBAC. Once we remove the roles and rolebindings, that problem should go away.

[ricky26]

@sciurus I am indeed running with the namespace default service account but it has global secrets read/write permissions (it needs them to store generated passwords/access tokens and TC instances could be in any namespace). Either way, yes removing the roles would solve my issue! :)