Helm charts that need more permissions than available on the provider kubeconfig

[devdattakulkarni]

When deploying a helm chart, the kubeplus-saas-provider service account may not have permissions that the Helm chart needs or is trying to grant.

Below are examples where the default kubeplus-saas-provider permissions seem to be inadequate.

With argo-cd chart
we get the following error:
sample-argocdservice:Error: Error: clusterroles.rbac.authorization.k8s.io "argocdservice-sample-argocdservice-argo-cd-sample-argocdservice" is forbidden: user "system:serviceaccount:default:kubeplus-saas-provider" (groups=["system:serviceaccounts" "system:serviceaccounts:default" "system:authenticated"]) is attempting to grant RBAC permissions not currently held: {APIGroups:["*"], Resources:["*"], Verbs:["*"]} {NonResourceURLs:["*"], Verbs:["*"]}

With argo-workflows chart,
we get the following error:
sample-argoworkflowsservice:Error: Error: clusterroles.rbac.authorization.k8s.io "argoworkflowsservice-sample-argoworkflowsservice-argo-workflows-cluster-template" is forbidden: user "system:serviceaccount:default:kubeplus-saas-provider" (groups=["system:serviceaccounts" "system:serviceaccounts:default" "system:authenticated"]) is attempting to grant RBAC permissions not currently held: {APIGroups:["argoproj.io"], Resources:["clusterworkflowtemplates"], Verbs:["create" "update" "patch" "delete"]}

With cert-manager chart
we get the following error:
sample-certmanagerservice:Error: Error: clusterroles.rbac.authorization.k8s.io "certmanagerservice-sample-certmanagerservice-cert-manager-cainj" is forbidden: user "system:serviceaccount:default:kubeplus-saas-provider" (groups=["system:serviceaccounts" "system:serviceaccounts:default" "system:authenticated"]) is attempting to grant RBAC permissions not currently held: {APIGroups:["admissionregistration.k8s.io"], Resources:["validatingwebhookconfigurations"], Verbs:["update"]} {APIGroups:["apiregistration.k8s.io"], Resources:["apiservices"], Verbs:["update"]} {APIGroups:["auditregistration.k8s.io"], Resources:["auditsinks"], Verbs:["update"]}

With contour chart
we get the following error:
Helmrelease: sample-cts:Error: Error: clusterroles.rbac.authorization.k8s.io "contourservice-sample-cts-contour" is forbidden: user "system:serviceaccount:default:kubeplus-saas-provider" (groups=["system:serviceaccounts" "system:serviceaccounts:default" "system:authenticated"]) is attempting to grant RBAC permissions not currently held: {APIGroups:["gateway.networking.k8s.io"], Resources:["gatewayclasses/status"], Verbs:["update"]} {APIGroups:["gateway.networking.k8s.io"], Resources:["gateways/status"], Verbs:["update"]} {APIGroups:["gateway.networking.k8s.io"], Resources:["httproutes/status"], Verbs:["update"]} {APIGroups:["gateway.networking.k8s.io"], Resources:["tcproutes/status"], Verbs:["update"]} {APIGroups:["gateway.networking.k8s.io"], Resources:["tlsroutes/status"], Verbs:["update"]} {APIGroups:["gateway.networking.k8s.io"], Resources:["udproutes/status"], Verbs:["update"]} {APIGroups:["networking.k8s.io"], Resources:["ingresses/status"], Verbs:["create" "update"]} {APIGroups:["networking.x-k8s.io"], Resources:["gatewayclasses/status"], Verbs:["update"]} {APIGroups:["networking.x-k8s.io"], Resources:["gateways/status"], Verbs:["update"]} {APIGroups:["networking.x-k8s.io"], Resources:["httproutes/status"], Verbs:["update"]} {APIGroups:["networking.x-k8s.io"], Resources:["tcproutes/status"], Verbs:["update"]} {APIGroups:["networking.x-k8s.io"], Resources:["tlsroutes/status"], Verbs:["update"]} {APIGroups:["networking.x-k8s.io"], Resources:["udproutes/status"], Verbs:["update"]} {APIGroups:["projectcontour.io"], Resources:["contourconfigurations/status"], Verbs:["create" "update"]} {APIGroups:["projectcontour.io"], Resources:["extensionservices/status"], Verbs:["create" "update"]} {APIGroups:["projectcontour.io"], Resources:["httpproxies/status"], Verbs:["create" "update"]}

With contour operator chart
we get the following error:
sample-contouropservice:Error: Error: clusterroles.rbac.authorization.k8s.io "contouropservice-sample-contouropservice-contour-operator-auth" is forbidden: user "system:serviceaccount:default:kubeplus-saas-provider" (groups=["system:serviceaccounts" "system:serviceaccounts:default" "system:authenticated"]) is attempting to grant RBAC permissions not currently held: {APIGroups:["authentication.k8s.io"], Resources:["tokenreviews"], Verbs:["create"]} {APIGroups:["authorization.k8s.io"], Resources:["subjectaccessreviews"], Verbs:["create"]}

With external-dns chart we get the following error:
Helmrelease: sample-externaldnsservice:Error: Error: clusterroles.rbac.authorization.k8s.io "externaldnsservice-sample-externaldnsservice-external-dns-sampl" is forbidden: user "system:serviceaccount:default:kubeplus-saas-provider" (groups=["system:serviceaccounts" "system:serviceaccounts:default" "system:authenticated"]) is attempting to grant RBAC permissions not currently held: {APIGroups:["zalando.org"], Resources:["routegroups/status"], Verbs:["patch" "update"]}

With grafana-operator
Helmrelease: sample-gfs:Error: Error: roles.rbac.authorization.k8s.io "grafanaopservice-sample-gfs-grafana-operator-leader-election" is forbidden: user "system:serviceaccount:default:kubeplus-saas-provider" (groups=["system:serviceaccounts" "system:serviceaccounts:default" "system:authenticated"]) is attempting to grant RBAC permissions not currently held: {APIGroups:["coordination.k8s.io"], Resources:["leases"], Verbs:["create" "update" "patch" "delete"]}

With kong chart
Helmrelease: sample-kg:kongservice-sample-kg Error: Error: clusterroles.rbac.authorization.k8s.io "kongservice-sample-kg" is forbidden: user "system:serviceaccount:default:kubeplus-saas-provider" (groups=["system:serviceaccounts" "system:serviceaccounts:default" "system:authenticated"]) is attempting to grant RBAC permissions not currently held: {APIGroups:[""], Resources:["endpoints/status"], Verbs:["patch" "update"]} {APIGroups:[""], Resources:["secrets/status"], Verbs:["patch" "update"]} {APIGroups:[""], Resources:["services/status"], Verbs:["patch" "update"]} {APIGroups:["configuration.konghq.com"], Resources:["kongclusterplugins/status"], Verbs:["patch" "update"]} {APIGroups:["configuration.konghq.com"], Resources:["kongconsumers/status"], Verbs:["patch" "update"]} {APIGroups:["configuration.konghq.com"], Resources:["kongingresses/status"], Verbs:["patch" "update"]} {APIGroups:["configuration.konghq.com"], Resources:["kongplugins/status"], Verbs:["patch" "update"]} {APIGroups:["configuration.konghq.com"], Resources:["tcpingresses/status"], Verbs:["patch" "update"]} {APIGroups:["configuration.konghq.com"], Resources:["udpingresses/status"], Verbs:["patch" "update"]} {APIGroups:["extensions"], Resources:["ingresses/status"], Verbs:["patch" "update"]} {APIGroups:["networking.k8s.io"], Resources:["ingresses/status"], Verbs:["patch" "update"]}

With metallb chart

Helmrelease: sample-metallbservice:Error: Error: clusterroles.rbac.authorization.k8s.io "metallbservice-sample-metallbservice:controller" is forbidden: user "system:serviceaccount:default:kubeplus-saas-provider" (groups=["system:serviceaccounts" "system:serviceaccounts:default" "system:authenticated"]) is attempting to grant RBAC permissions not currently held: {APIGroups:[""], Resources:["services/status"], Verbs:["update"]} {APIGroups:["admissionregistration.k8s.io"], Resources:["mutatingwebhookconfigurations"], Verbs:["patch"]} {APIGroups:["admissionregistration.k8s.io"], Resources:["validatingwebhookconfigurations"], Verbs:["create" "delete" "patch" "update"]} {APIGroups:["policy"], Resources:["podsecuritypolicies"], ResourceNames:["metallbservice-sample-metallbservice-controller"], Verbs:["use"]}

metrics-server chart
Helmrelease: sample-metricsserverservice:Error: Error: clusterroles.rbac.authorization.k8s.io "metricsserverservice-sample-metricsserverservice-metrics-server" is forbidden: user "system:serviceaccount:default:kubeplus-saas-provider" (groups=["system:serviceaccounts" "system:serviceaccounts:default" "system:authenticated"]) is attempting to grant RBAC permissions not currently held: {APIGroups:[""], Resources:["nodes/metrics"], Verbs:["create"]}

nginx-ingress-controller chart
Helmrelease: sample-nginxingressservice:Error: Error: clusterroles.rbac.authorization.k8s.io "nginxingressservice-sample-nginxingressservice-nginx-ingress-co" is forbidden: user "system:serviceaccount:default:kubeplus-saas-provider" (groups=["system:serviceaccounts" "system:serviceaccounts:default" "system:authenticated"]) is attempting to grant RBAC permissions not currently held: {APIGroups:["networking.k8s.io"], Resources:["ingresses/status"], Verbs:["update"]}

rabbitmq-cluster-operator
Helmrelease: sample-rabbitopservice:Error: Error: clusterroles.rbac.authorization.k8s.io "rabbitopservice-sample-rabbitopservice-rabbitmq-cluster-operato" is forbidden: user "system:serviceaccount:default:kubeplus-saas-provider" (groups=["system:serviceaccounts" "system:serviceaccounts:default" "system:authenticated"]) is attempting to grant RBAC permissions not currently held: {APIGroups:["rabbitmq.com"], Resources:["rabbitmqclusters"], Verbs:["create" "update"]} {APIGroups:["rabbitmq.com"], Resources:["rabbitmqclusters/finalizers"], Verbs:["update"]} {APIGroups:["rabbitmq.com"], Resources:["rabbitmqclusters/status"], Verbs:["update"]}

sealed-secrets
Helmrelease: sample-sealedservice:Error: Error: clusterroles.rbac.authorization.k8s.io "sealedservice-sample-sealedservice-sealed-secrets-sample-sealed-unsealer" is forbidden: user "system:serviceaccount:default:kubeplus-saas-provider" (groups=["system:serviceaccounts" "system:serviceaccounts:default" "system:authenticated"]) is attempting to grant RBAC permissions not currently held: {APIGroups:["bitnami.com"], Resources:["sealedsecrets/status"], Verbs:["update"]}

wavefront-hpa-adapter
Helmrelease: sample-whaservice:Error: Error: clusterrolebindings.rbac.authorization.k8s.io "whaservice-sample-whaservice-wavefront-hpa-adapter:system:auth-delegator" is forbidden: user "system:serviceaccount:default:kubeplus-saas-provider" (groups=["system:serviceaccounts" "system:serviceaccounts:default" "system:authenticated"]) is attempting to grant RBAC permissions not currently held: {APIGroups:["authentication.k8s.io"], Resources:["tokenreviews"], Verbs:["create"]} {APIGroups:["authorization.k8s.io"], Resources:["subjectaccessreviews"], Verbs:["create"]}

wavefront
Helmrelease: sample-wavefrontservice:Error: Error: clusterroles.rbac.authorization.k8s.io "wavefrontservice-sample-wavefrontservice-collector" is forbidden: user "system:serviceaccount:default:kubeplus-saas-provider" (groups=["system:serviceaccounts" "system:serviceaccounts:default" "system:authenticated"]) is attempting to grant RBAC permissions not currently held: {NonResourceURLs:["/metrics"], Verbs:["get"]}

[devdattakulkarni]

We need to address this issue at two levels.

First, we should detect the missing permissions at the time of CRD registration, instead of instance creation.
Second, we should provide a utility to update the provider kubeconfig permissions.