Listener ServiceAccount does not allow to add Annotations

[jonny-rimek]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.13.0

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

1. declare helm chart and set annotations, example is in tf



resource "helm_release" "gha_runner_scale_set" {
  name      = local.runner_name
  namespace = local.runner_namespace

  chart   = "oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set"
  version = var.arc_version

  create_namespace  = true
  dependency_update = true

  values = [jsonencode({
    githubConfigUrl    = "https://github.com/ORG_NAME"
    githubConfigSecret = local.github_secret_name

    controllerServiceAccount = {
      namespace = local.listener_namespace
      name      = "arc-gha-rs-controller"
    }

    minRunners = 1
    maxRunners = 10

    # this is the annotation i want to add to the listener service account
    annotations = {
      "eks.amazonaws.com/role-arn" = aws_iam_role.listener_secrets_access.arn
    }

    listenerTemplate = {
      spec = {

        volumes = [{
          name = "secrets-store"
          csi = {
            driver   = "secrets-store.csi.k8s.io"
            readOnly = true
            volumeAttributes = {
              secretProviderClass = local.github_secret_name
            }
          }
        }]

        containers = [{
          name = "listener"
          volumeMounts = [{
            name      = "secrets-store"
            mountPath = "/mnt/secrets-store"
            readOnly  = true
          }]
        }]
      }
    }

    labels = { for k, v in local.tags : k => tostring(v) }
  })]

  depends_on = [
    kubernetes_manifest.github_app_secret_provider,
    helm_release.gha_runner_scale_set_controller
  ]
}


2. apply and look up info of the listener ServiceAccount in the controller namespace


Name:                ***-754b578d-listener
Namespace:           arc-systems
Labels:              actions.github.com/organization=***
                     actions.github.com/scale-set-name=***    
                     actions.github.com/scale-set-namespace=arc-runners
                     app.kubernetes.io/component=runner-scale-set-listener
                     app.kubernetes.io/instance=***
                     app.kubernetes.io/managed-by=Helm
                     app.kubernetes.io/name=***
                     app.kubernetes.io/part-of=gha-runner-scale-set
                     app.kubernetes.io/version=0.13.0
                     helm.sh/chart=gha-rs-0.13.0
Annotations:         <none>
Image pull secrets:  <none>
Mountable secrets:   <none>
Tokens:              <none>
Events:              <none>


looking at the listener pod definition we see that this is the ServiceAccount that is used


Name:             ***-754b578d-listener
Namespace:        arc-systems
Priority:         0
Service Account:  ***-754b578d-listener

[...]

Events:
  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    12m                  default-scheduler  Successfully assigned arc-systems/***-754b578d-listener to ip-10-170-16-192.eu-central-1.compute.internal
  Warning  FailedMount  119s (x13 over 12m)  kubelet            MountVolume.SetUp failed for volume "secrets-store" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod arc-systems/***-754b578d-listener, err: rpc error: code = Unknown desc = eu-central-1: An IAM role must be associated with service account ***-754b578d-listener (namespace: arc-systems)


In order to avoid confusion, the ServiceAccounts in the runner namespace to get the annotations ([code](https://github.com/actions/actions-runner-controller/blob/3c1a323381a7f5ba782fb1a66c5db04767f70c7b/charts/gha-runner-scale-set/templates/kube_mode_serviceaccount.yaml#L12))

Describe the bug

It's not possible to add annotations to the listener ServiceAccount.

We are using the secret-store csi driver to sync from aws secrets manager to kubernetes secrets, we store the github app id/installation id/pem inside the aws secret and want to sync it to kubernetes.

My first problem is that there is a circular dependency issue, the the controller only creates the listener if the secret exists, but the secret is created when the volume is mounted when the listener pod is started. I worked around this by manually creating a secret, as this is a one time action and only necessary during the initial setup this is acceptable for us.

The next step is to allow the ServiceAccount to assume the iam role, so it can read the secret and sync it to the k8s secret. In order to do this you add an annotation to the ServiceAccount.

I looked up the code for the ServiceAccount for the listener, which is here

actions-runner-controller/controllers/actions.github.com/resourcebuilder.go

Line 433 in e46c929

Labels: b.mergeLabels(autoscalingListener.Labels, map[string]string{

and we see that it is not possible to pass in annotations. As far as I can tell we would only need to add the following line Annotations: autoscalingListener.Annotations,

I want to note that I had it up and running before I tried to add the csi driver secret store, using a normal kubernetes secret, so the problems are all related to that.

Describe the expected behavior

Custom annotation are correctly passed into the listener service account.

Additional Context

I added the passed in values above in tf

Controller Logs

https://gist.github.com/jonny-rimek/49eee36949fa2023f8c5b46036376c22

Runner Pod Logs

runner doesn't start, because the listener doesn't start

[ego93]

Here is a workaround using Terraform.

Create a kubernetes_service_account resource

resource "kubernetes_service_account" "gha_runner_scale_set" {
  metadata {
    name           = "arc-gha-rs-controller"
    namespace = local.listener_namespace
    annotations = {
      "eks.amazonaws.com/role-arn" = aws_iam_role.listener_secrets_access.arn
    }
    labels = {
      "app.kubernetes.io/managed-by" = "Terraform"
    }
  }
}

Then in your helm values

listenerTemplate = {
  spec = {
    serviceAccountName = "arc-gha-rs-controller"
...

[Steve-Glass]

Chiming in from the product side. This is something we will consider for our next major release 0.14.0, which is expected to ship in the Jan-Apr 2026 timeframe. Our team needs to take a look at the multiple options that have been identified so far and determine the best path forward.

[andreaspodskalsky]

I have a similar problem wit annotations on "Role" and "RoleBinding" for listener resources.

[Insofan]

I have a similar problem wit annotations on "Role" and "RoleBinding" for listener resources.

Have to fix it？

[jonny-rimek]

@ego93 we tried implementing the fix you mentioned, but it turns out that listenerTemplate.spec.serviceAccountName is also not passed through. Using that we still created the default ServiceAccount, that we could not add Annotations to.

In the following two links we can see that serviceAccountName does not get merged in

actions-runner-controller/controllers/actions.github.com/resourcebuilder.go

Line 392 in e46c929

func mergeListenerContainer(base, from *corev1.Container) {

actions-runner-controller/controllers/actions.github.com/resourcebuilder.go

Line 314 in e46c929

func mergeListenerPodWithTemplate(pod *corev1.Pod, tmpl *corev1.PodTemplateSpec) {

for completeness’s sake, i'll share our current tf code again

resource "kubernetes_service_account" "gha_runner_scale_set" {
  metadata {
    name      = "arc-gha-rs-controller"
    namespace = local.listener_namespace
    annotations = {
      "meta.helm.sh/release-name"      = "arc",
      "meta.helm.sh/release-namespace" = "arc-systems"
    }
    labels = merge(
      { for k, v in local.tags : k => tostring(v) },
      {
        "app.kubernetes.io/managed-by" = "Helm",
        "app.kubernetes.io/component"  = "runner-scale-set-listener",
        "app.kubernetes.io/part-of"    = "gha-rs-controller",
      }
    )
  }
}

resource "kubernetes_service_account" "listener_service_account" {
  metadata {
    name      = "scacap-dev-small-listener"
    namespace = local.listener_namespace
    annotations = {
      "eks.amazonaws.com/role-arn"     = aws_iam_role.listener_secrets_access.arn
    }
    labels = merge(
      { for k, v in local.tags : k => tostring(v) },
      {
        "actions.github.com/organization"        = "ScaCap"
        "actions.github.com/scale-set-name"      = "scacap-dev-small"
        "actions.github.com/scale-set-namespace" = "arc-runners"
        "app.kubernetes.io/component"            = "runner-scale-set-listener"
        "app.kubernetes.io/instance"             = "scacap-dev-small"
        "app.kubernetes.io/managed-by"           = "Helm"
        "app.kubernetes.io/name"                 = "scacap-dev-small"
        "app.kubernetes.io/part-of"              = "gha-runner-scale-set"
        "app.kubernetes.io/version"              = "0.13.0"
        "helm.sh/chart"                          = "gha-rs-0.13.0"
        "sidecar.istio.io/inject"                = false
      }
    )
  }
}

resource "helm_release" "gha_runner_scale_set" {
  name      = local.runner_name
  namespace = local.runner_namespace

  chart   = "oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set"
  version = var.arc_version

  create_namespace  = true
  dependency_update = true

  values = [jsonencode({
    githubConfigUrl    = "https://github.com/ScaCap"
    githubConfigSecret = local.github_secret_name

    controllerServiceAccount = {
      namespace = local.listener_namespace
      name      = "arc-gha-rs-controller"
    }

    minRunners = 1
    maxRunners = 10

    listenerTemplate = {
      spec = {
        serviceAccountName = "scacap-dev-small-listener" # <- does NOT work

        volumes = [{
          name = "secrets-store"
          csi = {
            driver   = "secrets-store.csi.k8s.io"
            readOnly = true
            volumeAttributes = {
              secretProviderClass = local.github_secret_name
            }
          }
        }]

        containers = [{
          name = "listener"
          volumeMounts = [{
            name      = "secrets-store"
            mountPath = "/mnt/secrets-store"
            readOnly  = true
          }]
        }]
      }
    }

    labels = { for k, v in local.tags : k => tostring(v) }

    labels = merge(
      { for k, v in local.tags : k => tostring(v) },
      {
        "sidecar.istio.io/inject" = false
    })
  })]

  depends_on = [
    kubernetes_manifest.github_app_secret_provider,
    kubernetes_service_account.gha_runner_scale_set,
    kubernetes_service_account.listener_service_account,
    helm_release.gha_runner_scale_set_controller,
  ]
}

I tried manually adding the annotation by modifying the resource directly and everything started to work, for obvious reasons this is not a real solution for us.

Not being able to add annotations to the listener Service Account is blocking us from adopting the new version of ARC.