Release 18+ Upgrade Guide Breaks Existing Deployments

[jseiser]

Description

Attempted to follow the upgrade guide to get to 18+. Our Terraform deployments generally run from a Jenkins worker pod, that exists ON the same cluster that we are upgrading. The pod has a service account on it, using the IRSA setup which grants it access to the cluster.
This all works/worked before the upgrade.

Reproduction

Attempt to follow the upgrade guide for 18.

Code Snippet to Reproduce

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "18.0.1"

  cluster_name    = format("eks-%s-%s-%s", var.layer, var.vpc_id_tag, var.platform_env)
  cluster_version = var.cluster_version

  subnet_ids = data.aws_subnet_ids.private.ids
  vpc_id     = data.terraform_remote_state.vpc.outputs.vpc_id

  cluster_endpoint_private_access = true
  cluster_endpoint_public_access  = false

  cluster_security_group_additional_rules = {
    admin_access = {
      description = "Admin ingress to Kubernetes API"
      cidr_blocks = [data.terraform_remote_state.vpc.outputs.vpc_cidr_block]
      protocol    = "tcp"
      from_port   = 443
      to_port     = 443
      type        = "ingress"
    }
  }

  eks_managed_node_group_defaults = {
    ami_type                   = "AL2_x86_64"
    disk_size                  = var.node_group_default_disk_size
    enable_bootstrap_user_data = true
    pre_bootstrap_user_data    = templatefile("${path.module}/templates/userdata.tpl", {})
    desired_size               = lower(var.platform_env) == "prod" ? 3 : 2
    max_size                   = lower(var.platform_env) == "prod" ? 6 : 3
    min_size                   = lower(var.platform_env) == "prod" ? 3 : 1
    instance_types             = lower(var.platform_env) == "prod" ? var.prod_instance_types : var.dev_instance_types
    capacity_type              = "ON_DEMAND"
    additional_tags = {
      Name = format("%s-%s-%s", var.layer, var.vpc_id_tag, var.platform_env)
    }
    update_config = {
      max_unavailable_percentage = 50
    }
    update_launch_template_default_version = true
    create_launch_template                 = true
    create_iam_role                        = true
    iam_role_name                          = format("iam-%s-%s-%s", var.layer, var.vpc_id_tag, var.platform_env)
    iam_role_use_name_prefix               = false
    iam_role_description                   = "EKS managed node group Role"
    iam_role_tags                          = local.tags
    iam_role_additional_policies = [
      "arn:aws-us-gov:iam::aws:policy/AmazonSSMManagedInstanceCore"
    ]
  }
  eks_managed_node_groups = {
    private1 = {
      subnet_ids = [tolist(data.aws_subnet_ids.private.ids)[0]]
    }
    private2 = {
      subnet_ids = [tolist(data.aws_subnet_ids.private.ids)[1]]
    }
    private3 = {
      subnet_ids = [tolist(data.aws_subnet_ids.private.ids)[2]]
    }
  }

  cluster_enabled_log_types              = ["api", "audit", "authenticator", "controllerManager", "scheduler"]
  cloudwatch_log_group_retention_in_days = 7

  enable_irsa = true

  cluster_encryption_config = [
    {
      provider_key_arn = aws_kms_key.eks.arn
      resources        = ["secrets"]
    }
  ]

  tags = merge(
    local.tags,
    {
      "Name"        = format("eks-%s-%s-%s", var.layer, var.vpc_id_tag, var.platform_env),
      "EKS_VERSION" = var.cluster_version
    }
  )

}

resource "null_resource" "patch" {
  triggers = {
    kubeconfig = base64encode(local.kubeconfig)
    cmd_patch  = "kubectl patch configmap/aws-auth --patch \"${local.aws_auth_configmap_yaml}\" -n kube-system --kubeconfig <(echo $KUBECONFIG | base64 --decode)"
  }

  provisioner "local-exec" {
    interpreter = ["/bin/bash", "-c"]
    environment = {
      KUBECONFIG = self.triggers.kubeconfig
    }
    command = self.triggers.cmd_patch
  }
}

locals

locals {

  kubeconfig = yamlencode({
    apiVersion      = "v1"
    kind            = "Config"
    current-context = "terraform"
    clusters = [{
      name = module.eks.cluster_id
      cluster = {
        certificate-authority-data = module.eks.cluster_certificate_authority_data
        server                     = module.eks.cluster_endpoint
      }
    }]
    contexts = [{
      name = "terraform"
      context = {
        cluster = module.eks.cluster_id
        user    = "terraform"
      }
    }]
    users = [{
      name = "terraform"
      user = {
        token = data.aws_eks_cluster_auth.eks.token
      }
    }]
  })

  aws_auth_configmap_yaml = <<-EOT
  ${chomp(module.eks.aws_auth_configmap_yaml)}
      - rolearn: arn:${var.iam_partition}:iam::${data.aws_caller_identity.current.account_id}:role/role-gitlab-runner-eks-${var.platform_env}
        username: gitlab:{{SessionName}}
        groups:
          - system:masters
      - rolearn: arn:${var.iam_partition}:iam::${data.aws_caller_identity.current.account_id}:role/role-jenkins-worker-eks-${var.platform_env}
        username: jenkins:{{SessionName}}
        groups:
          - system:masters
      - rolearn: arn:${var.iam_partition}:iam::${data.aws_caller_identity.current.account_id}:role/AWSReservedSSO_AdministratorAccess_f50fcd43baf05a89
        username: AWSAdministratorAccess:{{SessionName}}
        groups:
          - system:masters
  EOT
}

Expected behavior

Module will run to completion

Actual behavior

Current aws-auth

sh-4.2$ kubectl get configmap aws-auth -n kube-system -o yaml
apiVersion: v1
data:
  mapAccounts: |
    []
  mapRoles: |
    - "groups":
      - "system:bootstrappers"
      - "system:nodes"
      "rolearn": "arn:aws-us-gov:iam:::role/eks-ops-eks-dev20211104211936784200000009"
      "username": "system:node:{{EC2PrivateDNSName}}"
    - "groups":
      - "system:masters"
      "rolearn": "arn:aws-us-gov:iam:::role/role-gitlab-runner-eks-dev"
      "username": "gitlab-runner-dev"
    - "groups":
      - "system:masters"
      "rolearn": "arn:aws-us-gov:iam:::role/role-jenkins-worker-eks-dev"
      "username": "jenkins-dev"
  mapUsers: |
    - "groups":
      - "system:masters"
      "userarn": "arn:aws-us-gov:iam:::user/justin.seiser"
      "username": "jseiser"
kind: ConfigMap

The SA on the pod, that terraform is running from.

sh-4.2$ kubectl get sa jenkins-worker -n jenkins -o yaml
apiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws-us-gov:iam:::role/role-jenkins-worker-eks-dev

The error terraform returns

module.eks.kubernetes_config_map.aws_auth[0]: Refreshing state... [id=kube-system/aws-auth]

Error: configmaps "aws-auth" is forbidden: User "system:serviceaccount:jenkins:jenkins-worker" cannot get resource "configmaps" in API group "" in the namespace "kube-system"

Additional context

I do not doubt that im missing something, but that something does not appear to be covered in the documentation that I can find.

[jseiser]

Doesnt work if I run from a bastion host, which is also granted access to the cluster.

Error

╷
│ Error: Get "http://localhost/api/v1/namespaces/kube-system/configmaps/aws-auth": dial tcp 127.0.0.1:80: connect: connection refused
│
│
╵

Same server showing access

sh-4.2$ kubectl get configmap aws-auth -n kube-system
NAME       DATA   AGE
aws-auth   3      62d

sh-4.2$ rm ~/.kube/config
sh-4.2$ terraform plan
module.eks.aws_iam_openid_connect_provider.oidc_provider[0]: Refreshing state... [id=arn:aws-us-gov:iam:::oidc-provider/oidc.eks.us-gov-west-1.amazonaws.com/id/]
╷
│ Error: Get "http://localhost/api/v1/namespaces/kube-system/configmaps/aws-auth": dial tcp 127.0.0.1:80: connect: connection refused
│
sh-4.2$ kubectl get pods -n jenkins
The connection to the server localhost:8080 was refused - did you specify the right host or port?
sh-4.2$ aws eks update-kubeconfig --name eks-ops-eks-dev
Added new context arn:aws-us-gov:eks:us-gov-west-1::cluster/eks-ops-eks-dev to /home/ssm-user/.kube/config
sh-4.2$ kubectl get pods -n jenkins
NAME        READY   STATUS    RESTARTS   AGE
jenkins-0   2/2     Running   0          23d

So it doesnt matter whether the server is configured to access the cluster or not, it fails the same way

[jseiser]

Ran the first few times with the Terraform kubernetes provider configured as it has been when it was working, and have re-ran it with that provider completely removed.

Ive also ran it with and without the null provider stuff.

[jseiser]

Are there any requirements to upgrade an existing cluster from 17 to 18?

Im not getting warnings 'variable not expected here' so I think I have everything renamed, but not making any progress on the aws-auth configmap. Does it need removed from the state?

[jseiser]

Also, not sure if this matters, but in the current working deployment we have

  kubeconfig_aws_authenticator_command = "aws"
  kubeconfig_aws_authenticator_command_args = [
    "eks",
    "get-token",
    "--cluster-name",
    format("eks-%s-%s-%s", var.layer, var.vpc_id_tag, var.platform_env),
    "--region",
    data.aws_region.current.name
  ]

[sparqueur]

Same Error: Get "http://localhost/api/v1/namespaces/kube-system/configmaps/aws-auth" error on my side. You are not alone man ;-)

[bryantbiggs]

I believe your error message states the issue The connection to the server localhost:8080 was refused - did you specify the right host or port?

By default, this port is not open on the security groups created by this module - https://github.com/terraform-aws-modules/terraform-aws-eks#security-groups

[sparqueur]

I believe your error message states the issue The connection to the server localhost:8080 was refused - did you specify the right host or port?

By default, this port is not open on the security groups created by this module - https://github.com/terraform-aws-modules/terraform-aws-eks#security-groups

It's more like if it's a wrong url error more than a security group issue, no ? I expect this call being done from my laptop, not from a remote server. But I may be wrong.

[bryantbiggs]

I believe your error message states the issue The connection to the server localhost:8080 was refused - did you specify the right host or port?
By default, this port is not open on the security groups created by this module - terraform-aws-modules/terraform-aws-eks#security-groups

It's more like if it's a wrong url error more than a security group issue, no ? I expect this call being done from my laptop, not from a remote server. But I may be wrong.

The error message I was referring to was the one provided by @jseiser further up.

The error message you have provided does not provide enough detail - the module does not construct any URLs so I would suspect its also a security group access issue you are facing as well, but thats just a hunch based on whats provided

[GeoBSI]

Hello, I'm having the exact same issue when upgrading from v17 to v18. It happens during the state refresh of the terraform plan command.

The problem disappears if I manually remove the module.eks-cluster.local_file.kubeconfig[0] and module.eks-cluster.kubernetes_config_map.aws_auth[0] from the v17 state file, but I'm not exactly sure if there are any consequences in doing this.

[jseiser]

I believe your error message states the issue The connection to the server localhost:8080 was refused - did you specify the right host or port?

By default, this port is not open on the security groups created by this module - https://github.com/terraform-aws-modules/terraform-aws-eks#security-groups

@bryantbiggs

I guess i dont follow. This is a cluster/TF deployment created using the <18.0 module version. We are wanting/trying to upgrade this enviroment to the latest module version. Im not able to run a plan because of the error. The security groups have not changed.

[bryantbiggs]

Hello, I'm having the exact same issue when upgrading from v17 to v18. It happens during the state refresh of the terraform plan command.

The problem disappears if I manually remove the module.eks-cluster.local_file.kubeconfig[0] and module.eks-cluster.kubernetes_config_map.aws_auth[0] from the v17 state file, but I'm not exactly sure if there are any consequences in doing this.

I believe that would be the appropriate change (remove those from your state) - v18.x removes native support for kubeconfig and aws-auth configmap https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/UPGRADE-18.0.md#list-of-backwards-incompatible-changes

[PadillaBraulio]

How should we handle config-map roles now?
i had a role we used to connect to the cluster but since v18.x drop the support to aws-auth configmap, how can we address that?

For example i had somethig like this

  mapRoles: |
.
.
.
    - "groups":
      - "system:masters"
      "rolearn": "arn:aws:iam::000000000000:role/assumedRole
      "username": "administrator"
and that role was set in the map_roles variable something like this

module "eks" {
  source                      = "terraform-aws-modules/eks/aws"
  version                     = "17.24.0"

  map_roles = [
    {
      rolearn = aws_iam_role.eks_administrator.arn
      username = "administrator"
      groups = [ "system:masters" ]
    }
  ]

}

Moving forward what would be the best way to adding that types of roles into the aws-auth?

[bryantbiggs]

@PadillaBraulio this is left up to users to decide what suites them best (Terraform, Helm, some flavor of GitOps, bash scripts, etc.)

[martijnvdp]

These errors also occur on certain changes when your the provider config depends on output of the eks module
i got the same when i used coalesce(module.eks.cluster_id, "cluster1") for name name filter in the datablock for the provider
after changing that to the static name the error is gone

example:

data "aws_eks_cluster" "cluster1" {
  // name= coalesce(module.cluster1.cluster_id, "cluster1")  will generate error on certain changes
  name  = "cluster1"// works
}

data "aws_eks_cluster_auth" "cluster1" {
  name  = "cluster1"
}

provider "kubernetes" {
  host                   = element(concat(data.aws_eks_cluster.cluster1.*.endpoint, [""]), 0)
  cluster_ca_certificate = base64decode(element(concat(data.aws_eks_cluster.cluster1.*.certificate_authority.0.data, [""]), 0))
  token                  = element(concat(data.aws_eks_cluster_auth.cluster1.*.token, [""]), 0)
}