Setting cluster_endpoint_public_access_cidrs a value makes the nodegroup not join cluster

[abhilashkar]

Description

While creating a EKS cluster if I were to give addresses in cluster_endpoint_public_access_cidrs , the nodegroup isnt able to join the cluster. Cluster creation reports the following error

module.eks.module.eks_managed_node_group["default_node_group"].aws_eks_node_group.this[0]: Still creating... [26m41s elapsed]
╷
│ Error: error waiting for EKS Node Group (abc- default_node_group-20220210235029685300000001) to create: unexpected state 'CREATE_FAILED', wanted target 'ACTIVE'. last error: 1 error occurred:
│ * i-053f38656f5925c75: NodeCreationFailure: Instances failed to join the kubernetes cluster

If I were to change the cluster_endpoint_public_access_cidrs ( from 0.0.0.0/0 to any other ip) after the cluster creation it works fine

⚠️ Note

Before you submit an issue, please perform the following first:

1. Remove the local .terraform directory (! ONLY if state is stored remotely, which hopefully you are following that best practice!): rm -rf .terraform/
2. Re-initialize the project root to pull down modules: terraform init
3. Re-attempt your terraform plan or apply and check if the issue still persists

Versions

• Terraform: 1.0.11

• Provider(s):

• Module: eks

Reproduction

Steps to reproduce the behavior:
Create a cluster using the managed nodegroup example with a value set for cluster_endpoint_public_access_cidrs

Code Snippet to Reproduce

module "eks" {
#source = "../.."
source = "terraform-aws-modules/eks/aws"

cluster_name = local.name
cluster_version = local.cluster_version
cluster_endpoint_private_access = true
cluster_endpoint_public_access = true
cluster_endpoint_public_access_cidrs = var.cluster_endpoint_public_access_cidrs

Expected behavior

Cluster creation should be successful and nodegroups should join the cluster

Actual behavior

I could see the ip addresses in Public access source allowlist of the cluster but I dont see the nodegroups under that as the terraform errors out stating NodeCreationFailure: Instances failed to join the kubernetes cluster

Terminal Output Screenshot(s)

Error: error waiting for EKS Node Group (eks:default_node_group-20220210235029685300000001) to create: unexpected state 'CREATE_FAILED', wanted target 'ACTIVE'. last error: 1 error occurred:
│ * i-053f38656f5925c75: NodeCreationFailure: Instances failed to join the kubernetes cluster

Additional context

My requirement is to have a Nodegroup created in a private subnet ( SDWAN connected) and have them talk to the EKS cluster which has private and public endpoint. In the public endpoint I want to restrict the IP addresses which can connect to it.

[thamjieying]

I also face the same issue

[bryantbiggs]

can you provide a full reproduction please

[abhilashkar]

@bryantbiggs I am trying to create a EKS cluster with managed node group , All i do is supply a list of external ips
cluster_endpoint_public_access_cidrs = var.cluster_endpoint_public_access_cidrs
if this value is 0.0.0.0/0 , terraform succeeds. If I specify a list of ip addresses in it , the terraform fails with the reason

Error: error waiting for EKS Node Group (abc- default_node_group-20220210235029685300000001) to create: unexpected state 'CREATE_FAILED', wanted target 'ACTIVE'. last error: 1 error occurred:

[abhilashkar]

module "eks" {
#source = "../.."
source = "terraform-aws-modules/eks/aws"

cluster_name = local.name
cluster_version = local.cluster_version
cluster_endpoint_private_access = true
cluster_endpoint_public_access = true
cluster_endpoint_public_access_cidrs = var.cluster_endpoint_public_access_cidrs

[abhilashkar]

If I comment cluster_endpoint_public_access_cidrs or give a value of 0.0.0.0/0 the code succeeds

[abhilashkar]

Im using the example under examples/eks_managed_node_group

[thedusansky]

I have the same issue here. Basically when cluster_endpoint_public_access_cidrs is limited to some CIDRs, the node groups can't join the cluster and timed out.

cluster_endpoint_private_access = true
cluster_endpoint_public_access = true

Terraform v0.13.7
aws provider 2.64.0

[Zorgji]

Mine is working with provider version 4.2.

[dejwsz]

Maybe this may help? #1889 (comment)

[tculp]

@bryantbiggs I get this same issue, here is a pretty minimal example showing the issue:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.27"
    }
  }
  required_version = ">= 0.14.9"
}

provider "aws" {
  # Must match the profile name in your ~/.okta_aws_login_config file
  profile = "<profile>"
  region  = "us-east-1"

}

locals {
  name            = "test-1"
  cluster_version = "1.20"
  region          = "us-east-1"
}

data "aws_caller_identity" "current" {}

################################################################################
# EKS Module
################################################################################

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "18.8.1"

  cluster_name                    = local.name
  cluster_version                 = local.cluster_version
  cluster_endpoint_private_access = true
  cluster_endpoint_public_access  = true

  cluster_endpoint_public_access_cidrs = [<cidrs redacted>]

  vpc_id     = module.vpc.vpc_id
  subnet_ids = concat(
    module.vpc.private_subnets,
    module.vpc.public_subnets,
  )


  eks_managed_node_group_defaults = {
    disk_size      = 50
    instance_types = ["m5.large"]
    # iam_role_attach_cni_policy = true
  }

  eks_managed_node_groups = {
    # Default node group - as provided by AWS EKS
    default_node_group = {

    }
  }

}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "3.12.0"

  name = local.name
  cidr = "10.0.0.0/16"

  azs             = ["us-east-1a", "us-east-1b"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24"]

  enable_nat_gateway     = true
  single_nat_gateway     = true
  one_nat_gateway_per_az = false
}

resource "tls_private_key" "this" {
  algorithm = "RSA"
}

resource "aws_key_pair" "this" {
  key_name_prefix = local.name
  public_key      = tls_private_key.this.public_key_openssh

}

[tculp]

@bryantbiggs With regards to the example above, setting the public_access_cidrs to

cluster_endpoint_public_access_cidrs = concat(
    [<redacted cidrs>],
    ["${module.vpc.nat_public_ips[0]}/32"],
  )

makes it work.

EDIT:

Alternatively, it seems that adding the following to the VPC config sometimes makes it work too, but this seems to be inconsistent in how long the nodegroup takes to connect, varying between ~2m and ~9m

  enable_dns_support   = true
  enable_dns_hostnames = true

[onlinebaba]

@tculp Would you please help clarifying the reference to NAT public IPs?

["${module.vpc.nat_public_ips[0]}/32"],

[tchristie-meazure]

Adding the NAT addresses of the VPC the cluster is in does solve the issue.

I have enable_dns_support and enable_dns_hostnames but neither solved the problem. I also don't have a private endpoint exposed - only a public one.