data.azurerm_kubernetes_cluster do not get kube_config from AAD enabled v1.24 cluster

[torumakabe]

Is there an existing issue for this?

• I have searched the existing issues

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

1.2.2

AzureRM Provider Version

3.9.0

Affected Resource(s)/Data Source(s)

azurerm_kubernetes_cluster

Terraform Configuration Files

data "azurerm_kubernetes_cluster" "default" {
  depends_on          = [module.aks] # refresh cluster state before reading
  name                = module.aks.aks_cluster_name
  resource_group_name = module.aks.resource_group_name
}

provider "kubernetes" {
  host                   = data.azurerm_kubernetes_cluster.default.kube_config.0.host
  cluster_ca_certificate = base64decode(data.azurerm_kubernetes_cluster.default.kube_config.0.cluster_ca_certificate)

  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "kubelogin"
    args = [
      "get-token",
      "--login",
      "azurecli",
      "--server-id",
      "6dae42f8-4368-4678-94ff-3960e28e3630"
    ]
  }
}

Debug Output/Panic Output

data.azurerm_kubernetes_cluster.default: Read complete after 0s [id=/subscriptions/my-sub-id/resourceGroups/rg-aks-playground-blue-dev/providers/Microsoft.ContainerService/managedClusters/my-cluster]
╷
│ Error: Invalid index
│
│   on main.tf line 35, in provider "kubernetes":
│   35:   host                   = data.azurerm_kubernetes_cluster.default.kube_config.0.host
│     ├────────────────
│     │ data.azurerm_kubernetes_cluster.default.kube_config has a sensitive value
│
│ The given key does not identify an element in this collection value.
╵
╷
│ Error: Invalid index
│
│   on main.tf line 36, in provider "kubernetes":
│   36:   cluster_ca_certificate = base64decode(data.azurerm_kubernetes_cluster.default.kube_config.0.cluster_ca_certificate)
│     ├────────────────
│     │ data.azurerm_kubernetes_cluster.default.kube_config has a sensitive value
│
│ The given key does not identify an element in this collection value.
╵

Expected Behaviour

Get kube_config properly.

Actual Behaviour

I checked kube_config in tf state. It was empty.

"kube_config": [],

The cause is probably this week's release of AKS. Released kubernetes version 1.24 and changed kube_config format for AAD enabled cluster.

So, I confirmed the difference of kube_config between 1.23.5 and 1.24.0. Certainly, users.user has changed.

[1.23.5]

  user:
    auth-provider:
      config:
        apiserver-id: [my-server-id]
        client-id: [my-client-id]
        config-mode: "1"
        environment: AzurePublicCloud
        tenant-id: [my-tenant-id]
      name: azure

[1.24.0]

 user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - get-token
      - --environment
      - AzurePublicCloud
      - --server-id
      - [my-server-id]
      - --client-id
      - [my-client-id]
      - --tenant-id
      - [my-tenant-id]
      - --login
      - devicecode
      command: kubelogin
      env: null

In the current implementation, it seems that it is judged by "apiserver-id", so I think that kube_config cannot be obtained correctly.

Steps to Reproduce

terraform apply

Important Factoids

No response

References

No response

[DaleyKD]

Any update on this?

[jz-wilson]

I'm also having the same issue

[torumakabe]

AKS with 1.24 is now GA. Anyone triaging this issue?

[jozephp]

We also have been facing this issue in Central US now

[kkizer]

At MetLife are also experiencing a similar issue now that Kubernetes 1.24.0 and 1.24.3 have gone GA for AKS. Microsoft changed the cluster configuration format in 1.24 so that it requires the use of kubelogin so I am not surprised that this is broken. However, our clients are going to need to be on the latest GA version of Kubernetes so this has to be fixed ASAP.

[joberget]

We also get this problem with AKS 1.24.3.

[joberget]

@torumakabe Checked this?
"We provide an optional query parameter ‘format’ when getting clusterUser credential to overwrite the default behavior change, you can explicitly specify format to ‘azure’ to get old format kubeconfig."

Ref https://github.com/Azure/AKS/blob/master/CHANGELOG.md#release-2022-05-29

[torumakabe]

@joberget format optional query parameter is for Azure CLI, so I don't think the option will solve the problem we are discussing.

[torumakabe]

As a quick workaround, I decided not to use data source and use external data source temporarily.

// temporarily replace aks data source with external data source
// issue: https://github.com/hashicorp/terraform-provider-azurerm/issues/17182
/*
data "azurerm_kubernetes_cluster" "default" {
  depends_on          = [module.aks] # refresh cluster state before reading
  name                = module.aks.aks_cluster_name
  resource_group_name = module.aks.resource_group_name
}
*/

data "external" "cluster_info" {
  depends_on = [
    module.aks,
  ]
  program = ["/bin/bash", "-c", "${path.module}/scripts/get_clusterinfo.sh"]

  query = {
    rg_name      = local.aks.rg.name
    cluster_name = local.aks.cluster_name
  }
}

provider "kubernetes" {
  host                   = data.external.cluster_info.result["server"]
  cluster_ca_certificate = base64decode(data.external.cluster_info.result["cert_auth_data"])

  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "kubelogin"
    args = [
      "get-token",
      "--login",
      "azurecli",
      "--server-id",
      "6dae42f8-4368-4678-94ff-3960e28e3630"
    ]
  }
}

The external script is the following:

#!/bin/bash
set -eo pipefail

eval "$(jq -r '@sh "RG_NAME=\(.rg_name) CLUSTER_NAME=\(.cluster_name)"')"

az aks get-credentials -g "${RG_NAME}" -n "${CLUSTER_NAME}" --file ./tmpconfig

SERVER=$(yq '.clusters[0].cluster.server' ./tmpconfig)
CERT_AUTH_DATA=$(yq '.clusters[0].cluster.certificate-authority-data' ./tmpconfig)

rm ./tmpconfig

jq -n --arg server "${SERVER}" --arg cert_auth_data "${CERT_AUTH_DATA}" '{"server":$server, "cert_auth_data":$cert_auth_data}'

Not ideal as it depends on your environment and tools like yq, but good enough for short term solution.

The long-term solution should be to adapt this data source for new exec format such as adding a new condition and struct. It seems that the impact is somewhat large, so we need input from azurerm provider community leads on how to proceed.

[davidkarlsen]

We got screwed by this, as https://github.com/Azure/terraform-azurerm-aks eventually upgraded to the v3 provider, but that failed due to changed attribute model, and when we did state rm + import we end up in this state which effectively breaks as it will make tf recreate the resource if applied.

[kkizer]

The external data source option is...interesting. Thank you very much for sharing the details. But it is problematic for us. We are using Terraform Cloud and we have to have our own Terraform build agents in order to access resources on our private VNET in Azure (resources such as key vaults have private end points and Terraform configurations need to read values from the key vaults). We'd have to install the Azure CLI on each build agent as well as jq. For the Azure CLI, what credentials would the build agent use to login to Azure to use the az aks get-credentials command? It would have to be a managed identity or service principal that has the ability to download the cluster's configuration. All of our clusters have Azure Active Directory (AAD) integration enabled and Azure manages the RBAC. Local Kubernetes accounts are disabled in the cluster.

This approach also complicates using the Helm and Kubernetes providers. We are trying to change how we provision resources so that developers can do most everything themselves. Therefore we have to make it simple and provide a set of modules they can snap together (along with guardrails and bubble wrap). I suppose you could wrap the implementation into a module and make the module outputs be the "server" and "cert_auth_data" values.

We have a support ticket open with Hashicorp for this issue. It seems to be getting a lot of visibility since MetLife is doing a pilot test of Terraform. Hopefully they can come up with a good solution soon.

[DaleyKD]

@kkizer : Before 1.24, were you able to disable local accounts? If so, how did you authenticate when you used the helm and kubernetes providers?

I'm currently on 1.23.8, and seeing this is definitely making me NOT upgrade to 1.24, but I do want to be as close/prepared as possible for when the time comes.