Changing SKU Tier from paid to standard for azurerm_kubernetes_cluster fails

[Mahagon]

Is there an existing issue for this?

• I have searched the existing issues

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

1.4.0

AzureRM Provider Version

3.47.0

Affected Resource(s)/Data Source(s)

azurerm_kubernetes_cluster

Terraform Configuration Files

resource "azurerm_kubernetes_cluster" "k8s" {
  name                              = "skutest-saas-${var.environment}"
  location                          = data.azurerm_resource_group.saas.location
  resource_group_name               = data.azurerm_resource_group.saas.name
  dns_prefix                        = "k8s"
  kubernetes_version                = "1.24"
  role_based_access_control_enabled = true
  automatic_channel_upgrade         = "patch"
  sku_tier                          = "Standard"
  api_server_access_profile {
    authorized_ip_ranges = concat(var.api_server_authorized_ip_ranges, ["${data.azurerm_public_ip.loadbalancer-saas-buildagent.ip_address}/32"])
  }

  identity {
    type = "SystemAssigned"
  }
  default_node_pool {
    name                         = "system"
    enable_auto_scaling          = true
    min_count                    = var.min_system_agent_count
    max_count                    = var.max_system_agent_count
    vm_size                      = "Standard_E4ds_v5"
    os_disk_type                 = "Ephemeral"
    max_pods                     = 110
    only_critical_addons_enabled = true
    vnet_subnet_id               = azurerm_subnet.k8s-subnet.id
  }

  network_profile {
    load_balancer_sku  = "standard"
    network_plugin     = "azure"
    network_policy     = "calico"
    service_cidr       = "10.140.0.0/14"
    docker_bridge_cidr = "172.17.0.1/16"
    dns_service_ip     = "10.140.0.101"
  }

  maintenance_window {
    allowed {
      day   = "Monday"
      hours = [0, 23]
    }
    allowed {
      day   = "Tuesday"
      hours = [0, 23]
    }
    allowed {
      day   = "Wednesday"
      hours = [0, 23]
    }
    allowed {
      day   = "Thursday"
      hours = [0, 23]
    }
    allowed {
      day   = "Friday"
      hours = [0, 23]
    }
    allowed {
      day   = "Saturday"
      hours = [0, 23]
    }
    allowed {
      day   = "Sunday"
      hours = [0, 23]
    }
  }

  lifecycle {
    prevent_destroy = true
  }

  tags = {
  }
}

Debug Output/Panic Output

╷
│ Error: updating Managed Cluster (Subscription: "SUBSCRIPTIONID"
│ Resource Group Name: "RESOURCEGROUPNAME"
│ Managed Cluster Name: "CLUSTERNAME"): managedclusters.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="SkuNotAvailable" Message="Standard managed cluster SKU tier is invalid. 'Paid' has been replaced by 'Standard' since v2023-02-01. "
│ 
│   with azurerm_kubernetes_cluster.k8s,
│   on k8s.tf line 1, in resource "azurerm_kubernetes_cluster" "k8s":
│    1: resource "azurerm_kubernetes_cluster" "k8s" {
│ 
╵

Expected Behaviour

Changing the value in the terraformstate fromPaid to Standard

Actual Behaviour

Azure returns an error message, that Paid has been replaced by Standard

Steps to Reproduce

1. terraform apply with sku_tier = Paid
2. terraform apply with sku_tier = Standard

Important Factoids

No response

References

#20734 sku_tier standard was introduced in azurerm v3.46.0

[stevehipwell]

@Mahagon I saw a similar error when testing v3.46.0 straight after release and I suspect that it might due to the way the Azure API/machinery is deployed. If you wait for v2023-02-26 to have been released in your region and try again does it work?

[Mahagon]

Not sure if i read it wrong, but i guess "Currently in Operation" means it's released right?

Our AKS is deployed in "Germany West Central" and the "Currently in Operation" version is v20230226.

Still getting the same error

[stevehipwell]

Not sure if i read it wrong, but i guess "Currently in Operation" means it's released right?

@Mahagon it should do.

I've not tried this since the version rolled out in our regions but based on it still being broken for you on the latest version I think there might be a provider bug.

[mcscwizzy]

Ran into this issue today as well, so it is definitely a provider problem.

[stevehipwell]

I spoke to some of the AKS team today and they confirmed that the API version would be globally available before it was published and usable, I was confusing the AKS release versions which are also date based. So this is a provider issue.

[cippaciong]

Hello, I just received an email from Azure with respect to the changes to AKS SKU name and tiers (relevant snippet below).

I'm just doing a wild assumption here, but could it be that the error is caused by the fact that the provider is hardcoding the Basic SKUName instead of Base?

Provider: internal/services/containers/kubernetes_cluster_resource.go#L1433

terraform-provider-azurerm/internal/services/containers/kubernetes_cluster_resource.go

Line 1433 in 56bf26c

Name: utils.ToPtr(managedclusters.ManagedClusterSKUNameBasic), // the only possible value at this point

)
SDK: resource-manager/containerservice/2023-01-02-preview/managedclusters/constants.go#L536

Azure email

ManagedClusterSKU parameter changes in Azure API versions

	2023-01-01 API or earlier	Starting with 2023-02-01 API
ManagedClusterSKUName	"Basic"	"Base"
ManagedClusterSKUTier	"Free" "Paid"	"Free" "Standard"

Recommended action

To avoid disrupting your AKS workloads, update any automated scripts, CD pipelines, ARM templates, Terraform, or other third-party tooling that relies on the
ManagedClusterSKUName and ManagedClusterSKUTier parameters that use "Basic" or "Paid" to use "Base" or "Standard" when you adopt Azure API version 2023-02-01 or newer.

[george-zubrienko]

The easiest fix for this would be if the provider can update this field in a state on the next plan, instead of requesting a cluster SKU upgrade (imo).

[zioproto]

@lonegunmanb @jiaweitao001 please be aware of this issue. An email has been sent to AKS customers today about this issue.