terraform apply and NoRegisteredProviderFound

[bandirom]

Is there an existing issue for this?

• I have searched the existing issues

Greenfield/Brownfield provisioning

brownfield

Terraform Version

Terraform v1.10.5 on windows_386

Module Version

9.4.1

AzureRM Provider Version

3.117.0

Affected Resource(s)/Data Source(s)

module.aks.cluazapi_update_resource.aks_cluster_post_create

Terraform Configuration Files

terraform {
  required_version = ">=1.4"

  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "3.117.0"
    }
    azuread = {
      source  = "hashicorp/azuread"
      version = "3.1.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "2.35.1"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "2.17.0"
    }
  }
}


locals {
  cluster_name         = "${var.env}-${var.k8s_cluster_name}"
  dns_prefix           = "${var.env}-${var.k8s_dns_prefix}"
  node_pool_name       = "${var.env}node"
  cluster_principal_id = lookup(module.aks.cluster_identity, "principal_id")
  resource_group_name  = module.resource_group.resource_group_name
  node_resource_group  = var.k8s_node_resource_group
  kubernetes_version   = var.k8s_automatic_channel_upgrade == null ? var.k8s_version : null

  rbac_admin_group_object_ids = var.rbac_aad_enable ? var.full_access_group_id : null

  kubelogin = {
    api_version = "client.authentication.k8s.io/v1"
    command     = "kubelogin"
    args = [
      "get-token",
      "--login",
      "spn",
      "--environment",
      "AzurePublicCloud",
      "--tenant-id",
      var.tenant_id,
      "--server-id",
      module.aks_aad[0].aad_server_id,
      "--client-id",
      module.aks_aad[0].aad_spa_client_id,
      "--client-secret",
      module.aks_aad[0].aad_spa_client_secret,
    ]
  }
}

# contain only azurerm_resource_group.rg resource. Nothing special
module "resource_group" {
  source = "../resource_group"

  env      = var.env
  location = var.location
}

module "aks" {
  source  = "Azure/aks/azurerm"
  version = "9.4.1"

  cluster_name        = local.cluster_name
  location            = var.location
  resource_group_name = local.resource_group_name
  enable_auto_scaling = false
  kubernetes_version  = local.kubernetes_version
  prefix              = var.k8s_dns_prefix
  os_disk_size_gb     = var.k8s_os_disk_size
  node_resource_group = local.node_resource_group

  temporary_name_for_rotation = "noderotation"

  sku_tier                        = var.k8s_sku_tier
  identity_type                   = var.identity_type
  log_analytics_workspace_enabled = false
  automatic_channel_upgrade       = var.k8s_automatic_channel_upgrade

  agents_count = var.k8s_node_count
  agents_size  = var.k8s_vm_size

  network_plugin       = var.k8s_network_plugin
  network_policy       = var.k8s_network_policy
  net_profile_pod_cidr = var.k8s_net_profile_pod_cidr
  load_balancer_sku    = var.k8s_load_balancer_sku


  rbac_aad                          = var.rbac_aad_enable
  role_based_access_control_enabled = var.rbac_aad_enable
  rbac_aad_managed                  = var.rbac_aad_enable
  rbac_aad_admin_group_object_ids   = local.rbac_admin_group_object_ids
  rbac_aad_azure_rbac_enabled       = var.rbac_aad_enable
  local_account_disabled            = var.rbac_aad_enable

  key_vault_secrets_provider_enabled = var.key_vault_secrets_provider_enabled
  secret_rotation_interval           = "5m"

  tags = {
    env = var.env
  }
  depends_on = [module.resource_group]
}

tfvars variables values

k8s_node_count                = 5
k8s_automatic_channel_upgrade = "stable"
enable_read_only_access       = true
location = "switzerlandnorth"
k8s_sku_tier = "Free"
k8s_load_balancer_sku = "standard"
identity_type = "SystemAssigned"

Debug Output/Panic Output

When I run `terraform plan` I get


Planning failed. Terraform encountered an error while generating this plan.

Error: Failed to retrieve resource
│
│   with module.cluster.module.aks.azapi_update_resource.aks_cluster_post_create,
│   on .terraform\modules\cluster.aks\main.tf line 658, in resource "azapi_update_resource" "aks_cluster_post_create":
│  658: resource "azapi_update_resource" "aks_cluster_post_create" {
│
│ reading "Resource: (ResourceId \"/subscriptions/<subsription_id>/resourceGroups/dev-resources/providers/Microsoft.ContainerService/managedClusters/<cluster_name>\" / Api Version \"2023-01-02-preview\")":  
│ GET https://management.azure.com/subscriptions/<subsription_id>/resourceGroups/dev-resources/providers/Microsoft.ContainerService/managedClusters/<cluster_name>
│ --------------------------------------------------------------------------------
│ RESPONSE 400: 400 Bad Request
│ ERROR CODE: NoRegisteredProviderFound
│ --------------------------------------------------------------------------------
│ {
│   "error": {
│     "code": "NoRegisteredProviderFound",
│     "message": "No registered resource provider found for location 'switzerlandnorth' and API version '2023-01-02-preview' for type 'managedClusters'. The supported api-versions are '2017-08-31, 2018-03-31, 2019-02-01, 2019-04
-01, 2019-06-01, 2019-08-01, 2019-10-01, 2019-11-01, 2020-01-01, 2020-02-01, 2020-03-01, 2020-04-01, 2020-06-01, 2020-07-01, 2020-09-01, 2020-11-01, 2020-12-01, 2021-02-01, 2021-03-01, 2021-05-01, 2021-07-01, 2021-08-01, 2021-09
-01, 2021-10-01, 2022-01-01, 2022-02-01, 2022-03-01, 2022-04-01, 2022-06-01, 2022-07-01, 2022-08-01, 2022-09-01, 2022-11-01, 2023-01-01, 2023-02-01, 2023-03-01, 2023-04-01, 2023-04-02-preview, 2023-05-01, 2023-05-02-preview, 202
3-06-01, 2023-06-02-preview, 2023-07-01, 2023-07-02-preview, 2023-08-01, 2023-08-02-preview, 2023-09-01, 2023-09-02-preview, 2023-10-01, 2023-10-02-preview, 2023-11-01, 2023-11-02-preview, 2024-01-01, 2024-01-02-preview, 2024-02
-01, 2024-02-02-preview, 2024-03-02-preview, 2024-04-02-preview, 2024-05-01, 2024-05-02-preview, 2024-06-01, 2024-06-02-preview, 2024-07-01, 2024-07-02-preview, 2024-08-01, 2024-08-02-preview, 2024-09-01, 2024-09-02-preview, 202
4-10-01, 2024-10-02-preview'. The supported locations are 'australiacentral, australiacentral2, australiaeast, australiasoutheast, brazilsouth, brazilsoutheast, canadacentral, canadaeast, centralindia, centralus, eastasia, eastu
s, eastus2, francecentral, francesouth, germanynorth, germanywestcentral, israelcentral, italynorth, japaneast, japanwest, jioindiacentral, jioindiawest, koreacentral, koreasouth, mexicocentral, newzealandnorth, northcentralus, 
northeurope, norwayeast, norwaywest, polandcentral, qatarcentral, southafricanorth, southafricawest, southcentralus, southindia, southeastasia, spaincentral, swedencentral, switzerlandnorth, switzerlandwest, uaecentral, uaenorth, uksouth, ukwest, westcentralus, westeurope, westus, westus2, westus3'."
│   }
│ }


My folder structure like this: 

   │main
   │   ├── module.cluster (custom module)
   │       ├── Azure/aks/azurerm module

Expected Behaviour

I also found this bug in the azure cli
Azure/azure-cli#23985 (comment)

My az cli version is

{
  "azure-cli": "2.69.0",
  "azure-cli-core": "2.69.0",
  "azure-cli-telemetry": "1.1.0",
  "extensions": {}
}

Actual Behaviour

~ 1 month ago everything worked without any problems

I met this problem with Azure/aks/azurerm: v9.1.0 after that I decided to update the module version, azurerm, but it didnt resolve the issue

az provider show -n Microsoft.ContainerService -o table

Namespace                   RegistrationPolicy    RegistrationState
--------------------------  --------------------  -------------------
Microsoft.ContainerService  RegistrationRequired  Registered

Steps to Reproduce

No response

Important Factoids

No response

References

Azure/azure-cli#23985 (comment)

[lonegunmanb]

Hi @bandirom , thanks for opening this issue to us! This one seems like a Terraform AzureRM provider issue, but I would try to answer this one. According to the document:

The User, Service Principal or Managed Identity running Terraform should have permissions to register Azure Resource Providers. If the principal running Terraform has insufficient permissions to register Resource Providers then we recommend setting the property resource_provider_registrations to none in the provider block to prevent auto-registration.

Would you please check whether the account you're using now have the right permissions? Or maybe you can set resource_provider_registrations to none?

[bandirom]

Thank you for your reply, @lonegunmanb!

provider "azurerm" {
  resource_provider_registrations = "none"
  features {}
}

│ Error: Unsupported argument
│ 
│   on ..\modules\cluster\main.tf line 60, in provider "azurerm":
│   60:   resource_provider_registrations = "none"
│
│ An argument named "resource_provider_registrations" is not expected here.

According this example: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs#example-usage

As well for the resource_providers_to_register = "none" :

An argument named "resource_providers_to_register" is not expected here

UPD: looks like for v3.117I need to use skip_provider_registration

Looks like its deprecated (expected) but still does not work :(

As I logged in with my AZ user I can register/unregister

---

I want to mention that this bug appears suddenly. ~3-4 weeks ago I was working without any stuff updates (az cli, terraform, providers) and it worked. Yesterday, when I just started to work, it already broken

[zioproto]

Did you double check if you are pointing Terraform to the correct subscription ?

Please double check how you are authenticating Terraform with Azure.

https://registry.terraform.io/providers/Azure/azapi/latest/docs#authenticating-to-azure

The problem is with aks_cluster_post_create so it seems there is an issue with the azapi provider.

[ppetrovgh]

I second this. We are seeing the same error on our side.

[ramilano]

Maybe this is related: hashicorp/terraform-provider-azurerm#28838
it seems there will be 3.117.1 release of the provider

[kandreasyan]

You should have triggered an update of the resource before 2023-01-02-preview was deprecated, but this is not logical, the provider should handle this. Now, Terraform still tries to use the old API, which no longer exists. Taint and replace won’t help because the provider keeps referencing the deprecated version.
To fix this, either manually update the API version in the state file or remove the resource from the state and re-import it.

[bandirom]

@ramilano
v3.117.1 released, but still not fixed. Looks like it solved the problem in your mentioned link

[ppetrovgh]

@ramilano v3.117.1 released, but still not fixed. Looks like it solved the problem in your mentioned link

Same on our side, not fixed

[ppetrovgh]

You should have triggered an update of the resource before 2023-01-02-preview was deprecated, but this is not logical, the provider should handle this. Now, Terraform still tries to use the old API, which no longer exists. Taint and replace won’t help because the provider keeps referencing the deprecated version. To fix this, either manually update the API version in the state file or remove the resource from the state and re-import it.

manually updating the state file to 2024-10-01 worked.

[bandirom]

manually updating tfstate fixed for me as well

[dk-study9900]

can someone provide an example of the TFSTATE file change?

[zioproto]

@ms-henglu @lonegunmanb is this a confirmed provider problem related to deprecations listed here ?

Related to #28838