fix: enforce control plane upgrade before node pool upgrades (fixes #465)

[lonegunmanb]

Problem

Fixes #465

When upgrading AKS cluster Kubernetes versions (e.g., from 1.27 to 1.28), users encounter NodePoolMcVersionIncompatible error:

Error: updating Default Node Pool Agent Pool ...
Code="NodePoolMcVersionIncompatible"
Message="Node pool version 1.27.3 and control plane version 1.26.6 are incompatible.
Minor version of node pool version 27 is bigger than control plane version 26."

AKS requires the control plane to be upgraded before node pools. The module's current code violates this ordering, causing the node pool upgrade to race ahead of the control plane upgrade.

Multiple users independently confirmed this issue (see #465 comments by @dunefro, @zioproto, @nnstt1, and others).

Root Cause

PR #336 refactored control plane upgrades to use azapi_update_resource.aks_cluster_post_create and added kubernetes_version to ignore_changes on azurerm_kubernetes_cluster.main. However, it did not add default_node_pool[0].orchestrator_version to ignore_changes.

This creates a DAG race condition:

1. azurerm_kubernetes_cluster.main sees a diff on default_node_pool[0].orchestrator_version and attempts to update the default node pool version directly via the azurerm provider
2. The azurerm provider sends the update request to Azure before aks_cluster_post_create upgrades the control plane
3. Azure rejects the request because node pool version > control plane version
4. aks_cluster_post_create (which would have upgraded the control plane first) never gets a chance to run because it depends on azurerm_kubernetes_cluster.main completing successfully

Extra node pools have the same issue: although they declare depends_on = [azapi_update_resource.aks_cluster_post_create], without ignore_changes = [orchestrator_version], Terraform still computes a diff and sends a version update via the azurerm provider. The depends_on only controls execution ordering — it does not prevent the azurerm provider from issuing its own update. As module maintainer @zioproto confirmed: the fix must cover both azurerm_kubernetes_cluster (default node pool) and azurerm_kubernetes_cluster_node_pool (extra node pools).

Key code references:

• main.tf:602-608 — existing lifecycle.ignore_changes (missing default_node_pool[0].orchestrator_version)
• main.tf:749-766 — aks_cluster_post_create (control plane upgrade via azapi, introduced by PR [Breaking] - Ignore changes on kubernetes_version from outside of Terraform #336)
• extra_node_pool.tf:148-152 — node_pool_create_before_destroy lifecycle (missing orchestrator_version)
• extra_node_pool.tf:320 — node_pool_create_after_destroy lifecycle (no ignore_changes at all)

Solution

Three-tier approach following the established pattern from PR #336:

1. Blind the azurerm provider (ignore_changes)

Prevent the azurerm provider from computing diffs on orchestrator_version for all node pool resources:

• Default node pool: Add default_node_pool[0].orchestrator_version to ignore_changes in azurerm_kubernetes_cluster.main
• Extra node pools (create_before_destroy): Add orchestrator_version to existing ignore_changes
• Extra node pools (create_after_destroy): Add new ignore_changes block with orchestrator_version

2. Build external schedulers (azapi_update_resource)

Delegate node pool version upgrades to azapi_update_resource resources that we can explicitly order:

• aks_cluster_default_nodepool_version — Upgrades the default node pool via Azure REST API (PUT .../agentPools/{pool_name})

  • resource_id follows the same pattern as existing aks_cluster_agents_pool_local_dns_config (main.tf:861): "${azurerm_kubernetes_cluster.main.id}/agentPools/${var.agents_pool_name}"
  • API version uses local.aks_api_version (2025-09-01), consistent with all other azapi resources in the module
  • Conditionally created only when var.orchestrator_version != null

• node_pool_version — Upgrades extra node pools via for_each

  • resource_id follows the same try() pattern as existing aks_cluster_local_dns_config (main.tf:807-810): try(node_pool_create_before_destroy[each.key].id, node_pool_create_after_destroy[each.key].id)
  • Handles both create-before-destroy and create-after-destroy node pool strategies

3. Enforce ordering (depends_on)

Both new azapi_update_resource resources declare depends_on = [azapi_update_resource.aks_cluster_post_create], ensuring the control plane upgrade completes before any node pool upgrade begins.

Version change tracking uses null_resource + replace_triggered_by (same pattern as existing kubernetes_version_keeper):

• orchestrator_version_keeper — tracks var.orchestrator_version for the default node pool
• node_pool_orchestrator_version_keeper — tracks each.value.orchestrator_version for extra pools

DAG Execution Order

azurerm_kubernetes_cluster.main
  (ignore_changes: kubernetes_version, default_node_pool[0].orchestrator_version)
    |
    v
null_resource.kubernetes_version_keeper ─────────────────┐
null_resource.orchestrator_version_keeper ───────────┐    │
    │                                                │    │
    v                                                │    │
time_sleep.interval_before_cluster_update            │    │
    │                                                │    │
    v                                                │    │
azapi_update_resource.aks_cluster_post_create  ◄─────┘────┘
  (upgrades CONTROL PLANE to var.kubernetes_version)
    │
    ├──────────────────────────────────┐
    │                                  │
    v                                  v
azapi_update_resource.               azapi_update_resource.
aks_cluster_default_                 node_pool_version
nodepool_version                     (for_each extra pools)
  (upgrades DEFAULT node pool)         (upgrades EXTRA node pools)
  (replace_triggered_by:               (replace_triggered_by:
   orchestrator_version_keeper)         node_pool_orchestrator_version_keeper)

Changes (2 files, +64/-1 lines)

main.tf

Change	Location	Description
Add ignore_changes	L606	Add default_node_pool[0].orchestrator_version to azurerm_kubernetes_cluster.main lifecycle
Add orchestrator_version_keeper	L727-731	New null_resource to track var.orchestrator_version changes (triggers replace_triggered_by)
Add aks_cluster_default_nodepool_version	L768-785	New azapi_update_resource to upgrade default node pool version via Azure REST API, gated by depends_on = [aks_cluster_post_create]

extra_node_pool.tf

Change	Location	Description
Add ignore_changes	L152	Add orchestrator_version to node_pool_create_before_destroy lifecycle
Add ignore_changes	L321-323	Add orchestrator_version to node_pool_create_after_destroy lifecycle
Add node_pool_orchestrator_version_keeper	L371-377	New null_resource with for_each to track per-pool orchestrator_version changes
Add node_pool_version	L379-401	New azapi_update_resource with for_each to upgrade extra node pool versions, gated by depends_on = [aks_cluster_post_create]

Verification

Sandbox experiment performed: AKS cluster upgraded from 1.32 → 1.33 with 1 default node pool + 1 extra node pool.

Execution log (confirms correct ordering):

# Step 1: Control plane upgrade starts FIRST
azapi_update_resource.aks_cluster_post_create: Modifying...   [0s elapsed]
azapi_update_resource.aks_cluster_post_create: Still modifying... [3m0s elapsed]
azapi_update_resource.aks_cluster_post_create: Modifications complete after 3m1s

# Step 2: Node pool upgrades start ONLY AFTER control plane completes
azapi_update_resource.aks_cluster_default_nodepool_version[0]: Modifying...
azapi_update_resource.node_pool_version["extra"]: Modifying...
azapi_update_resource.node_pool_version["extra"]: Modifications complete after 7m37s
azapi_update_resource.aks_cluster_default_nodepool_version[0]: Modifications complete after 9m54s

Key assertions verified:

• ✅ Control plane (aks_cluster_post_create) completed before any node pool upgrade started
• ✅ Zero NodePoolMcVersionIncompatible errors throughout the entire process
• ✅ azurerm_kubernetes_cluster_node_pool showed no orchestrator_version diff (ignore_changes confirmed working)
• ✅ Both default and extra node pools successfully upgraded to target version

Compatibility

• Non-breaking change: Existing users only need to upgrade the module version. No migration steps required.
• No plan drift: ignore_changes prevents Terraform from detecting version drift on orchestrator_version, which is intentional — version management is delegated to azapi_update_resource.
• Existing pattern: This follows the exact same architecture established by PR [Breaking] - Ignore changes on kubernetes_version from outside of Terraform #336 for control plane upgrades (kubernetes_version → ignore_changes + azapi_update_resource + null_resource keeper).
• Rollback risk: Low. The azapi_update_resource pattern has been battle-tested for control plane upgrades since PR [Breaking] - Ignore changes on kubernetes_version from outside of Terraform #336.

[jiaweitao001]

🚢