add drain_timeout_in_minutes and node_soak_duration_in_minutes

[zioproto]

Describe your changes

Implements the following GA functionalities:

• Node Soak duration for upgraded: https://azure.microsoft.com/en-us/updates/generally-available-node-soak-duration-for-upgrades
• Set node drain timeout value: https://learn.microsoft.com/en-us/azure/aks/upgrade-aks-cluster?tabs=azure-cli#set-node-drain-timeout-value

Requires provider version v3.106.0
hashicorp/terraform-provider-azurerm#26137

Issue number

#530

Checklist before requesting a review

• The pr title can be used to describe what this pr did in CHANGELOG.md file
• I have executed pre-commit on my machine
• I have passed pr-check on my machine

[zioproto]

When testing this PR consider all the following user reported issue with the upstream provider:

• If you add "kubelet_config" to the "azurerm_kubernetes_cluster_node_pool" entry, you will try to replace the node. #563
• Add Support for drain_timeout_in_minutes #561
• Support for node soak time #530
• Terraform wants to replace my cluster #557

[morbendor]

Hey,

when the upcoming version of the module, which includes this changes, will be released?

Our CI/CD pipeline is dependent on that fix.

Thank you for your assistance.

[aditya-enthu]

@zioproto can we consider adding upgrade config when auto-scaling is set to true:
https://github.com/zioproto/terraform-azurerm-aks/blob/feature/node-soak-duration-in-minutes/main.tf#L252

In addition, default value for agents_pool_drain_timeout_in_minutes is 30 min and setting it to null create resource to force-recreate.

upgrade_settings {
      - drain_timeout_in_minutes      = 30 -> null # forces replacement
      ~ node_soak_duration_in_minutes = 0 -> 10
        # (1 unchanged attribute hidden)
 }

while setting it to 30 min, or other values, it tries to update-in-place,

upgrade_settings {
      ~ drain_timeout_in_minutes      = 30 -> 40
      ~ node_soak_duration_in_minutes = 0 -> 10
        # (1 unchanged attribute hidden)
}

https://learn.microsoft.com/en-us/rest/api/aks/agent-pools/create-or-update?view=rest-aks-2024-02-01&tabs=HTTP#agentpoolupgradesettings

[zioproto]

when the upcoming version of the module, which includes this changes, will be released?

Because we have added two new required fields this will be released with version v10

[zioproto]

@aditya-enthu thanks for your review ! great feedback! I implemented your suggestions. Please have a look.

[aditya-enthu]

@aditya-enthu thanks for your review ! great feedback! I implemented your suggestions. Please have a look.

thank you for the quick fix, i tested locally and works fine.

[morbendor]

when the upcoming version of the module, which includes this changes, will be released?

Because we have added two new required fields this will be released with version v10

Thank you for the update. Could you please let us know the estimated release date for version v10, which includes the new required fields?

[zioproto]

Please @lonegunmanb could you comment on the release date ? thanks

[zioproto]

The end to end version upgrade test is failing :(
https://github.com/Azure/terraform-azurerm-aks/actions/runs/9641915202/job/26588582582?pr=564#step:6:247

It seems a problem with the default value of drain_timeout_in_minutes:

~ default_node_pool {
                    name                          = "nodepool"
                    tags                          = {}
                    # (33 unchanged attributes hidden)
        
                  ~ upgrade_settings {
                      ~ drain_timeout_in_minutes      = 0 -> 30
                        # (2 unchanged attributes hidden)
                    }
                }

@ms-henglu @lonegunmanb do you have any suggestion on how to activate this feature without recreating the cluster ?

Edit: after a closer look I don't see any "Force replacement" statement. I dont understand why the TestExampleUpgrade_without_monitor is failing. Can anyone help me understand why that step is failing ? thanks

[lonegunmanb]

LGTM!