Disabling public network access and using UserDefinedRouting

[andrewkreuzer]

Describe your scenario
I have created a cluster with outbound_type = UserDefinedRouting and public_network_access_enabled = false using the terraform provider. I am now receiving error:

Code="BadRequest" Message="UserDefinedRouting is not supported when Cluster has public network access set to Disabled.

or from the portal:

Failed to save Kubernetes service 'MyCluster'.
Error: UserDefinedRouting is not supported when Cluster has
public network access set to Disabled.

A support ticket was opened and I was told:

When running a Terraform plan that includes the option/value "publicnetworkaccess: 'disabled'" and using a UDR, the cluster creation should have failed validation and the cluster should not have been created. Prior to the last Azure CLI update, this validation was skipped and the cluster was allowed to be built, however, that should not have been allowed

Feedback
I'm confused as to why this is not supported.

Setting private_cluster_enabled keeps the api endpoint within the vnet, setting public_network_access_enabled to false keeps the loadbalancer within our vnet, and using outbound_type UserDefinedRouting to control egress traffic through our firewall ensures we control all outbound traffic. The fact that this was allowed and the cluster is functioning is more confusing. If this is intended to not be supported why does it work?

We're now stuck in a state where we can't make changes to the cluster unless we enable public access (which would cause cluster re-creation)... and we have three clusters.

If there's something I'm misunderstanding or a technical reason as to why this is not supported I would be grateful of some insight

[zadigus]

For what it's worth, you can find here a repository that reproduces the issue. I am experiencing the very same problem since last Tuesday. It was working fine until that time for the last months.

[matthiasguentert]

Same issue here while trying to upgrade from 1.24.6 to 1.25.6 by using Azure CLI.

az aks upgrade --name <cluster> --resource-group <group> --subscription <subscription> --no-wait --kubernetes-version "1.25.6"
Kubernetes may be unavailable during cluster upgrades.
 Are you sure you want to perform this operation? (y/N): y
Since control-plane-only argument is not specified, this will upgrade the control plane AND all nodepools to version 1.25.6. Continue? (y/N): y
(BadRequest) UserDefinedRouting is not supported when Cluster has public network access set to Disabled.
Code: BadRequest
Message: UserDefinedRouting is not supported when Cluster has public network access set to Disabled.

It seems as if the Azure RM API has changed, as I was able to upgrade another cluster (created with the exact same version of terraform module) two weeks ago. The terraform module uses:

...
network_profile {
  ...
  outbound_type = "userDefinedRouting"
}
public_network_access_enabled = false
private_cluster_enabled = true

How am I now able to upgrade my clusters? 😐

[jvikes11]

According to my ticket with Microsoft, this is the new normal:

Recent changes to cluster validation during creation or update have caused this validation error to surface for customers using public_network_access_enabled=false in their Terraform template with UDR. Public_network_access_enabled must be set to True in the Terraform template for deployment to succeed. It is not possible to deploy a cluster with public_network_access_enabled=false via Az CLI or Azure Portal.

Will update any solution we find - it is not happening to all subscriptions, so might be a phased rollout

[zadigus]

What is the public_network_access_enabled = false option doing, other than setting very restrictive network security rules on the AKS workers subnet? Isn't that option redundant anyway?

[andrewkreuzer]

What is the public_network_access_enabled = false option doing, other than setting very restrictive network security rules on the AKS workers subnet?

It's my understanding it places the default loadbalancer in your vnet as opposed to being publicly accessible
kubernetes-internal vs kubernetes loadbalancer name

[zadigus]

What is the public_network_access_enabled = false option doing, other than setting very restrictive network security rules on the AKS workers subnet?

It's my understanding it places the default loadbalancer in your vnet as opposed to being publicly accessible

Well, I don't know how you deploy your AKS cluster, but in my case, my internal load-balancer (ILB) is deployed through the nginx ingress with the following annotations (among others):

controller:
  service:
    annotations:
      service.beta.kubernetes.io/azure-load-balancer-internal: "true"
      service.beta.kubernetes.io/azure-pls-name: "some-name"
      service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "some-private-subnet"

Therefore, my ILB gets deployed to a private subnet, and this has nothing to do with the public_network_access_enabled = false option.

[andrewkreuzer]

ya I didn't have to add those annotations because the loadbalancer was created in the subnet we have designated to aks (nodepools, api server private endpoint, loadbalancer's frontend ips (private))

[zadigus]

@andrewkreuzer ok I didn't know it was possible

[andrewkreuzer]

I'm beginning to believe it shouldn't be

[andrewkreuzer]

my bad I do have those annotations

annotations:
  service.beta.kubernetes.io/azure-load-balancer-internal: true
  service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /healthz

[jvikes11]

I haven't tried this yet because my test cluster was recreated, but this is a quick fix that Microsoft gave me to change the public access property without recreating the cluster:

az deployment group create --template-file clusterupdate_publicaccess.bicep -g ResourceGroupoftheCluster

Apparently if you have the private cluster enabled the public access option isn't needed and the cluster is still closed off from public access.

[matthiasguentert]

@jvikes11 what does the content of clusterupdate_publicaccess.bicep look like?

[zadigus]

I haven't tried this yet because my test cluster was recreated, but this is a quick fix that Microsoft gave me to change the public access property without recreating the cluster:

az deployment group create --template-file clusterupdate_publicaccess.bicep -g ResourceGroupoftheCluster

Apparently if you have the private cluster enabled the public access option isn't needed and the cluster is still closed off from public access.

that might be true if, and only if, your workers are deployed to a private subnet

[matthiasguentert]

In my case, there are two potential methods for fixing this configuration issue (taken from the troubleshooting guide from within the azure portal):

1. Update the cluster through ARM or Bicep and Azure CLI. An example Bicep script is available below and can be used via az deployment group create --template-file clusterupdate.bicep -g

param cluster_name string
param location string

resource aks_cluster 'Microsoft.ContainerService/managedClusters@2023-03-02-preview' = {
  name: cluster_name
  location: location
  properties: {
        publicNetworkAccess: 'Enabled'

  }
}

For convenience, here is the ARM version as well

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "metadata": {
    "_generator": {
      "name": "bicep",
      "version": "0.17.1.54307",
      "templateHash": "15811174133861481625"
    }
  },
  "parameters": {
    "cluster_name": {
      "type": "string"
    },
    "location": {
      "type": "string"
    }
  },
  "resources": [
    {
      "type": "Microsoft.ContainerService/managedClusters",
      "apiVersion": "2023-03-02-preview",
      "name": "[parameters('cluster_name')]",
      "location": "[parameters('location')]",
      "properties": {
        "publicNetworkAccess": "Enabled"
      }
    }
  ]
}

2. Update the value in your Terraform code or ARM template to set publicNetworkAccess: "Enabled" (or in the case of Terraform, public_network_access_enabled = true) and apply the updated template. In the case of Terraform, this will result in a deletion and recreation of the cluster.