"dry-run failed" when upgrading API version

[Tolsto]

Describe the bug

I'm trying to upgrade the API of a CRD resource through flux and get dry-run failed: .spec.accessPolicy: field not declared in schema. When applying the same change via kubectl it works without issues.

Steps to reproduce

Consider this CRD object already being applied in the cluster:

apiVersion: policy.linkerd.io/v1beta1
kind: Server
metadata:
  name: rabbitmq-management
  namespace: rabbitmq-instances
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/part-of: rabbitmq
  port: management

When upgrading this to v1beta3

apiVersion: policy.linkerd.io/v1beta3
kind: Server
metadata:
  name: rabbitmq-management
  namespace: rabbitmq-instances
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/part-of: rabbitmq
  port: management

and applying the change through Flux it will result in error Server/rabbitmq-instances/rabbitmq-management dry-run failed: .spec.accessPolicy: field not declared in schema

You can find the CRD here: https://github.com/linkerd/linkerd2/blob/main/charts/linkerd-crds/templates/policy/server.yaml

accessPolicy is a new, optional field of version v1beta3. Applying the same change via kubectl apply works.

Expected behavior

No error when applying the change

Screenshots and recordings

No response

OS / Distro

N/A

Flux version

N/A

Flux check

► checking prerequisites
✗ flux 2.7.3 <2.7.5 (new CLI version is available, please upgrade)
✔ Kubernetes 1.34.2-eks-b3126f4 >=1.32.0-0
► checking version in cluster
✔ distribution: flux-v2.7.5
✔ bootstrapped: true
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v1.4.5
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v1.7.3
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v1.7.5
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v1.7.4
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta3
✔ buckets.source.toolkit.fluxcd.io/v1
✔ externalartifacts.source.toolkit.fluxcd.io/v1
✔ gitrepositories.source.toolkit.fluxcd.io/v1
✔ helmcharts.source.toolkit.fluxcd.io/v1
✔ helmreleases.helm.toolkit.fluxcd.io/v2
✔ helmrepositories.source.toolkit.fluxcd.io/v1
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1
✔ ocirepositories.source.toolkit.fluxcd.io/v1
✔ providers.notification.toolkit.fluxcd.io/v1beta3
✔ receivers.notification.toolkit.fluxcd.io/v1
✔ all checks passed

Git provider

No response

Container Registry provider

No response

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[matheuscscp]

What kubectl command are you using? Do you have cluster-admin with this kubectl command?

[Tolsto]

Yes, cluster-admin. Tried it with both kubectl apply -f manifest.yaml and kubectl apply --dry-run=server -f manifest.yaml

[matheuscscp]

And I bet the Flux Kustomization applying this object for you is not cluster-admin?

[matheuscscp]

I bet it has .spec.serviceAccountName set. Try kubectl apply --dry-run=server -f manifest.yaml --as=system:serviceaccount:<sa-namespace>:<sa-name>

[Tolsto]

No, the Kustomization doesn't have .spec.serviceAccountName set, and the kustomize-controller uses the kustomize-controller serviceaccount which has a ClusterRoleBinding to cluster-admin

[matheuscscp]

Is the --default-service-account flag set in kustomize-controller?

[Tolsto]

Is the --default-service-account flag set in kustomize-controller?

No, all Flux manifests are vanilla via flux install --export

[matheuscscp]

Ok, this may actually be a bug, I'll investigate

[stefanprodan]

The CRD cache may be stale, kill the kustomize-controller pod, and see if it still fails after the restart.

To simulate what Flux does, you need to do:

kubectl apply --server-side --dry-run=server --field-manager=kustomize-controller -f

[Tolsto]

Thank you, I was able to reproduce the issue with kubectl and --servers-side. Looks like a bug in Kubernetes itself. The bug ticket for it didn't get much love though: kubernetes/kubernetes#123582

[matheuscscp]

I don't think Kubernetes has the concept of applying multiple resources at the same time. It's always one per API call. What you confirmed is a very basic and correct behavior in Kubernetes: you cannot add fields to a resource if they are not in the schema. kubectl gives you an illusion that you are applying them at the same time, but this is not true.

However, this is 100% possible with Flux, which stages the apply of CRDs before any CRs of this CRD are dry-run'd:

https://github.com/fluxcd/pkg/blob/0ce88c6c9b226059a73f56da30d8cbce472a92fa/ssa/manager_apply.go#L250-L271

// ApplyAllStaged extracts the cluster and class definitions, applies them with ApplyAll,
// waits for them to become ready, then it applies all the other objects.
// This function should be used when the given objects have a mix of custom resource definition
// and custom resources, or a mix of namespace definitions with namespaced objects.
// If an error occurs during the apply of the cluster or class definitions, the change set is
// returned with the applied entries, up to that point, and the error is returned.
func (m *ResourceManager) ApplyAllStaged(ctx context.Context, objects []*unstructured.Unstructured, opts ApplyOptions) (*ChangeSet, error) {
	changeSet := NewChangeSet()

	var (
		// Contains only CRDs, ClusterRoles, and Namespaces.
		defStage []*unstructured.Unstructured

		// Contains only Class definitions.
		classStage []*unstructured.Unstructured

		// Contains the custom kinds.
		customStage []*unstructured.Unstructured

		// Contains all objects except for cluster definitions and class definitions.
		resStage []*unstructured.Unstructured
	)

Each stage is atomic within itself: an SSA dry-run is performed for every object and if at least one errors out, nothing is applied.

The defStage applies CRDs. Any CRs for a CRD can only be applied in one of the remaining 3 stages. Which means you probably have a cache bug...

[Tolsto]

The problem happens when the CRD upgrade was already applied and I only try to bump the API version of the CR object itself.
I tried restarting the flux controllers, it didn't help. And since I can reproduce the problem with kubectl I assume it's a Kubernetes bug and there isn't much that Flux can do about it?

To reiterate, the cluster already has the CRD definitions for both v1beta1 and v1beta3.

# manifest.yaml

apiVersion: policy.linkerd.io/v1beta1
kind: Server
metadata:
  name: test
  namespace: default
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/part-of: test
  port: test

apply this

> kubectl apply --server-side  -f manifest.yaml
Warning: policy.linkerd.io/v1beta1 Server is deprecated; use policy.linkerd.io/v1beta3 Server
server.policy.linkerd.io/test serverside-applied

then

# manifest_new.yaml

apiVersion: policy.linkerd.io/v1beta3
kind: Server
metadata:
  name: test
  namespace: default
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/part-of: test
  port: test

> kubectl apply --server-side  -f manifest_new.yaml
Error from server: .spec.accessPolicy: field not declared in schema

However, running without server-side apply works (which is the workaround I now use to upgrade these resources)

> kubectl apply -f manifest_new.yaml
server.policy.linkerd.io/test configured

[matheuscscp]

And you are not even adding .spec.accessPolicy at all in the YAML, are you? Can you please run a kubectl get server test -o yaml --show-managed-fields?

[Tolsto]

And you are not even adding .spec.accessPolicy at all in the YAML, are you?

No

Can you please run a kubectl get server test -o yaml --show-managed-fields?

apiVersion: policy.linkerd.io/v1beta3
kind: Server
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"policy.linkerd.io/v1beta3","kind":"Server","metadata":{"annotations":{},"name":"test","namespace":"default"},"spec":{"podSelector":{"matchLabels":{"app.kubernetes.io/part-of":"test"}},"port":"test"}}
  creationTimestamp: "2026-02-16T19:11:35Z"
  generation: 1
  name: test
  namespace: default
  resourceVersion: "1864383457"
  uid: a4614675-6dc9-4c71-a436-a78250f6bb24
spec:
  accessPolicy: deny
  podSelector:
    matchLabels:
      app.kubernetes.io/part-of: test
  port: test
  proxyProtocol: unknown

[matheuscscp]

I was expecting to see .metadata.managedFields in the output...