Kustomize-controller is reconciling stale source artifact revision

[dipti-pai]

Seeing an issue where kustomize-controller is reconciling older OCI artifact. The OCIrepo is healthy and continuously reconciled by source-controller.

Manifests

Oci Repo :

apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
  creationTimestamp: "2026-01-28T08:38:39Z"
  finalizers:
  - finalizers.fluxcd.io
  generation: 2
  name: my-controller-deployment-oci-repo
  namespace: myapp-infra-agent-flux-ns
  resourceVersion: "693649634"
  uid: ae5d6cf3-d1f2-4cfb-b978-115856c23f9d
spec:
  interval: 2h0m0s
  provider: azure
  ref:
    digest: sha256:575e4da2cfe2f22c8420d1448b42834550f6f1b30b32b0b9956b02d10fdde35a
    tag: 1.0.03320.1109
  timeout: 1m0s
  url: <url>
  verify:
    provider: notation
    secretRef:
      name: my-controller-deployment-notation-config
status:
  artifact:
    digest: sha256:e639ec90c519eef050d875a4c16bfd400094e753aecfca6cbab8f87bfabc3631
    lastUpdateTime: "2026-02-10T06:56:48Z"
    metadata:
      org.opencontainers.image.created: "2026-02-03T23:43:47Z"
    path: ocirepository/myapp-infra-agent-flux-ns/my-controller-deployment-oci-repo/sha256:575e4da2cfe2f22c8420d1448b42834550f6f1b30b32b0b9956b02d10fdde35a.tar.gz
    revision: sha256:575e4da2cfe2f22c8420d1448b42834550f6f1b30b32b0b9956b02d10fdde35a
    size: 3225
    url: http://source-controller.flux-system.svc.cluster.local./ocirepository/myapp-infra-agent-flux-ns/my-controller-deployment-oci-repo/sha256:575e4da2cfe2f22c8420d1448b42834550f6f1b30b32b0b9956b02d10fdde35a.tar.gz
  conditions:
  - lastTransitionTime: "2026-02-10T06:56:48Z"
    message: stored artifact for digest 'sha256:575e4da2cfe2f22c8420d1448b42834550f6f1b30b32b0b9956b02d10fdde35a'
    observedGeneration: 2
    reason: Succeeded
    status: "True"
    type: Ready
  - lastTransitionTime: "2026-02-10T06:56:48Z"
    message: stored artifact for digest 'sha256:575e4da2cfe2f22c8420d1448b42834550f6f1b30b32b0b9956b02d10fdde35a'
    observedGeneration: 2
    reason: Succeeded
    status: "True"
    type: ArtifactInStorage
  - lastTransitionTime: "2026-02-10T06:56:48Z"
    message: verified signature of revision sha256:575e4da2cfe2f22c8420d1448b42834550f6f1b30b32b0b9956b02d10fdde35a
    observedGeneration: 2
    reason: Succeeded
    status: "True"
    type: SourceVerified
  observedGeneration: 2
  url: http://source-controller.flux-system.svc.cluster.local./ocirepository/myapp-infra-agent-flux-ns/my-controller-deployment-oci-repo/latest.tar.gz

Kustomization

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  creationTimestamp: "2026-01-28T08:40:54Z"
  generation: 2
  name: my-controller-deployment-kustomization
  namespace: myapp-infra-agent-flux-ns
  resourceVersion: "694929473"
  uid: e07f4b97-f243-4615-a0be-36c3eee8e16b
spec:
  force: false
  interval: 2h56m0s
  path: ./configuration/my-controller-deployment/flux/E3B0C4429
  postBuild:
    substitute:      
       <omitted>
  prune: true
  serviceAccountName: my-controller-deployment-flux-sa
  sourceRef:
    kind: OCIRepository
    name: my-controller-deployment-oci-repo
    namespace: myapp-infra-agent-flux-ns
  timeout: 10m0s
  wait: true
status:
  conditions:
  - lastTransitionTime: "2026-02-10T06:40:36Z"
    message: 'Applied revision: sha256:b0347e217b6a9d8031df3309b275733dbcf8d2277266a651ed9355cc26c7848c'
    observedGeneration: 1
    reason: ReconciliationSucceeded
    status: "True"
    type: Ready
  - lastTransitionTime: "2026-02-10T06:40:36Z"
    message: Health check passed in 295.984986ms
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: Healthy
  inventory:
    entries:
       <omitted>
  lastAppliedRevision: sha256:b0347e217b6a9d8031df3309b275733dbcf8d2277266a651ed9355cc26c7848c
  lastAttemptedRevision: sha256:b0347e217b6a9d8031df3309b275733dbcf8d2277266a651ed9355cc26c7848c
  observedGeneration: 2

Restarting kustomize-controller solves the issue and the newer artifact is reconciled after a restart indicating that kustomize-controller is not getting the latest revision from the source.

[stefanprodan]

This means that the API server didn't sent the event to the kustomize-controller informer, nothing we can do in Flux if the Kubernetes API doesn't behave like it should. The new artifact will be reconcile at the Kustomization interval as we query the API directly.

[dipti-pai]

@stefanprodan Thanks!

kustomize-controller is reconciling the older artifact even at the kustomization interval. The newer artifact is seen by the kustomize-controller only after restarting.

Have we seen this behavior before where kustomize-controller is getting stale artifact even while querying API server?

Any pointers to debug the same ?

[stefanprodan]

I've never seen this, the only way this can happen is if the controller-runtime cache that Get calls hit is stale. Hints to the same issue I mentioned above.

[stefanprodan]

The new artifact will be reconcile at the Kustomization interval as we query the API directly.

This is not true, sorry. All client.Get calls hit the controller-runtime cache:

kustomize-controller/internal/controller/kustomization_controller.go

Line 666 in c0589b4

func (r *KustomizationReconciler) getSource(ctx context.Context,

[matheuscscp]

The default cache resync period is quite long (hours long). Try patching something in the source object to see if kustomize-controller gets an update, e.g. add/change/remove some annotation

[anagg929]

We updated the annotations/labels on the OCIRepository object, but the changes don’t seem to be reflected in the Kustomization reconciliation. Could this be due to the predicate configured in the manager?

Specifically referring to:

kustomize-controller/internal/controller/kustomization_manager.go

Lines 163 to 164 in c0589b4

	builder.WithPredicates(SourceRevisionChangePredicate{}),
	).

Is there a way to force a cache refresh?

Even after waiting for over a day, the cache doesn’t seem to update automatically, which makes sense given there’s no resync interval configured on the manager.

[stefanprodan]

You should be looking into why informers are broken in your cluster, this affects all Kubernetes controllers build with controller-runtime. We are not doing anything special in Flux, it's standard controller-runtime usage.

[dipti-pai]

Thanks @stefanprodan and all!

We root caused this to an AKS Node Pool upgrade happening at the same time that the OCI repo was updated. The AKS cluster had a few mutating webhooks installed that were failing at the same time as the OCIRepo update and our hypothesis is that this led to interruptions in the watch stream that kustomize-controller has with the API server leading to missed events.

Once the event is missed, kustomize-controller is not able to recover from this situation without a restart OR a new source artifact being published. What are your thoughts on the following improvements to make kustomize-controller more robust to missed events -

1. Add configuration to kustomize-controller's controller-runtime cache options with a resync period so all objects are periodically re-listed from the API server, re-triggering the predicates.

2. Query the API server directly at kustomization .spec.interval. This could add an additional overhead, thoughts?

3. There were no errors in the kustomize-controller logs regarding broken watch. --log-level allows configuring logs with -v=0,1,2. Support klog -v=3 for more verbose logging? It might be possible to set this via environment variable too, need to confirm.

[stefanprodan]

Add configuration to kustomize-controller's controller-runtime cache options with a resync period so all objects are periodically re-listed from the API server, re-triggering the predicates.

A cache resync will not only re-triggering the predicates, it will also cause a storm of SSA dry-run of all Kustomizations. We have users with 10K Kustomizations in a cluster, doing this would have a major impact on their API server.

Query the API server directly at kustomization .spec.interval. This could add an additional overhead, thoughts?

We could switch the source fetch from the normal client to the APIReader which will bypass the cache. This would be the less impactful change, but we can't do this only here, this must be done in helm-controller also as both controllers should behave the same.

[matheuscscp]

this must be done in helm-controller also as both controllers should behave the same.

source-watcher also fetches sources, not for applying manifests, but from the perspective of what happened here I think it's also relevant

[stefanprodan]

I agree, source-watcher should also bypass the cache if we're doing this.

[stefanprodan]

So far we've strictly limited the use of APIReader to dependency resolution, on clusters with 10K Kustomizations most of them will not have dependsOn, usually deps are used in the top level Kustomizations. Now if we decide to bypass the cache for source fetching, users running Flux at scale will see an increase in API calls. Knowing the setup of some of these users, they will see an extra 480K API calls per day.

[matheuscscp]

So far we've strictly limited the use of APIReader to dependency resolution, on clusters with 10K Kustomizations most of them will not have dependsOn, usually deps are used in the top level Kustomizations. Now if we decide to bypass the cache for source fetching, users running Flux at scale will see an increase in API calls. Knowing the setup of some of these users, they will see an extra 480K API calls per day.

It doesn't look like it's worth solving this problem at this cost

[anagg929]

A cache resync will not only re-triggering the predicates, it will also cause a storm of SSA dry-run of all Kustomizations. We have users with 10K Kustomizations in a cluster, doing this would have a major impact on their API server.

Why don't we add a predicate , that validates the create event's source revision against the cache of the Kustomization status last synced. This would make sure that the resync messages (coming as create events) are not going past the predicate stage if the last revision is applied by kustomization. This will avoid any SSA dry-runs and all we are doing is running List calls and comparing against caches.

[stefanprodan]

Why don't we add a predicate , that validates the create event's source revision against the cache

A cache resync will send an event for all Kustomizations as this is the main watcher used. No matter if we filter sources, the controller will still have to reconcile all like at restart.