helm chart build fails

[aojea]

@fmuyassarov this is failing https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/post-dranet-image/2042591551317610496

Originally posted by @aojea in #130 (comment)

[fmuyassarov]

I'm on it.

[fmuyassarov]

I understand the issue and know how to fix. But i'm away from my keyboard right now. Should be back in 2 hours or so and file the fix.

[aojea]

I understand the issue and know how to fix. But i'm away from my keyboard right now. Should be back in 2 hours or so and file the fix.

@fmuyassarov this is not a production critical thing, so take your time, enjoy the weekend if you already started and do not rush it

[fmuyassarov]

Thanks @aojea . Here is what happened and I would like to hear if there is any objection or better idea for fixing the issue.
/cc @gauravkghildiyal

The issue is release target previously always ran helm-package and helm-push, which caused failures in postsubmit CI job triggered on commit merges (not tags). Because, helm-package requires an exact git tag to produce a valid semver chart version. I made it intentionally to run only on git tags so that we don't package and push the charts to staging on every commit.

To fix this, I would propose that we update the release target to skip helm packaging and push when NOT on an exact git tag. This should make postsubmit job to run make release on every commit merge without failing.

diff --git a/Makefile b/Makefile
index afe6e0f..72398cd 100644
--- a/Makefile
+++ b/Makefile

-# The main release target, which pushes all images and helm charts
-release: ensure-helm image-push helm-push
+# The main release target, which pushes all images and helm charts.
+# Helm chart packaging and push is skipped when not on an exact git tag.
+release: image-push
+       @if [ -n "$(HELM_TAG)" ]; then \
+               echo "On tag $(HELM_TAG), packaging and pushing helm chart..."; \
+               $(MAKE) ensure-helm helm-push; \
+       else \
+               echo "Not on a tag, skipping helm chart push"; \

However, only this isn't enough. Sure, we won't see the prow failure but we won't see the actual charts published either. That's because, current post-dranet-image postsubmit job runs on every commit merge and but never runs on a git tag. In other words, helm chart packaging will never get triggered for actual releases/tags. For this, we can add a new post-dranet-release postsubmit job that triggers exclusively on semver tags (e.g. v1.1.1). This job will run the same cloudbuild pipeline, and with the conditional release target in the Makefile, will build and push both the container image and the helm chart to the staging registry.

diff --git a/config/jobs/image-pushing/k8s-staging-dranet.yaml b/config/jobs/image-pushing/k8s-staging-dranet.yaml
index 67604a6673..ef1e739c37 100644
--- a/config/jobs/image-pushing/k8s-staging-dranet.yaml
+++ b/config/jobs/image-pushing/k8s-staging-dranet.yaml
@@ -30,3 +30,33 @@ postsubmits:
       rerun_auth_config:
         github_team_ids:
           - 2921947 # kind-maintainers
+
+    - name: post-dranet-release
+      branches:
+        # only run on semver tags (e.g. v0.1.0)
+        - ^v(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(?:-((?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+([0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$
+      cluster: k8s-infra-prow-build-trusted
+      annotations:
+        testgrid-dashboards: sig-network-dranet, sig-k8s-infra-gcb
+        testgrid-alert-email: maspinwall@google.com, gauravkg@google.com, antonio.ojea.garcia@gmail.com
+        testgrid-num-columns-recent: '3'
+      decorate: true
+      spec:
+        serviceAccountName: gcb-builder
+        containers:
+          - image: gcr.io/k8s-staging-test-infra/image-builder:v20251215-d7853fe2a6
+            command:
+              - /run.sh
+            args:
+              - --project=k8s-staging-networking
+              - --scratch-bucket=gs://k8s-staging-networking-gcb
+              - --env-passthrough=PULL_BASE_SHA
+              - --build-dir=.
+              - --with-git-dir
+              - .
+            env:
+              - name: LOG_TO_STDOUT
+                value: "y"
+      rerun_auth_config:
+        github_team_ids:
+          - 2921947 # kind-maintainers

[fmuyassarov]

And with the new job, we would fully follow the same releasing procedure we are trying to described in #131 . In other words:

1. Push a release tag (e.g. v1.1.0)
2. Generate release notes
3. Cloud Build builds & pushes staging artifacts (<- this is missing without the new job)
4. Retrieve artifact digests from staging
5. Promote artifacts via PR to k8s.io
6. Verify artifacts are live at registry.k8s.io
7. Publish GitHub Release

[gauravkghildiyal]

Thanks for looking into this @fmuyassarov

Is it okay if we converge to the pattern officially described at https://github.com/kubernetes/test-infra/blob/master/config/jobs/image-pushing/README.md? I believe our cloudbuild config and the upstream prow config is lacking a couple of things which introduces this problem. Let's try to move towards the recommended pattern?

[fmuyassarov]

What I've described above (the new job) is actually based on the same document or do you mean to say we should stop pushing images on commits and only stick to a single job that would run only on tags?

[gauravkghildiyal]

I was thinking a single job should be sufficient for both, as long as both the job and cloudbuild and appropriately configured. Is that not the case?

[fmuyassarov]

If Prow supports adding a trigger (for both commit and tag, I need to check this) - then we can have a single job. But then we need to anyways tune the cloudbuild to have a condition that it would skip the helm related bits in case it is called based on commit. Does this match your thinking ?

[gauravkghildiyal]

Yep I've the same understanding

[fmuyassarov]

Alright thanks. Then I will check the Prow job and cloudbuild options and soon confirm here so that we can decide which approach is better to go forward.

[fmuyassarov]

@gauravkghildiyal Based on my understanding of the Prow documentation, we can use a single job to trigger Cloud Build for both commits to main and semver tag pushes. Then, within the Makefile (invoked by cloudbuild), we can control whether Helm packaging is required or should be skipped.

The Makefile changes are needed anyway to unblock the current Prow job failure. The k8s-staging-dranet.yaml changes below is for triggering the cloudbuild on tag creation as well and not just on commits as it is today. Also, it is following the recommendation from the Prow documentation.

postsubmits:
  kubernetes-sigs/dranet:
    - name: post-dranet-image
      branches:
        - ^main$
        # Build semver tags, too
        - ^v(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(?:-((?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+([0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$
      cluster: k8s-infra-prow-build-trusted
      annotations:
        testgrid-dashboards: sig-network-dranet, sig-k8s-infra-gcb
        testgrid-alert-email: maspinwall@google.com, gauravkg@google.com, antonio.ojea.garcia@gmail.com
        testgrid-num-columns-recent: '3'
      decorate: true
      spec:
        serviceAccountName: gcb-builder
        containers:
          - image: gcr.io/k8s-staging-test-infra/image-builder:v20251215-d7853fe2a6
            command:
              - /run.sh
            args:
              - --project=k8s-staging-networking
              - --scratch-bucket=gs://k8s-staging-networking-gcb
              - --env-passthrough=PULL_BASE_SHA
              - --build-dir=.
              - --with-git-dir
              - .
            env:
              - name: LOG_TO_STDOUT
                value: "y"
      rerun_auth_config:
        github_team_ids:
          - 2921947 # kind-maintainers

Overall, this feels like a cleaner approach than what I initially proposed in my previous comments. I’ve already filed the patch for the Makefile in #141 and for the test infra in kubernetes/test-infra#36794. Would appreciate your thoughts.