feat: per-job oidc_tokens block for custom audiences

[cchristous]

Status: Draft — proto changes pending maintainer assistance

This PR adds a per-job oidc_tokens: block to the pipeline yaml that lets pipelines mint additional OIDC tokens with custom audiences, alongside the existing default SEMAPHORE_OIDC_TOKEN.

Why: The OIDC aud claim (RFC 7519 §4.1.3) identifies the token's intended consumer. Today Semaphore hardcodes aud = https://<org>.semaphoreci.com, which prevents Semaphore tokens from being used with services that strictly verify a specific audience. The motivating case is PyPI, which requires aud="pypi" (downstream PR pypi/warehouse#19048 implementing pypi/warehouse#18882). Other ecosystems with the same constraint include npm trusted publishing, Docker Hub OIDC, and HashiCorp Vault bound_audiences.

What's in this PR

Schema (plumber/spec/priv/v1.0.yml)

jobs:
  - name: Publish to PyPI
    oidc_tokens:
      PYPI_OIDC_TOKEN:
        aud: pypi
      NPM_TOKEN:
        aud: ["npm-prod", "npm-staging"]
    commands:
      - ...

Validation rules (yaml-level, fail fast):

1. Each entry MUST have aud. Missing aud → schema validation failure.
2. aud is a non-empty string (≤256 chars) OR a non-empty list of non-empty strings (1–8 entries, each ≤256 chars).
3. Yaml key (env var name) MUST match ^[A-Z_][A-Z0-9_]*$. Up to 16 entries per job.
4. Reserved key SEMAPHORE_OIDC_TOKEN rejected at semantic-validation time.

Secrethub JWT generator

secrethub/lib/secrethub/open_id_connect/jwt.ex learns to honor an optional :audience field on the request struct. When set, it overrides the default org-URL audience. Single-element list flattens to a string per RFC 7519 convention (necessary for strict_aud-checking consumers like PyPI). Multi-element list becomes a JSON array.

req.audience	JWT aud claim
absent / [] (default)	"https://<org>.<domain>" (current behavior)
["pypi"] (single-element list)	"pypi" (string)
["a", "b"] (multi-element list)	["a", "b"] (JSON array)

Pure additive change — existing callers unaffected.

Semantic validator

plumber/ppl/lib/ppl/definition_reviser/oidc_tokens_validator.ex rejects SEMAPHORE_OIDC_TOKEN as a custom token name (it's the reserved name for the auto-injected default). Wired into the definition_reviser with chain alongside JobMatrixValidator / ParallelismValidator.

Documentation

User guide and reference pages updated under docs/docs/using-semaphore/openid.md and docs/docs/reference/openid.md.

What's NOT in this PR (needs maintainer assistance)

The full feature requires changes to renderedtext/internal_api (the private proto-source repo) which I don't have access to:

1. Add optional audience field to GenerateOpenIDConnectTokenRequest in secrethub.proto. Once added and .pb.ex files regenerated, internal_grpc_api.ex can pass the field through to JWT.generate_and_sign/1 (which already reads it via Map.get(req, :audience, [])).

2. Add OIDCTokenSpec message and oidc_tokens field on Job.Spec in jobs.proto (or whichever proto file currently defines Semaphore.Jobs.V1alpha.Job.Spec). Once added:

   • plumber/ppl needs a small change to compile yaml oidc_tokens (a map from name to {aud}) into the proto repeated OIDCTokenSpec oidc_tokens field.
   • zebra/lib/zebra/workers/job_request_factory/open_id_connect.ex needs a new load_oidc_tokens/5 function that mints one additional OIDC token per Job.Spec.oidc_tokens entry (with the requested audience) and emits the env vars. The default-token path stays untouched.

The implementation pattern for both Zebra and the plumber compilation follows the existing secrets: block precisely.

I've intentionally not modified any .pb.ex files in this PR (those are generated). Once the proto changes land, regenerating in the consuming services (plumber/proto, secrethub, zebra, front, guard, public-api, ee/gofer) plus the Zebra and plumber/ppl changes above complete the feature.

I'm happy to do those final pieces myself if you can grant me the proto-repo access, or to review/iterate if a maintainer wants to drive them.

Tests

• secrethub/test/secrethub/open_id_connect/jwt_test.exs — 7 tests:

  • 4 audience-handling cases (default-via-absent, default-via-empty-list, single-element flatten, multi-element JSON array)
  • audience: nil falls back to default (covers proto-deserialization edge case)
  • iss invariance: overriding aud does not change iss (asserted on default and override paths)
  • JWTFilter interaction (with :open_id_connect_filter enabled): asserts aud survives when allowlisted; complementary test asserts aud is stripped when not allowlisted (plus refute of a non-allowlisted claim to prove the filter actually ran).
    Existing 134 secrethub regression tests pass.

• plumber/spec/test/pipelines/v1.0-oidc_tokens*.yml — 11 fixtures auto-discovered by SemaphoreYamlSpecTest:

  • 1 positive (string + list aud forms)
  • 7 negative .fail.yml covering each rejection rule:
    missing_aud, invalid_key, empty_aud_list, aud_empty_string, aud_too_long (257 chars), too_many_audiences (9 entries), too_many_tokens (17 entries)
  • 3 positive at-bound fixtures (at_max_tokens = 16, at_max_audiences = 8, at_max_aud_length = 256) so a regression that tightens a bound flips a passing fixture.

• plumber/ppl/test/definition_reviser/oidc_tokens_validator_test.exs — 6 unit tests covering happy paths, reserved name in regular block, reserved name in after_pipeline, and reserved name on second job in a block (proves block iteration isn't short-circuited).

Schema bounds

The oidc_tokens schema has explicit upper limits to prevent claim-bloat / DoS-style abuse:

• maxProperties: 16 — at most 16 entries per job
• maxLength: 256 — each aud string up to 256 chars
• maxItems: 8 — aud list up to 8 audiences
• minLength: 1, minItems: 1 — empty values rejected

Each bound has a positive fixture at the limit and a negative fixture one over.

Out-of-scope test fix

plumber/ppl/test/definition_reviser/global_job_config_test.exs had a brittle on_exit callback that called System.put_env/2 with a nil value (returned by System.get_env/1 when the env var was unset before the test). Replaced with a small restore_env/2 helper that calls System.delete_env/1 for the nil case. Strictly an improvement; the test now passes regardless of host env state. Out of scope for the feature, but it was the only thing keeping CI red on this branch.

mix format clean, mix credo --strict clean on new files.

Backward compatibility

• Existing pipelines unchanged.
• SEMAPHORE_OIDC_TOKEN semantics unchanged.
• :audience field on the JWT request map is optional (read via Map.get(..., [])); old callers unaffected.
• oidc_tokens is purely additive; pipelines that don't use it behave identically to today.

Open questions for maintainers

1. Naming. I chose oidc_tokens to align with Semaphore's existing SEMAPHORE_OIDC_TOKEN env var vocabulary. GitLab CI uses id_tokens. Open to either.
2. Feature flag scoping. Today's OIDC is gated by :open_id_connect. Should oidc_tokens ride this same flag, or get a new one for staged rollout?
3. Self-hosted (CE) eligibility. OIDC is Enterprise-only on SaaS today. The original requestor of this feature (pypi/warehouse#18882) is a self-hosted user. Should oidc_tokens be available in CE?
4. Yaml shape. I chose a map (yaml key = env var name); existing secrets: is a list of objects. Map matches GitLab and is more compact for the common single-entry case. Open to flipping if you prefer.

Related

• Downstream PR: pypi/warehouse#19048
• Original feature request: pypi/warehouse#18882
• Internal design doc: linked in the PR-affiliated repo