Infra Update v6: Modules Injection, More Provenance

[CodeGat]

References issue ACCESS-NRI/build-cd#311 and PR ACCESS-NRI/build-cd#306

Important

This PR is a major update to the deployment infrastructure. See below for the prerequisites for this repository to be able to merge this PR.

Background

This update is a bundle of major-level changes.

The main new features include:

• spack.modules Section is Now Injected At Build Time: A long-standing pain point for developers was having to update spec and dependency versions in multiple places at once. Now, this is injected at build time. Users can still specify their own projections and includes as required (with the others injected automatically to get a working build), but this makes it much simpler. We recommend removing the spack.modules section entirely - it has been done for this PR, but can be reverted if required.

• Special Packages Now Specified In config/packages.json: Rather than a relatively obscure vars.BUILD_DB_PACKAGES, we now specify special packages for inclusion in the release provenance database in the config/packages.json file (specifically the .provenance field). Furthermore, we can include modulefiles for additional packages from the .injection field.

• All Schema Now Specified In Workflow Entrypoint Inputs: Schema versions for manifests and files under config/ were kept track on in vars - this was obscure, repo-scoped and untracked. It is now a new, required input to the CI workflow (inputs.*-schema-version), meaning users can pick schema versions for their files at the PR level, and it is now tracked by git. We can also pick a particular manifest schema to test against via the new, optional inputs.spack-manifest-schema-path - useful for manifests that are less restrictive, like the ones for software deployment rather than model deployment.

• .github/CODEOWNERS File Sets Required Reviewers: Currently only requires that changes to the .github folder require review from @CodeGat. This file can be changed as required by the MDR in this PR, or later. There was talk of adding CODEOWNERS for various spack.yamls.

Prerequisites for Merging

• Update build-cd entrypoints (this PR!)
• Add CODEOWNERS file (also this PR!)
• Add config/packages.json file (again in this PR!)
• Remove spack.modules section from spack.yaml, revert if there are breaking changes (yep, in this PR)
• Delete vars.*_SCHEMA_VERSION and vars.BUILD_DB_PACKAGE only when all non-draft PRs have rebased onto this PR
• Verify that modules are injected as appropriate for this MDR

[CodeGat]

An example of injected modules for the current spack.yaml on initial-manifest:

Release

spack:
  specs:
  - coastri-roms@git.2025.06.000
  packages:
    roms:
      require:
      - '@git.c1c5efd8276dfa67029363499ff7370b4582c47b'
    openmpi:
      require:
      - '@4.1.4'
    netcdf-c:
      require:
      - '@4.7.3'
    all:
      require:
      - '%intel@2021.8.0'
      - target=x86_64
  view: true
  concretizer:
    unify: true
  modules:
    default:
      tcl:
        include:
        - coastri-roms
        - openmpi
        - roms
        projections:
          coastri-roms: '{name}/2025.06.000'
          openmpi: '{name}/4.1.4-{hash:7}'
          roms: '{name}/c1c5efd8276dfa67029363499ff7370b4582c47b-{hash:7}'

Prerelease

spack:
  specs:
  - coastri-roms
  packages:
    roms:
      require:
      - '@git.c1c5efd8276dfa67029363499ff7370b4582c47b'
    openmpi:
      require:
      - '@4.1.4'
    netcdf-c:
      require:
      - '@4.7.3'
    all:
      require:
      - '%intel@2021.8.0'
      - target=x86_64
  view: true
  concretizer:
    unify: true
  modules:
    default:
      tcl:
        include:
        - coastri-roms
        - openmpi
        - roms
        projections:
          coastri-roms: '{name}/prX-Y'
          openmpi: 'coastri-roms/dependencies/prX-Y/{name}/4.1.4-{hash:7}'
          roms: 'coastri-roms/dependencies/prX-Y/{name}/c1c5efd8276dfa67029363499ff7370b4582c47b-{hash:7}'

[aidanheerdegen]

As we don't have a working deployment is it best to just cherry-pick these commits directly into #1 otherwise we'll end up making a deployment that is effectively #1 anyway.

[CodeGat]

But coastri-roms/pr1-1 is a working deployment? And do you mean not to merge these changes into main?

[aidanheerdegen]

But coastri-roms/pr1-1 is a working deployment?

It's a pre-release, and there is no actual deployment yet.

And do you mean not to merge these changes into main?

Yep. Either change the target and merge this PR into initial-manifest, or just cherry-pick the commits into it.

Or close #1 and merge this one after review.

[CodeGat]

This PR is being closed as we will cherry pick these changes onto the initial-manifest branch - just so we don't have to rebase. But it will be on main eventually!

[CodeGat]

Reopening and merging as I realised some of the infra requires changes on the default branch.