Infra Update v6: Modules Injection, More Provenance

[CodeGat]

References issue ACCESS-NRI/build-cd#311 and PR ACCESS-NRI/build-cd#306

Important

This PR is a major update to the deployment infrastructure. See below for the prerequisites for this repository to be able to merge this PR.

Background

This update is a bundle of major-level changes.

The main new features include:

• spack.modules Section is Now Injected At Build Time: A long-standing pain point for developers was having to update spec and dependency versions in multiple places at once. Now, this is injected at build time. Users can still specify their own projections and includes as required (with the others injected automatically to get a working build), but this makes it much simpler. We recommend removing the spack.modules section entirely - it has been done for this PR, but can be reverted if required.

• Special Packages Now Specified In config/packages.json: Rather than a relatively obscure vars.BUILD_DB_PACKAGES, we now specify special packages for inclusion in the release provenance database in the config/packages.json file (specifically the .provenance field). Furthermore, we can include modulefiles for additional packages from the .injection field.

• All Schema Now Specified In Workflow Entrypoint Inputs: Schema versions for manifests and files under config/ were kept track on in vars - this was obscure, repo-scoped and untracked. It is now a new, required input to the CI workflow (inputs.*-schema-version), meaning users can pick schema versions for their files at the PR level, and it is now tracked by git. We can also pick a particular manifest schema to test against via the new, optional inputs.spack-manifest-schema-path - useful for manifests that are less restrictive, like the ones for software deployment rather than model deployment.

• .github/CODEOWNERS File Sets Required Reviewers: Currently only requires that changes to the .github folder require review from @CodeGat. This file can be changed as required by the MDR in this PR, or later. There was talk of adding CODEOWNERS for various spack.yamls.

Prerequisites for Merging

• Update build-cd entrypoints (this PR!)
• Add CODEOWNERS file (also this PR!)
• Add config/packages.json file (again in this PR!)
• Remove spack.modules section from spack.yaml, revert if there are breaking changes (yep, in this PR)
• Delete vars.*_SCHEMA_VERSION and vars.BUILD_DB_PACKAGE only when all non-draft PRs have rebased onto this PR
• Verify that modules are injected as appropriate for this MDR

[CodeGat]

Release:

spack:
  specs:
  - access-esm1p5@git.2024.12.1
  packages:
    mom5:
      require:
      - '@git.access-esm1.5-2025.03.002=access-esm1.5'
    cice4:
      require:
      - '@git.access-esm1.5-2025.04.001=access-esm1.5'
    um7:
      require:
      - '@git.2024.10.17=access-esm1.5'
    gcom4:
      require:
      - '@git.2024.05.28=access-esm1.5'
    oasis3-mct:
      require:
      - '@git.access-esm1.5_2024.05.24=access-esm1.5'
    openmpi:
      require:
      - '@4.0.2'
    netcdf-c:
      require:
      - '@4.7.4'
    netcdf-fortran:
      require:
      - '@4.5.2'
    hdf5:
      require:
      - '@1.10.11'
    all:
      require:
      - '%intel@19.0.3.199'
      - target=x86_64_v4
  view: true
  concretizer:
    unify: true
  config:
    install_tree:
      root: $spack/../restricted/ukmo/release
    source_cache: $spack/../restricted/ukmo/source_cache
    build_stage:
    - $TMPDIR/restricted/spack-stage
  modules:
    default:
      tcl:
        projections:
          access-esm1p5: '{name}/2024.12.1'
          cice4: '{name}/access-esm1.5-2025.04.001-{hash:7}'
          mom5: '{name}/access-esm1.5-2025.03.002-{hash:7}'
          oasis3-mct: '{name}/access-esm1.5_2024.05.24-{hash:7}'
          openmpi: '{name}/4.0.2-{hash:7}'
          um7: '{name}/2024.10.17-{hash:7}'
        include:
        - access-esm1p5
        - cice4
        - mom5
        - oasis3-mct
        - openmpi
        - um7

Prerelease:

spack:
  specs:
  - access-esm1p5
  packages:
    mom5:
      require:
      - '@git.access-esm1.5-2025.03.002=access-esm1.5'
    cice4:
      require:
      - '@git.access-esm1.5-2025.04.001=access-esm1.5'
    um7:
      require:
      - '@git.2024.10.17=access-esm1.5'
    gcom4:
      require:
      - '@git.2024.05.28=access-esm1.5'
    oasis3-mct:
      require:
      - '@git.access-esm1.5_2024.05.24=access-esm1.5'
    openmpi:
      require:
      - '@4.0.2'
    netcdf-c:
      require:
      - '@4.7.4'
    netcdf-fortran:
      require:
      - '@4.5.2'
    hdf5:
      require:
      - '@1.10.11'
    all:
      require:
      - '%intel@19.0.3.199'
      - target=x86_64_v4
  view: true
  concretizer:
    unify: true
  config:
    install_tree:
      root: $spack/../restricted/ukmo/release
    source_cache: $spack/../restricted/ukmo/source_cache
    build_stage:
    - $TMPDIR/restricted/spack-stage
  modules:
    default:
      tcl:
        projections:
          access-esm1p5: '{name}/prX-Y'
          cice4: 'access-esm1p5/dependencies/prX-Y/{name}/access-esm1.5-2025.04.001-{hash:7}'
          mom5: 'access-esm1p5/dependencies/prX-Y/{name}/access-esm1.5-2025.03.002-{hash:7}'
          oasis3-mct: 'access-esm1p5/dependencies/prX-Y/{name}/access-esm1.5_2024.05.24-{hash:7}'
          openmpi: 'access-esm1p5/dependencies/prX-Y/{name}/4.0.2-{hash:7}'
          um7: 'access-esm1p5/dependencies/prX-Y/{name}/2024.10.17-{hash:7}'
        include:
        - access-esm1p5
        - cice4
        - mom5
        - oasis3-mct
        - openmpi
        - um7

[aidanheerdegen]

LGTM.

Just FYI I think I would have preferred not to [skip-ci] so I could check the module load of the pre-release, but should be fine to merge.