Skip to content

Commit 7dc203b

Browse files
CodeGataidanheerdegen
CodeGat
and
aidanheerdegen
authored
build-ci v3: Infrastructure Update for spack v1.X (#253)
* REMOVED inputs.spack-packages-ref, added inputs.builtin-spack-packages-ref, inputs.access-spack-packages-ref. Similar for outputs.*spack-packages-sha * Updated image for spack > v1, now uses spack repo update command --------- Co-authored-by: Aidan Heerdegen <aidan.heerdegen@gmail.com>
1 parent 6d4a94a commit 7dc203b

16 files changed

Lines changed: 499 additions & 158 deletions

File tree

‎.github/actions/spack-checkout-updated-ref/README.md‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
# Update Existing Repo and Checkout Ref
2+
3+
Action that updates an existing repository, and checks out the updated ref.
4+
5+
## Inputs
6+
7+
| Name | Type | Description | Required | Default | Example |
8+
| ---- | ---- | ----------- | -------- | ------- | ------- |
9+
| `spack-packages-repository-name` | `string` | The name of the repository used by spack to update and checkout the ref (given in the spack config file `repos.yaml`) | `true` | N/A | `"builtin"` |
10+
| `spack-packages-repository-path` | `string` (path) | The path to the repository to update and check out the ref | `true` | N/A | `"/root/.spack/package_repos/fncqgg4/repos/spack_repo/builtin"` |
11+
| `ref` | `string` (git branch, tag or sha) | The git ref to check out | `true` | N/A | `"main"` or `"v1"` or `"f8r73g3"` |
12+
| `spack-instance-root-path` | `string` (path) | The path to the spack instance root, used to setup the spack environment | `true` | N/A | `"/opt/spack"` |
13+
An example [`repos.yaml` file](https://github.com/ACCESS-NRI/spack-config/blob/main/common-api-v2/repos.yaml) as referenced above.
14+
## Outputs
15+
16+
| Name | Type | Description | Example |
17+
| ---- | ---- | ----------- | ------- |
18+
| `sha` | `string` (sha) | The SHA of the checked out ref | `"5a1cdc4e4617fcd6ba1cccf1cd0432b5631983be"` |
19+
| `updated` | `string` (boolean) | Whether there was actually an update to the ref | `"true"` or `"false"` |
20+
21+
## Examples
22+
23+
### Simple
24+
25+
```yaml
26+
# ...
27+
jobs:
28+
update-repo:
29+
runs-on: ubuntu-latest
30+
env:
31+
SPACK_ROOT: /opt/spack
32+
steps:
33+
- id: repo
34+
run: |
35+
. ${{ env.SPACK_ROOT }}/share/spack/setup-env.sh
36+
echo "path=$(spack location --repo builtin)" >> $GITHUB_OUTPUT
37+
38+
- id: update
39+
uses: ./.github/actions/spack-checkout-updated-ref
40+
with:
41+
spack-packages-repository-name: builtin
42+
spack-packages-repository-path: ${{ steps.repo.outputs.path }}
43+
ref: develop
44+
spack-instance-root-path: ${{ env.SPACK_ROOT }}
45+
46+
- run: echo "The builtin spack-packages repo was updated to ${{ steps.update.outputs.sha }}"
47+
```

‎.github/actions/spack-checkout-updated-ref/action.yml‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
name: Checkout Ref Via Spack
2+
description: Updates and checks out a given ref for an existing spack-packages repository via spack
3+
inputs:
4+
spack-packages-repository-name:
5+
description: |
6+
The spack name of the repository to update and check out the ref.
7+
This is the repos.NAME key in spacks repos.yaml config file - eg. builtin
8+
required: true
9+
spack-packages-repository-path:
10+
description: The path to the repository to update and check out the ref
11+
required: true
12+
ref:
13+
description: The git ref to check out
14+
required: true
15+
spack-instance-root-path:
16+
description: The path to the spack instance root, used to setup the spack environment
17+
required: true
18+
outputs:
19+
sha:
20+
description: 'The SHA of the checked out ref'
21+
value: ${{ steps.ref-sha.outputs.sha }}
22+
updated:
23+
description: 'Whether the repository was updated'
24+
value: ${{ steps.post-update.outputs.updated }}
25+
runs:
26+
using: composite
27+
steps:
28+
- name: Get initial SHA
29+
id: initial
30+
shell: bash
31+
run: |
32+
sha=$(git -C ${{ inputs.spack-packages-repository-path }} rev-parse HEAD^{})
33+
echo "${{ inputs.spack-packages-repository-name }} Initial SHA: $sha"
34+
35+
echo "sha=$sha" >> $GITHUB_OUTPUT
36+
37+
- name: Fetch latest for repo
38+
shell: bash
39+
run: git -C ${{ inputs.spack-packages-repository-path }} fetch --force --tags
40+
41+
- name: Get SHA for ref
42+
id: ref-sha
43+
uses: access-nri/actions/.github/actions/get-git-ref-info@main
44+
with:
45+
repository-path: ${{ inputs.spack-packages-repository-path }}
46+
ref: ${{ inputs.ref }}
47+
48+
- name: Update ref via Spack
49+
shell: bash
50+
id: update
51+
continue-on-error: true
52+
run: |
53+
. ${{ inputs.spack-instance-root-path }}/share/spack/setup-env.sh
54+
spack repo update ${{ inputs.spack-packages-repository-name }} --commit ${{ steps.ref-sha.outputs.sha }}
55+
56+
- name: Force Update ref via Git if Failure
57+
if: steps.update.outcome == 'failure'
58+
shell: bash
59+
# FIXME: If there is ever a spack repo update --force option, use that for the above command and delete this step
60+
run: git -C ${{ inputs.spack-packages-repository-path }} checkout --force ${{ steps.ref-sha.outputs.sha }}
61+
62+
- name: Check if updated
63+
id: post-update
64+
shell: bash
65+
run: |
66+
if [ "${{ steps.initial.outputs.sha }}" != "${{ steps.ref-sha.outputs.sha }}" ]; then
67+
echo "Repository was updated from ${{ steps.initial.outputs.sha }} to ${{ inputs.ref }} (${{ steps.ref-sha.outputs.sha }})"
68+
echo "updated=true" >> $GITHUB_OUTPUT
69+
else
70+
echo "Repository was not updated, stayed at ${{ inputs.ref }} (${{ steps.initial.outputs.sha }})"
71+
echo "updated=false" >> $GITHUB_OUTPUT
72+
fi

‎.github/workflows/README.md‎

Lines changed: 16 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -14,9 +14,10 @@ This workflow handles building and running short CI tests on a given spack manif
1414
| `spack-manifest-data-path`| `string` (Path relative to component repository root) | File path in the caller model component repository that contains jinja data to fill in to the spack manifest jinja template given by `inputs.spack-manifest-path`. This doesn't include the pull request ref (`{{ pr }}`), which is filled in automatically | `false` | N/A | `".github/build/data/template-data.json"` |
1515
| `spack-compiler-manifest-path` | `string` (Path relative to component repository root) | A file path in the caller model component repository that contains the spack manifest to install local compilers not in the upstream | `false` | N/A | `".github/build/compilers/intel-2021.11.0.spack.yml"` |
1616
| `spack-manifest-data-pairs` | `string` | An optional, multi-line string of space-separated key-value pairs to fill in `inputs.spack-manifest-path`. This is useful for filling in template values created dynamically by earlier jobs needed by this workflow. This doesn't include `{{ ref }}`, which is filled in automatically. | `false` | N/A | `"package mom5`(newline)`compiler intel"` |
17-
| `ref` | `string` (Git ref) | The branch, tag, or commit SHA of the caller model component repository | `false` | `github.event.pull_request.head.sha` for PRs, `github.sha` otherwise | `"02125b01eb7c778c8d0ae0a02a260de474782e81"`, `"main"`, `"2025.01.000"` |
17+
| `ref` | `string` (Git ref) | The branch, tag, or commit SHA of the caller model component repository | `false` | `github.event.pull_request.head.sha` for PRs, `github.sha` otherwise | `"c0fef23fc1e69d3a31ec18fd8b7102acdf95f651"`, `"main"`, `"2025.01.000"` |
1818
| `spack-config-ref` | `string` (Git ref) | The branch, tag, or commit SHA of the access-nri/spack-config repository to use | `false` | `"main"` | `"02125b01eb7c778c8d0ae0a02a260de474782e81"`, `"main"`, `"2025.01.000"` |
19-
| `spack-packages-ref` | `string` (Git ref) | The branch, tag, or commit SHA of the access-nri/spack-packages repository to use | `false` | `"main"` | `"02125b01eb7c778c8d0ae0a02a260de474782e81"`, `"main"`, `"2025.01.000"` |
19+
| `builtin-spack-packages-ref` | `string` (Git ref) | The branch, tag, or commit SHA of the `spack/spack-packages` repository to use | `false` | `"main"` | `"f7314790111ec43cf9cff60421c155b922c349ad"`, `"main"`, `"2025.01.000"` |
20+
| `access-spack-packages-ref` | `string` (Git ref) | The branch, tag, or commit SHA of the `access-nri/access-spack-packages` repository to use | `false` | `"main"` | `"e4ba85db0be4a9b9493cf7581623f9997b9404a5"`, `"main"`, `"2025.01.000"` |
2021
| `allow-ssh-into-spack-install` | `boolean` | Enable the actor of the workflow to SSH into the container where the spack packages have been installed. This is useful for gathering post-install information before the container is destroyed. This will also make the workflow wait until the actor SSHs into the container, or it times out, before continuing | `false` | `false` | `true`, `false` |
2122
| `container-image-version` | `string` (Docker version ref) | The version of the container image to use for the runner. Can be either a `:TAG` or a `@sha256:SHA`. | `false` | `":rocky"` | `':8.9'` (tag), `'@sha256:1234...'` (SHA) |
2223
| `spack-oci-buildcache-url` | `string` (OCI URL) | The URL to an oci-backed buildcache, available in spack >= v1.0. OCI-backed buildcaches are the only option for GitHub-hosted CI, and can be used as a backup for self-hosted CI's runner buildcache | `false` | N/A | `"oci://ghcr.io/ACCESS-NRI/build-ci-buildcache"`, `"oci://ghcr.io/ORG/IMAGE"` |
@@ -43,9 +44,10 @@ This workflow handles building and running short CI tests on a given spack manif
4344
| ---- | ---- | ----------- | ------- |
4445
| `spec-concretization-graph` | `string` (multiline) | A visual representation of the dependencies and constraints of the spack manifest file installed | N/A |
4546
| `spack-sha` | `string` (Git commit SHA) | The SHA of the `ACCESS-NRI/spack` repository checked out | `"02125b01eb7c778c8d0ae0a02a260de474782e81"` |
46-
| `spack-config-sha` | `string` (Git commit SHA) | The SHA of the `ACCESS-NRI/spack-config` repository checked out | `"02125b01eb7c778c8d0ae0a02a260de474782e81"` |
47-
| `spack-packages-sha` | `string` (Git commit SHA) | The SHA of the `ACCESS-NRI/spack-packages` repository checked out | `"02125b01eb7c778c8d0ae0a02a260de474782e81"` |
48-
| `sha` | `string` (Git commit SHA) | The SHA of the caller model component repository checked out | `"02125b01eb7c778c8d0ae0a02a260de474782e81"` |
47+
| `spack-config-sha` | `string` (Git commit SHA) | The SHA of the `ACCESS-NRI/spack-config` repository checked out | `"c0fef23fc1e69d3a31ec18fd8b7102acdf95f651"` |
48+
| `builtin-spack-packages-sha` | `string` (Git commit SHA) | The SHA of the `spack/spack-packages` repository checked out | `"9225ee95da5c6e212a80d933db8aca44271417a3"` |
49+
| `access-spack-packages-sha` | `string` (Git commit SHA) | The SHA of the `ACCESS-NRI/access-spack-packages` repository checked out | `"2acb57187fc75c0fc83c898d1dd47bdaced2fca9"` |
50+
| `sha` | `string` (Git commit SHA) | The SHA of the caller model component repository checked out | `"43ef5da423028a9a25133884e6e402900e87a3ce"` |
4951
| `container-id` | `string` | The ID of the container where the spack packages have been installed | `"ohfn2ofy2h2uyfg2uyg3uyg3uh"` |
5052
| `spack-files-artifact-pattern` | `string` (glob) | Wildcard pattern to match all spack file artifacts across a matrix job | `'spack-files-*'` |
5153
| `spack-files-artifact-url` | `string` (URL) | The URL of the spack manifest and lock files artifact | `"https://github.com/ACCESS-NRI/MOM5/actions/runs/15890554355/artifacts/3406449135"` |
@@ -67,7 +69,7 @@ This workflow handles building and running short CI tests on a given spack manif
6769
```yaml
6870
jobs:
6971
test:
70-
uses: access-nri/build-ci/.github/workflows/ci.yml@v2
72+
uses: access-nri/build-ci/.github/workflows/ci.yml@v3
7173
with:
7274
spack-manifest-path: .github/build/spack.yaml.j2
7375
```
@@ -77,13 +79,14 @@ jobs:
7779
```yaml
7880
jobs:
7981
test:
80-
uses: access-nri/build-ci/.github/workflows/ci.yml@v2
82+
uses: access-nri/build-ci/.github/workflows/ci.yml@v3
8183
with:
8284
spack-manifest-path: .github/build/spack.yaml.j2
8385
spack-manifest-data-path: .github/build/data/data.json
8486
spack-compiler-manifest-path: .github/build/compiler/intel.spack.yaml
8587
spack-ref: releases/v0.22
86-
spack-packages-ref: 2025.05.000
88+
builtin-spack-packages-ref: 2025.07.0
89+
access-spack-packages-ref: 2025.05.000
8790
spack-config-ref: 2025.10.001
8891
allow-ssh-into-spack-install: true
8992
secrets:
@@ -106,7 +109,7 @@ jobs:
106109
- .github/build/one.spack.yaml.j2
107110
- .github/build/two.spack.yaml.j2
108111
- .github/build/three.spack.yaml.j2
109-
uses: access-nri/build-ci/.github/workflows/ci.yml@v2
112+
uses: access-nri/build-ci/.github/workflows/ci.yml@v3
110113
with:
111114
spack-manifest-path: ${{ matrix.manifest }}
112115
```
@@ -126,7 +129,7 @@ jobs:
126129
compiler: .github/build/compiler/intel.spack.yaml
127130
- manifest: .github/build/two.spack.yaml.j2
128131
compiler: .github/build/compiler/gcc.spack.yaml
129-
uses: access-nri/build-ci/.github/workflows/ci.yml@v2
132+
uses: access-nri/build-ci/.github/workflows/ci.yml@v3
130133
with:
131134
spack-manifest-path: ${{ matrix.values.manifest }}
132135
spack-compiler-manifest-path: ${{ matrix.values.compiler }}
@@ -144,7 +147,7 @@ jobs:
144147
# This would be a combination of all defined manifest/compilers (eg, 4 jobs)
145148
manifest: [".github/build/one.spack.yaml.j2", ".github/build/two.spack.yaml.j2"]
146149
compiler: [".github/build/compiler/intel.spack.yaml", ".github/build/compiler/gcc.spack.yaml"]
147-
uses: access-nri/build-ci/.github/workflows/ci.yml@v2
150+
uses: access-nri/build-ci/.github/workflows/ci.yml@v3
148151
with:
149152
spack-manifest-path: ${{ matrix.values.manifest }}
150153
spack-compiler-manifest-path: ${{ matrix.values.compiler }}
@@ -174,7 +177,7 @@ The jinja data file (and the jinja-templatable spack manifest) can be much more
174177
Alternatively, you can supply a newline-separated list of space-separated template-value pairs through `inputs.spack-manifest-data-pairs`, which are more useful if you are supplying data to this workflow through `need`ed job outputs. For example:
175178

176179
```yaml
177-
uses: access-nri/build-ci/.github/workflows/ci.yml@v2
180+
uses: access-nri/build-ci/.github/workflows/ci.yml@v3
178181
with:
179182
spack-manifest-path: .github/build-ci/manifests/spack.yaml.j2
180183
spack-manifest-data-pairs: |-
@@ -229,7 +232,7 @@ jobs:
229232
file:
230233
- .github/build-ci/manifests/some.spack.yaml.j2
231234
- .github/build-ci/manifests/another.spack.yaml.j2
232-
uses: access-nri/build-ci/.github/workflows/ci.yaml@v2
235+
uses: access-nri/build-ci/.github/workflows/ci.yaml@v3
233236
with:
234237
spack-manifest-path: ${{ matrix.file }}
235238

0 commit comments

Comments
 (0)
⚡