Skip to content

Commit 13008b5

Browse files
CodeGataidanheerdegen
CodeGat
and
aidanheerdegen
authored
v6 Changes Bundle (#306)
* Injection of Modules over Checks (#295) * Add projection injection module * Add requirements for injection modules * Add tests for injection projection module, and inputs/requirements * Add prerelease injection module * Add tests for prerelease injection module * Run injection.projection and injection.prerelease modules * deploy-1-setup: Remove projection checks * Remove unneeded mergedeep requirement * Add module inits to folders * Add step for installing dependencies, update names for checkout steps * Used cd module prefix rather than custom working-directory so we can find the manifest * Added required input for prerelease modification, always print out updated manifest at end * Add back EOT * Update non-root-spec projections to namespaced `{name}/prX-Y/VERSION` format * Updated tests for namespaced non-root-spec projections * Update prerelease module argument from --root-spec-version to --version * Pass in sys.argv to parse_args, add tests * Refactor workflow to split generation of root spec and packages * Rename projections script to modules * Add inject_includes function * For test inputs, comment what is expected * Reordered functions * Updated tests after code updates * Apply suggestions from code review Co-authored-by: Aidan Heerdegen <aidan.heerdegen@anu.edu.au> Signed-off-by: Tommy Gatti <tommy.gatti@outlook.com> * Add root_spec to includes set, but remove it later when alphabetically sorting the includelist since it always comes first * Updated output of manifest * Ran black * Added getter module for accessing various parts of a spack manifest * Update `[test_]modules.py` to use new getter module * Update `[test_]prerelease.py` to use getter module, update deploy-2-start to remove --root-spec arg * Ran black again * Raise NotImplementedErrors, add comments for successful matches * Remove comments of old code * Apply suggestions from code review Co-authored-by: Aidan Heerdegen <aidan.heerdegen@anu.edu.au> Signed-off-by: Tommy Gatti <tommy.gatti@outlook.com> * Add example to multi-target spec to test function * Parameterized tests * Fixes following more test cases * Converted manifest with no packages to fixture * Improved from_file classmethod tests * Parameterized valid and mistakenly invalid tests * Remove `root-spec` arg and instead get from manifest directly, update tests * Add new prerelease script flag `--keep-root-spec-intact`, that doesn't remove version information --------- Signed-off-by: Tommy Gatti <tommy.gatti@outlook.com> Co-authored-by: Aidan Heerdegen <aidan.heerdegen@anu.edu.au> * `vars` to File-Based Special Packages (#305) * Validate MDR config/packages.json * Pass info from MDR config/packages.json to module injection script * Update variable name to `vars.CONFIG_PACKAGES_SCHEMA_VERSION` * Change cwd relative to build-cd rather than model for scripts * Add output of packages to be injected/added to provenance DB * Update prerelease projection to `ROOT_SPEC/prX-Y/{name}/VERSION` * Update tests for new prerelease injection logic * Replace all internal refs with v6 * Dump quoted strings for prerelease projections, updated expected output * Schema Versions as Entrypoint Inputs Over `vars` (#310) * Change schema vars to required inputs, add optional spack manifest schema path for potentially different schemas * Add required schema version inputs, and optional spack manifest schema paths to entrypoint workflows * Get packages for provenance from an earlier step * Update prerelease injection to the form `ROOT_SPEC/.dependencies/PRX-Y/VERSION-{hash:7}` * Fixed up an unused variable * Remove `.` from `dependencies directory --------- Signed-off-by: Tommy Gatti <tommy.gatti@outlook.com> Co-authored-by: Aidan Heerdegen <aidan.heerdegen@anu.edu.au>
1 parent 8a9762d commit 13008b5

21 files changed

Lines changed: 1977 additions & 109 deletions

‎.github/workflows/cd.yml‎

Lines changed: 29 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,27 @@ on:
3838
description: |
3939
Whether to upload the deployment information to the build database (requires tag-deployment to also be true).
4040
NOTE: Setting this to false is not recommended for model deployments.
41+
spack-manifest-schema-path:
42+
type: string
43+
required: false
44+
default: au.org.access-nri/model/spack/environment/deployment
45+
description: |
46+
The path to the Spack manifest schema directory relative to the ACCESS-NRI/schema repository.
47+
spack-manifest-schema-version:
48+
type: string
49+
required: true
50+
description: |
51+
The version of the Spack manifest schema to use for validation, from the ACCESS-NRI/schema repository.
52+
config-versions-schema-version:
53+
type: string
54+
required: true
55+
description: |
56+
The version of the config/versions.json schema to use for validation, from the ACCESS-NRI/schema repository.
57+
config-packages-schema-version:
58+
type: string
59+
required: true
60+
description: |
61+
The version of the config/packages.json schema to use for validation, from the ACCESS-NRI/schema repository.
4162
# Callers usually have the trigger:
4263
# push:
4364
# branches:
@@ -70,7 +91,7 @@ jobs:
7091
7192
- name: Generate Deployment Target Matrix
7293
id: target
73-
uses: access-nri/build-cd/.github/actions/get-target-matrix@v5
94+
uses: access-nri/build-cd/.github/actions/get-target-matrix@v6
7495
with:
7596
targets: ${{ vars.RELEASE_DEPLOYMENT_TARGETS }}
7697

@@ -87,7 +108,7 @@ jobs:
87108
with:
88109
repository: access-nri/build-cd
89110

90-
- uses: access-nri/build-cd/.github/actions/validate-deployment-settings@v5
111+
- uses: access-nri/build-cd/.github/actions/validate-deployment-settings@v6
91112
with:
92113
settings-path: ./config/settings.json
93114
target: ${{ matrix.target }}
@@ -105,7 +126,7 @@ jobs:
105126

106127
- name: Get root spec ref
107128
id: tag
108-
uses: access-nri/build-cd/.github/actions/get-spack-root-spec@v5
129+
uses: access-nri/build-cd/.github/actions/get-spack-root-spec@v6
109130
with:
110131
spack-manifest-path: ${{ inputs.spack-manifest-path }}
111132

@@ -159,14 +180,18 @@ jobs:
159180
strategy:
160181
matrix:
161182
target: ${{ fromJson(needs.defaults.outputs.targets) }}
162-
uses: access-nri/build-cd/.github/workflows/deploy-1-setup.yml@v5
183+
uses: access-nri/build-cd/.github/workflows/deploy-1-setup.yml@v6
163184
with:
164185
deployment-target: ${{ matrix.target }}
165186
deployment-ref: ${{ github.ref_name }}
166187
deployment-type: Release
167188
deployment-version: ${{ needs.get-root-spec-ref.outputs.root-spec-ref }}
168189
spack-manifest-path: ${{ inputs.spack-manifest-path }}
169190
expected-root-spec-name: ${{ needs.defaults.outputs.root-sbd }}
191+
spack-manifest-schema-path: ${{ inputs.spack-manifest-schema-path }}
192+
spack-manifest-schema-version: ${{ inputs.spack-manifest-schema-version }}
193+
config-versions-schema-version: ${{ inputs.config-versions-schema-version }}
194+
config-packages-schema-version: ${{ inputs.config-packages-schema-version }}
170195
secrets: inherit
171196
permissions:
172197
contents: write

‎.github/workflows/ci-closed.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ jobs:
3838

3939
- name: Generate Deployment Target Matrix
4040
id: target
41-
uses: access-nri/build-cd/.github/actions/get-target-matrix@v5
41+
uses: access-nri/build-cd/.github/actions/get-target-matrix@v6
4242
with:
4343
targets: ${{ vars.PRERELEASE_DEPLOYMENT_TARGETS }}
4444

@@ -50,7 +50,7 @@ jobs:
5050
matrix:
5151
target: ${{ fromJson(needs.setup.outputs.targets) }}
5252
fail-fast: false
53-
uses: access-nri/build-cd/.github/workflows/undeploy-1-start.yml@v5
53+
uses: access-nri/build-cd/.github/workflows/undeploy-1-start.yml@v6
5454
with:
5555
version-pattern: ${{ inputs.root-sbd }}-pr${{ github.event.pull_request.number }}-*
5656
target: ${{ matrix.target }}

‎.github/workflows/ci-comment.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,7 @@ jobs:
7373

7474
- name: Original version
7575
id: original
76-
uses: access-nri/build-cd/.github/actions/get-spack-root-spec@v5
76+
uses: access-nri/build-cd/.github/actions/get-spack-root-spec@v6
7777

7878
- name: Setup
7979
id: setup

‎.github/workflows/ci.yml‎

Lines changed: 28 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,27 @@ on:
3333
description: |
3434
The Spack manifest path relative to the caller repository that will be used for the deployment.
3535
This is usually `./spack.yaml`, but can be overridden if needed.
36+
spack-manifest-schema-path:
37+
type: string
38+
required: false
39+
default: au.org.access-nri/model/spack/environment/deployment
40+
description: |
41+
The path to the Spack manifest schema directory relative to the ACCESS-NRI/schema repository.
42+
spack-manifest-schema-version:
43+
type: string
44+
required: true
45+
description: |
46+
The version of the Spack manifest schema to use for validation, from the ACCESS-NRI/schema repository.
47+
config-versions-schema-version:
48+
type: string
49+
required: true
50+
description: |
51+
The version of the config/versions.json schema to use for validation, from the ACCESS-NRI/schema repository.
52+
config-packages-schema-version:
53+
type: string
54+
required: true
55+
description: |
56+
The version of the config/packages.json schema to use for validation, from the ACCESS-NRI/schema repository.
3657
# Callers usually have the trigger:
3758
# pull_request:
3859
# branches:
@@ -132,7 +153,7 @@ jobs:
132153
133154
- name: Generate Deployment Target Matrix
134155
id: target
135-
uses: access-nri/build-cd/.github/actions/get-target-matrix@v5
156+
uses: access-nri/build-cd/.github/actions/get-target-matrix@v6
136157
with:
137158
targets: ${{ vars.PRERELEASE_DEPLOYMENT_TARGETS }}
138159

@@ -196,7 +217,7 @@ jobs:
196217
matrix:
197218
# Example: ['Gadi', 'Setonix', ...]
198219
target: ${{ fromJson(needs.defaults.outputs.targets) }}
199-
uses: access-nri/build-cd/.github/workflows/deploy-1-setup.yml@v5
220+
uses: access-nri/build-cd/.github/workflows/deploy-1-setup.yml@v6
200221
with:
201222
deployment-target: ${{ matrix.target }}
202223
deployment-ref: ${{ needs.defaults.outputs.head-ref }}
@@ -208,6 +229,11 @@ jobs:
208229
prerelease-compare-ref: ${{ needs.defaults.outputs.base-ref }}
209230
spack-manifest-path: ${{ inputs.spack-manifest-path }}
210231
expected-root-spec-name: ${{ needs.defaults.outputs.root-sbd }}
232+
# Pass versions and paths for schemas into the deploy workflow
233+
spack-manifest-schema-path: ${{ inputs.spack-manifest-schema-path }}
234+
spack-manifest-schema-version: ${{ inputs.spack-manifest-schema-version }}
235+
config-versions-schema-version: ${{ inputs.config-versions-schema-version }}
236+
config-packages-schema-version: ${{ inputs.config-packages-schema-version }}
211237
secrets: inherit
212238

213239
redeploy-post:

‎.github/workflows/deploy-1-setup.yml‎

Lines changed: 36 additions & 67 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,26 @@ on:
4444
description: |
4545
The expected overarching spack bundle that contains all other packages, set in the caller repository.
4646
Usually .spack.specs[0] or .spack.definitions.ROOT_PACKAGE in the manifest, but draft PRs can override this.
47+
spack-manifest-schema-path:
48+
type: string
49+
required: true
50+
description: |
51+
The path to the Spack manifest schema directory relative to the ACCESS-NRI/schema repository.
52+
spack-manifest-schema-version:
53+
type: string
54+
required: true
55+
description: |
56+
The version of the Spack manifest schema to use for validation, from the ACCESS-NRI/schema repository.
57+
config-versions-schema-version:
58+
type: string
59+
required: true
60+
description: |
61+
The version of the config/versions.json schema to use for validation, from the ACCESS-NRI/schema repository.
62+
config-packages-schema-version:
63+
type: string
64+
required: true
65+
description: |
66+
The version of the config/packages.json schema to use for validation, from the ACCESS-NRI/schema repository.
4767
outputs:
4868
general-metadata-artifact-glob:
4969
value: ${{ jobs.deployment.outputs.general-metadata-artifact-glob }}
@@ -144,20 +164,27 @@ jobs:
144164
- name: Validate ${{ github.repository }} config/versions.json
145165
uses: access-nri/schema/.github/actions/validate-with-schema@main
146166
with:
147-
schema-version: ${{ vars.CONFIG_VERSIONS_SCHEMA_VERSION }}
167+
schema-version: ${{ inputs.config-versions-schema-version }}
148168
schema-location: au.org.access-nri/model/deployment/config/versions
149169
data-location: ./model/config/versions.json
150170

171+
- name: Validate ${{ github.repository }} config/packages.json
172+
uses: access-nri/schema/.github/actions/validate-with-schema@main
173+
with:
174+
schema-version: ${{ inputs.config-packages-schema-version }}
175+
schema-location: au.org.access-nri/model/deployment/config/packages
176+
data-location: ./model/config/packages.json
177+
151178
- name: Validate spack-packages version
152179
id: spack-packages
153-
uses: access-nri/build-cd/.github/actions/validate-repo-version@v5
180+
uses: access-nri/build-cd/.github/actions/validate-repo-version@v6
154181
with:
155182
repo-to-check: spack-packages
156183
pr: ${{ inputs.deployment-ref }}
157184

158185
- name: Validate spack version
159186
id: spack
160-
uses: access-nri/build-cd/.github/actions/validate-repo-version@v5
187+
uses: access-nri/build-cd/.github/actions/validate-repo-version@v6
161188
with:
162189
repo-to-check: spack
163190
pr: ${{ inputs.deployment-ref }}
@@ -188,7 +215,7 @@ jobs:
188215
189216
- name: Validate build-cd config/settings.json
190217
id: settings
191-
uses: access-nri/build-cd/.github/actions/validate-deployment-settings@v5
218+
uses: access-nri/build-cd/.github/actions/validate-deployment-settings@v6
192219
with:
193220
settings-path: ./cd/config/settings.json
194221
target: ${{ inputs.deployment-target }}
@@ -217,13 +244,13 @@ jobs:
217244
if: env.IS_NOT_DRAFT == 'true' || inputs.deployment-type == 'Release'
218245
uses: access-nri/schema/.github/actions/validate-with-schema@main
219246
with:
220-
schema-version: ${{ vars.SPACK_YAML_SCHEMA_VERSION }}
221-
schema-location: au.org.access-nri/model/spack/environment/deployment
247+
schema-version: ${{ inputs.spack-manifest-schema-version }}
248+
schema-location: ${{ inputs.spack-manifest-schema-path }}
222249
data-location: ${{ inputs.spack-manifest-path }}
223250

224251
- name: Get current (${{ inputs.deployment-ref }}) root spec version
225252
id: current
226-
uses: access-nri/build-cd/.github/actions/get-spack-root-spec@v5
253+
uses: access-nri/build-cd/.github/actions/get-spack-root-spec@v6
227254
with:
228255
spack-manifest-path: ${{ inputs.spack-manifest-path }}
229256

@@ -262,7 +289,7 @@ jobs:
262289
- name: Get base root spec version
263290
id: base
264291
if: inputs.deployment-type != 'Release' && steps.checkout-base-spack.outcome != 'failure'
265-
uses: access-nri/build-cd/.github/actions/get-spack-root-spec@v5
292+
uses: access-nri/build-cd/.github/actions/get-spack-root-spec@v6
266293
with:
267294
spack-manifest-path: ${{ inputs.spack-manifest-path }}
268295

@@ -293,64 +320,6 @@ jobs:
293320
You can update it manually.
294321
${{ inputs.spack-manifest-path != './spack.yaml' && env.BUMP_COMMENT_BODY || '' }}
295322
296-
297-
- name: Check Projections - Specs
298-
# This step checks that the root spec has an appropriate projection in the manifest file.
299-
id: projection-check-spec
300-
continue-on-error: true
301-
run: |
302-
git checkout ${{ inputs.deployment-ref }}
303-
304-
current_projection=$(yq '.spack.modules.default.tcl.projections.${{ steps.current.outputs.root-spec-name }}' ${{ inputs.spack-manifest-path }})
305-
expected_projection_prefix='{name}/${{ steps.current.outputs.root-spec-ref }}'
306-
307-
if [[ "$current_projection" == "$expected_projection_prefix"* ]]; then
308-
echo "${{ inputs.spack-manifest-path }}: Current root spec projection ($current_projection) and expected projection prefix ($expected_projection_prefix) match."
309-
else
310-
echo "::error::${{ inputs.spack-manifest-path }}: Current root spec projection ($current_projection) and expected projection prefix ($expected_projection_prefix) don't match."
311-
exit 1
312-
fi
313-
314-
- name: Check Projections - Packages
315-
# This step checks that the defined projections have the same versions as the referenced packages in the spack.yaml file.
316-
# For example, mom5@git.2023.12.12 matches with the
317-
# modulefile mom5/2023.12.12 (specifically, the version strings match)
318-
id: projection-check-packages
319-
continue-on-error: true
320-
if: env.IS_NOT_DRAFT == 'true' || inputs.deployment-type == 'Release'
321-
run: |
322-
FAILED="false"
323-
# Get all the defined projections (minus 'all' and the root spec) and make them suitable for a bash for loop
324-
DEPS=$(yq '.spack.modules.default.tcl.projections | del(.all) | del(.${{ steps.current.outputs.root-spec-name }}) | keys | join(" ")' ${{ inputs.spack-manifest-path }})
325-
326-
# for each of the module names
327-
for DEP in $DEPS; do
328-
# Capture the section after '@git.' or '@' (if it's not a git-attributed version) and before a possible '=' for a given dependency.
329-
# Ex. '@git.2024.02.11' -> '2024.02.11', '@access-esm1.5' -> 'access-esm1.5', '@git.2024.05.21=access-esm1.5' -> '2024.05.21'
330-
package_ver=$(yq ".spack.packages.\"$DEP\".require[0] | match(\"^@(?:git.)?([^=]*)\").captures[0].string" ${{ inputs.spack-manifest-path }})
331-
expected_projection_prefix="{name}/$package_ver"
332-
333-
# Projections must start with '{name}/VERSION'. For example, '{name}/2024.11.11', or '{name}/2024.11.11-{hash:7}'
334-
current_projection=$(yq ".spack.modules.default.tcl.projections.\"$DEP\"" ${{ inputs.spack-manifest-path }})
335-
336-
if [[ "$current_projection" == "$expected_projection_prefix"* ]]; then
337-
echo "$DEP in ${{ inputs.spack-manifest-path }}: Current projection ($current_projection) and expected projection prefix ($expected_projection_prefix) match."
338-
else
339-
echo "::error::$DEP in ${{ inputs.spack-manifest-path }}: Current projection ($current_projection) and expected projection prefix ($expected_projection_prefix) don't match."
340-
FAILED='true'
341-
fi
342-
done
343-
if [[ "$FAILED" == "true" ]]; then
344-
exit 1
345-
fi
346-
347-
- name: Check Projections - Error Notifier
348-
# If either of the Check Projection steps failed earlier, exit here
349-
if: steps.projection-check-spec.outcome == 'failure' || steps.projection-check-packages.outcome == 'failure'
350-
run: |
351-
echo "::error::Some projections in ${{ inputs.spack-manifest-path }} don't match."
352-
exit 1
353-
354323
- name: Set Spack Env Name String
355324
id: get-env-name
356325
# replace occurences of '.' with '_' in environment name as spack doesn't support '.'. Ex: 'access-om2-v1.0.0' -> 'access-om2-v1_0_0'.
@@ -361,7 +330,7 @@ jobs:
361330
needs:
362331
- check-config # Verify configuration information is correct
363332
- check-spack-yaml # Verify spack manifest information is correct
364-
uses: access-nri/build-cd/.github/workflows/deploy-2-start.yml@v5
333+
uses: access-nri/build-cd/.github/workflows/deploy-2-start.yml@v6
365334
with:
366335
ref: ${{ inputs.deployment-ref }}
367336
version: ${{ inputs.deployment-version }}

0 commit comments

Comments
 (0)
⚡