Infra Update `v8`: `spack v1` Support by CodeGat · Pull Request #46 · ACCESS-NRI/ACCESS-ESM1.5

CodeGat · 2025-11-18T03:06:02Z

References issue ACCESS-NRI/build-cd#313 and PR ACCESS-NRI/build-cd#326
References rollout issue ACCESS-NRI/build-cd#328
References project Spack `0.22` to `1.0` Migration

Note

This infra update breaks reproducibility with the latest release (https://github.com/ACCESS-NRI/ACCESS-ESM1.5/releases/tag/2024.12.1). It will be merged without a release, so future PRs will need to keep this in mind.

Important

This PR is a major update to the deployment infrastructure. See below for the prerequisites for this repository to be able to merge this PR.

Important

This major version change marks the end of major infrastructure updates for deployments to spack < 1.0.
If you want to deploy to instances of spack < 1.0, you must use build-cd < v8.
If you want to deploy to instances of spack >= 1.0, you must use build-cd >= v8.

Background

We are moving on up to spack v1! This update to spack contains many bug fixes, optimisations and new features that can be incorporated into build-cd.

This update also contains changes that make spack v0.X incompatible with this build-cd v8 update - the most prominent one being the splitting of spacks core codebase from it's builtin spack-packages repo. This means that for provenance, one to keep track of both builtin spack-packages and our own, renamed access-spack-packages. build-cd will centrally control the version of builtin spack-packages for a given instance, and config/versions.jsons spack-packages key is renamed to access-spack-packages.

As noted earlier, this major version of build-cd can only support spack v1, due to spack v1-specific commands in the infrastructure. If you still want to deploy to spack v0.X, you will need to change build-cd to v7, and update the config/versions.json file/spack.yaml.

Features

Support for spack v1: We're moving to a new, non-beta version of spack! It contains bug fixes, features and optimisations over the old version.

Prerequisites for Merging

Update build-cd entrypoints (this PR!)
Update config/versions.json with new inputs (also this PR!)
Validate compiler additions to the spack manifest (this PR, too!)
Test and Verify Model built with spack v1

🚀 The latest prerelease access-esm1p5/pr46-3 at 8a613a7 is here: #46 (comment) 🚀

…o 2-0-0

…-scopes rather than environment

…comment

…common api-v2 tag

github-actions · 2026-02-16T23:57:43Z

🚀 Attempted to deploy access-esm1p5 Prerelease pr46-1 with commit beddb0a

🖥️ `Gadi` Deployment ✔️

Usage Instructions

access-esm1.5, defined in spack.yaml, will be deployed to Gadi as:

2024.12.1 as a Release (when merged).
pr46-1 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-esm1p5/pr46-1

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/1.1/spack in the access-esm1p5-pr46-1 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

access-nri/spack on branch 1.1
spack/spack-packages version 39aeb9dbcc13b6a53e40c77b672c397e1f4c93ec
access-nri/access-spack-packages version 2026.02.002
access-nri/spack-config version 2026.02.001

If the above was not what was expected, commit changes to config/versions.json in this PR, or propose changes to build-cds config/settings.json.

CodeGat · 2026-02-17T00:03:17Z

!update-configs

github-actions · 2026-02-17T00:04:10Z

🔧 Opening Model Configuration PRs in ACCESS-NRI/access-esm1.5-configs

Configurations Requested

Configurations requested from profile default:

dev-preindustrial+concentrations

Pull Requests Opened

The following PRs were opened:

Auto update dev-preindustrial+concentrations to use access-esm1p5/pr46-1 access-esm1.5-configs#142

More details can be found in the workflow run: https://github.com/ACCESS-NRI/ACCESS-ESM1.5/actions/runs/22081185963

CodeGat · 2026-02-17T00:14:52Z

Given the above, this is NOT bitwise reproducible :/ ACCESS-NRI/access-esm1.5-configs#142 (comment)

github-actions · 2026-02-18T02:05:08Z

🚀 Attempted to deploy access-esm1p5 Prerelease pr46-2 with commit 42e43dc

🖥️ `Gadi` Deployment ✔️

Usage Instructions

access-esm1.5, defined in spack.yaml, will be deployed to Gadi as:

2024.12.1 as a Release (when merged).
pr46-2 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-esm1p5/pr46-2

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/1.1/spack in the access-esm1p5-pr46-2 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

access-nri/spack on branch 1.1
spack/spack-packages version 39aeb9dbcc13b6a53e40c77b672c397e1f4c93ec
access-nri/access-spack-packages version 2026.02.002
access-nri/spack-config version 2026.02.001

If the above was not what was expected, commit changes to config/versions.json in this PR, or propose changes to build-cds config/settings.json.

CodeGat · 2026-02-18T03:05:40Z

!update-configs

github-actions · 2026-02-18T03:06:33Z

🔧 Opening/Updating Model Configuration PRs in ACCESS-NRI/access-esm1.5-configs

Configurations Requested

Configurations requested from profile default:

dev-preindustrial+concentrations

Pull Requests Opened/Updated

The following PRs were opened/updated:

Auto update dev-preindustrial+concentrations to use access-esm1p5/pr46-2 access-esm1.5-configs#143

More details can be found in the workflow run: https://github.com/ACCESS-NRI/ACCESS-ESM1.5/actions/runs/22124845557

github-actions · 2026-03-03T01:28:30Z

🚀 Attempted to deploy access-esm1p5 Prerelease pr46-3 with commit 8a613a7

🖥️ `Gadi` Deployment ✔️

Usage Instructions

access-esm1.5, defined in spack.yaml, will be deployed to Gadi as:

2024.12.1 as a Release (when merged).
pr46-3 as a Prerelease (during this PR).

This Prerelease is accessible on Gadi using:

module use /g/data/vk83/prerelease/modules
module load access-esm1p5/pr46-3

When using the above modules, the binaries shall be on your $PATH.

For advanced users, this Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/1.1/spack in the access-esm1p5-pr46-3 environment.
Due to inode-saving measures, one will have to manually untar the environment metadata before environment activation with tar -xf .spack-env .spack-env.tar. It will require one to have write privileges.

Configuration Information

This Prerelease is deployed using:

access-nri/spack on branch 1.1
spack/spack-packages version 383e969358c951abe17623696083e6f862c4488e
access-nri/access-spack-packages version 2026.02.002
access-nri/spack-config version 2026.02.003

If the above was not what was expected, commit changes to config/versions.json in this PR, or propose changes to build-cds config/settings.json.

CodeGat · 2026-03-03T02:10:56Z

!update-configs

github-actions · 2026-03-03T02:11:55Z

🔧 Opening/Updating Model Configuration PRs in ACCESS-NRI/access-esm1.5-configs

Configurations Requested

Configurations requested from profile default:

dev-preindustrial+concentrations

Pull Requests Opened/Updated

The following PRs were opened/updated:

Auto update dev-preindustrial+concentrations to use access-esm1p5/pr46-2 access-esm1.5-configs#143

More details can be found in the workflow run: https://github.com/ACCESS-NRI/ACCESS-ESM1.5/actions/runs/22605084210

harshula · 2026-03-04T00:38:39Z

Docker container (not on Gadi)

Spack: 0.22.5 (422a81b3d75f3f119e7c06234418763803a4f070)
Python: 3.6.8
Platform: linux-rocky8-zen5
Concretizer: clingo

commit 9da88099e0f0c63dc415451cfdd1c1ba151a55bf (HEAD -> main, origin/main, origin/HEAD)
Author: Tommy Gatti <tommy.gatti@anu.edu.au>
Date:   Thu Jan 8 16:16:59 2026 +1100

    [no ci] Infra Update `v7`: `!update-configs` Comment Command (#41)
    
    * [no ci] infra: Update to v7
    
    * [no ci] Fix up ci-command.yml
    
    * [no ci] Update auto-configs-pr.json based on feedback

[ACCESS-ESM1.5] [root@10d211b91e6f ACCESS-ESM1.5]# spack -V
0.22.5 (422a81b3d75f3f119e7c06234418763803a4f070)

[ACCESS-ESM1.5] [root@10d211b91e6f ACCESS-ESM1.5]# diff -u spack-cc-netcdf-c-hbwkaat.in.log spack-cc-netcdf-c-hbwkaat.out.log | grep bytebuffer

-[cc] /opt/release/linux-rocky8-x86_64/gcc-8.5.0/intel-oneapi-compilers-2023.2.4-hxxociow5cqbgxig7jtdtiwpg6tst7bs/compiler/2023.2.4/linux/bin/intel64/icc -DHAVE_CONFIG_H -I. -I.. -I../include -MT bytebuffer.o -MD -MP -MF .deps/bytebuffer.Tpo -c -o bytebuffer.o bytebuffer.c -I/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/openmpi-4.0.2-btwtzezafpl5gc3xsiqkxtvpbowtnbsk/include -pthread
-[ccld] /opt/release/linux-rocky8-x86_64/gcc-8.5.0/intel-oneapi-compilers-2023.2.4-hxxociow5cqbgxig7jtdtiwpg6tst7bs/compiler/2023.2.4/linux/bin/intel64/icc -o .libs/ncgen generate.o main.o cdata.o bindata.o genchar.o cvt.o data.o debug.o escapes.o genc.o genbin.o generr.o genlib.o getfill.o odom.o semantics.o dump.o util.o bytebuffer.o list.o genf77.o f77data.o genj.o jdata.o nc_iter.o ncgeny.o -L/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/hdf5-1.10.11-t2e7pgbnglunryddu3hpktwom47sy6qq/lib ../liblib/.libs/libnetcdf.so -lhdf5_hl -lhdf5 -lm -Wl,-rpath -Wl,/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/netcdf-c-4.7.4-hbwkaatp5grgjv2adk5eelh7crhqhskn/lib -I/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/openmpi-4.0.2-btwtzezafpl5gc3xsiqkxtvpbowtnbsk/include -pthread -L/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/hwloc-2.9.1-p6soq54v4rppjzrqmfculeuziisjgvaa/lib -L/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/libevent-2.1.12-5oqymlhwmyjk6d6o7uukwgai4drw5id3/lib -Wl,-rpath,/opt/release/linux-rocky8-x86_64/gcc-8.5.0/intel-oneapi-compilers-2023.2.4-hxxociow5cqbgxig7jtdtiwpg6tst7bs/compiler/2023.2.4/linux/compiler/lib/intel64_lin -Wl,-rpath -Wl,/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/hwloc-2.9.1-p6soq54v4rppjzrqmfculeuziisjgvaa/lib -Wl,-rpath -Wl,/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/libevent-2.1.12-5oqymlhwmyjk6d6o7uukwgai4drw5id3/lib -Wl,-rpath -Wl,/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/openmpi-4.0.2-btwtzezafpl5gc3xsiqkxtvpbowtnbsk/lib -L/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/openmpi-4.0.2-btwtzezafpl5gc3xsiqkxtvpbowtnbsk/lib -lmpi
+[cc] /opt/release/linux-rocky8-x86_64/gcc-8.5.0/intel-oneapi-compilers-2023.2.4-hxxociow5cqbgxig7jtdtiwpg6tst7bs/compiler/2023.2.4/linux/bin/intel64/icc -march=skylake-avx512 -mtune=skylake-avx512 -I. -I.. -I../include -I/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/openmpi-4.0.2-btwtzezafpl5gc3xsiqkxtvpbowtnbsk/include -I/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/hdf5-1.10.11-t2e7pgbnglunryddu3hpktwom47sy6qq/include -I/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/openmpi-4.0.2-btwtzezafpl5gc3xsiqkxtvpbowtnbsk/include -I/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/pmix-4.2.2-hdv4t6rbpyop4yh63vgr272cn2zb4hz5/include -I/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/numactl-2.0.14-3grq4d56qzmoq6lx6xmk3bt4njhf34tm/include -I/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/libevent-2.1.12-5oqymlhwmyjk6d6o7uukwgai4drw5id3/include -I/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/hwloc-2.9.1-p6soq54v4rppjzrqmfculeuziisjgvaa/include -I/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/libxml2-2.10.3-qbqc235pet2nfutg77cusifdtakrdytx/include/libxml2 -I/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/libxml2-2.10.3-qbqc235pet2nfutg77cusifdtakrdytx/include -I/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/xz-5.4.6-ptr4q3rditbzjwvmcgdpbqj36xab7gxw/include -I/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/libpciaccess-0.17-np6wwn6o2q2auw3e4vtuibyozyaqct44/include -I/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/ncurses-6.5-kj63bsmkj5yqshntpucmoaok24ar2m4f/include -I/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/openssl-3.3.0-4kl2j7rpk264t33ziw5no64kgaqvgl53/include -I/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/zlib-ng-2.1.6-ssevr4js2voelffrhd4fpazsfl3cel2x/include -I/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/libiconv-1.17-kgtp2nmv464qesavdvr6ydgxpuzovzz2/include -DHAVE_CONFIG_H -MT bytebuffer.o -MD -MP -MF .deps/bytebuffer.Tpo -c -o bytebuffer.o bytebuffer.c -pthread -fPIC -O2
+[ccld] /opt/release/linux-rocky8-x86_64/gcc-8.5.0/intel-oneapi-compilers-2023.2.4-hxxociow5cqbgxig7jtdtiwpg6tst7bs/compiler/2023.2.4/linux/bin/intel64/icc -march=skylake-avx512 -mtune=skylake-avx512 -I/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/openmpi-4.0.2-btwtzezafpl5gc3xsiqkxtvpbowtnbsk/include -I/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/hdf5-1.10.11-t2e7pgbnglunryddu3hpktwom47sy6qq/include -I/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/openmpi-4.0.2-btwtzezafpl5gc3xsiqkxtvpbowtnbsk/include -I/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/pmix-4.2.2-hdv4t6rbpyop4yh63vgr272cn2zb4hz5/include -I/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/numactl-2.0.14-3grq4d56qzmoq6lx6xmk3bt4njhf34tm/include -I/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/libevent-2.1.12-5oqymlhwmyjk6d6o7uukwgai4drw5id3/include -I/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/hwloc-2.9.1-p6soq54v4rppjzrqmfculeuziisjgvaa/include -I/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/libxml2-2.10.3-qbqc235pet2nfutg77cusifdtakrdytx/include/libxml2 -I/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/libxml2-2.10.3-qbqc235pet2nfutg77cusifdtakrdytx/include -I/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/xz-5.4.6-ptr4q3rditbzjwvmcgdpbqj36xab7gxw/include -I/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/libpciaccess-0.17-np6wwn6o2q2auw3e4vtuibyozyaqct44/include -I/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/ncurses-6.5-kj63bsmkj5yqshntpucmoaok24ar2m4f/include -I/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/openssl-3.3.0-4kl2j7rpk264t33ziw5no64kgaqvgl53/include -I/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/zlib-ng-2.1.6-ssevr4js2voelffrhd4fpazsfl3cel2x/include -I/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/libiconv-1.17-kgtp2nmv464qesavdvr6ydgxpuzovzz2/include -L/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/hdf5-1.10.11-t2e7pgbnglunryddu3hpktwom47sy6qq/lib -L/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/hwloc-2.9.1-p6soq54v4rppjzrqmfculeuziisjgvaa/lib -L/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/libevent-2.1.12-5oqymlhwmyjk6d6o7uukwgai4drw5id3/lib -L/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/openmpi-4.0.2-btwtzezafpl5gc3xsiqkxtvpbowtnbsk/lib -L/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/hdf5-1.10.11-t2e7pgbnglunryddu3hpktwom47sy6qq/lib -L/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/openmpi-4.0.2-btwtzezafpl5gc3xsiqkxtvpbowtnbsk/lib -L/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/pmix-4.2.2-hdv4t6rbpyop4yh63vgr272cn2zb4hz5/lib -L/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/numactl-2.0.14-3grq4d56qzmoq6lx6xmk3bt4njhf34tm/lib -L/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/libevent-2.1.12-5oqymlhwmyjk6d6o7uukwgai4drw5id3/lib -L/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/hwloc-2.9.1-p6soq54v4rppjzrqmfculeuziisjgvaa/lib -L/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/libxml2-2.10.3-qbqc235pet2nfutg77cusifdtakrdytx/lib -L/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/xz-5.4.6-ptr4q3rditbzjwvmcgdpbqj36xab7gxw/lib -L/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/libpciaccess-0.17-np6wwn6o2q2auw3e4vtuibyozyaqct44/lib -L/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/ncurses-6.5-kj63bsmkj5yqshntpucmoaok24ar2m4f/lib -L/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/openssl-3.3.0-4kl2j7rpk264t33ziw5no64kgaqvgl53/lib64 -L/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/zlib-ng-2.1.6-ssevr4js2voelffrhd4fpazsfl3cel2x/lib -L/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/libiconv-1.17-kgtp2nmv464qesavdvr6ydgxpuzovzz2/lib -Wl,--disable-new-dtags -Wl,-rpath,/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/netcdf-c-4.7.4-hbwkaatp5grgjv2adk5eelh7crhqhskn/lib -Wl,-rpath,/opt/release/linux-rocky8-x86_64/gcc-8.5.0/intel-oneapi-compilers-2023.2.4-hxxociow5cqbgxig7jtdtiwpg6tst7bs/compiler/2023.2.4/linux/compiler/lib/intel64_lin -Wl,-rpath,/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/hwloc-2.9.1-p6soq54v4rppjzrqmfculeuziisjgvaa/lib -Wl,-rpath,/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/libevent-2.1.12-5oqymlhwmyjk6d6o7uukwgai4drw5id3/lib -Wl,-rpath,/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/openmpi-4.0.2-btwtzezafpl5gc3xsiqkxtvpbowtnbsk/lib -Wl,-rpath,/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/netcdf-c-4.7.4-hbwkaatp5grgjv2adk5eelh7crhqhskn/lib -Wl,-rpath,/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/netcdf-c-4.7.4-hbwkaatp5grgjv2adk5eelh7crhqhskn/lib64 -Wl,-rpath,/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/hdf5-1.10.11-t2e7pgbnglunryddu3hpktwom47sy6qq/lib -Wl,-rpath,/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/openmpi-4.0.2-btwtzezafpl5gc3xsiqkxtvpbowtnbsk/lib -Wl,-rpath,/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/pmix-4.2.2-hdv4t6rbpyop4yh63vgr272cn2zb4hz5/lib -Wl,-rpath,/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/numactl-2.0.14-3grq4d56qzmoq6lx6xmk3bt4njhf34tm/lib -Wl,-rpath,/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/libevent-2.1.12-5oqymlhwmyjk6d6o7uukwgai4drw5id3/lib -Wl,-rpath,/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/hwloc-2.9.1-p6soq54v4rppjzrqmfculeuziisjgvaa/lib -Wl,-rpath,/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/libxml2-2.10.3-qbqc235pet2nfutg77cusifdtakrdytx/lib -Wl,-rpath,/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/xz-5.4.6-ptr4q3rditbzjwvmcgdpbqj36xab7gxw/lib -Wl,-rpath,/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/libpciaccess-0.17-np6wwn6o2q2auw3e4vtuibyozyaqct44/lib -Wl,-rpath,/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/ncurses-6.5-kj63bsmkj5yqshntpucmoaok24ar2m4f/lib -Wl,-rpath,/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/openssl-3.3.0-4kl2j7rpk264t33ziw5no64kgaqvgl53/lib64 -Wl,-rpath,/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/zlib-ng-2.1.6-ssevr4js2voelffrhd4fpazsfl3cel2x/lib -Wl,-rpath,/opt/release/linux-rocky8-x86_64_v4/intel-2021.10.0/libiconv-1.17-kgtp2nmv464qesavdvr6ydgxpuzovzz2/lib -Wl,-rpath,/opt/release/linux-rocky8-x86_64/gcc-8.5.0/intel-oneapi-compilers-2023.2.4-hxxociow5cqbgxig7jtdtiwpg6tst7bs/compiler/2023.2.4/linux/compiler/lib/intel64_lin -o .libs/ncgen generate.o main.o cdata.o bindata.o genchar.o cvt.o data.o debug.o escapes.o genc.o genbin.o generr.o genlib.o getfill.o odom.o semantics.o dump.o util.o bytebuffer.o list.o genf77.o f77data.o genj.o jdata.o nc_iter.o ncgeny.o ../liblib/.libs/libnetcdf.so -lhdf5_hl -lhdf5 -lm -pthread -lmpi -fPIC -O2

harshula · 2026-03-04T00:46:42Z

Docker container (not on Gadi)

Spack: 1.1.1 (spack/spack@2e2169d)
Builtin repo: spack/spack-packages@383e969
Python: 3.6.8
Platform: linux-rocky8-zen5

[ACCESS-ESM1.5] [root@878b48111968 ACCESS-ESM1.5]# git show
commit 8a613a7459429fc5b06f0fc52f099ea56396e9d1 (HEAD -> infra-update-v8, origin/infra-update-v8)
Author: Tommy Gatti <tommy.gatti@anu.edu.au>
Date:   Tue Mar 3 11:22:30 2026 +1100

    Use x86_64_v2 target for oasis3-mct specifically

[ACCESS-ESM1.5] [root@878b48111968 ACCESS-ESM1.5]# spack -V
1.1.1 (2e2169d5282d166f63e3ee4db8d4446c43cefa8a)

[ACCESS-ESM1.5] [root@878b48111968 ACCESS-ESM1.5]# diff -u spack-cc-netcdf-c-vse7qmw.in.log spack-cc-netcdf-c-vse7qmw.out.log | grep bytebuffer

-[cc] /opt/release/linux-x86_64/intel-oneapi-compilers-classic-2021.10.0-q2p33iufjfzwfjr7iqspe4luits7y6lc/bin/icc -DHAVE_CONFIG_H -I. -I.. -I../include -MT bytebuffer.o -MD -MP -MF .deps/bytebuffer.Tpo -c -o bytebuffer.o bytebuffer.c -I/opt/release/linux-x86_64_v2/openmpi-4.0.2-4sz27c4hm2gfxrgklkvknzjngwi3gvbf/include -pthread
-[ccld] /opt/release/linux-x86_64/intel-oneapi-compilers-classic-2021.10.0-q2p33iufjfzwfjr7iqspe4luits7y6lc/bin/icc -o .libs/ncgen generate.o main.o cdata.o bindata.o genchar.o cvt.o data.o debug.o escapes.o genc.o genbin.o generr.o genlib.o getfill.o odom.o semantics.o dump.o util.o bytebuffer.o list.o genf77.o f77data.o genj.o jdata.o nc_iter.o ncgeny.o -L/opt/release/linux-x86_64_v2/hdf5-1.10.11-gwi5iry4mwl5qmqvstnjtuku26kw4ond/lib ../liblib/.libs/libnetcdf.so -lhdf5_hl -lhdf5 -lm -Wl,-rpath -Wl,/opt/release/linux-x86_64_v2/netcdf-c-4.7.4-vse7qmw3vciqznlmvq34yd6ttchhmnkn/lib -I/opt/release/linux-x86_64_v2/openmpi-4.0.2-4sz27c4hm2gfxrgklkvknzjngwi3gvbf/include -pthread -Wl,-rpath -Wl,/opt/release/linux-x86_64_v2/openmpi-4.0.2-4sz27c4hm2gfxrgklkvknzjngwi3gvbf/lib -L/opt/release/linux-x86_64_v2/openmpi-4.0.2-4sz27c4hm2gfxrgklkvknzjngwi3gvbf/lib -lmpi
+[cc] /opt/release/linux-x86_64/intel-oneapi-compilers-classic-2021.10.0-q2p33iufjfzwfjr7iqspe4luits7y6lc/bin/icc -march=corei7 -mtune=generic -mpopcnt -I. -I.. -I../include -I/opt/release/linux-x86_64_v2/openmpi-4.0.2-4sz27c4hm2gfxrgklkvknzjngwi3gvbf/include -I/opt/release/linux-x86_64_v2/zlib-ng-2.3.2-frpuy2irgznanl26jax3ckunu6hdbywt/include -I/opt/release/linux-x86_64_v2/numactl-2.0.19-4c6o34fhrilx5ydtf4o426tczjvg7736/include -I/opt/release/linux-x86_64_v2/openmpi-4.0.2-4sz27c4hm2gfxrgklkvknzjngwi3gvbf/include -I/opt/release/linux-x86_64_v2/hdf5-1.10.11-gwi5iry4mwl5qmqvstnjtuku26kw4ond/include -DHAVE_CONFIG_H -MT bytebuffer.o -MD -MP -MF .deps/bytebuffer.Tpo -c -o bytebuffer.o bytebuffer.c -pthread -diag-disable=10441 -fPIC -O2
+[ccld] /opt/release/linux-x86_64/intel-oneapi-compilers-classic-2021.10.0-q2p33iufjfzwfjr7iqspe4luits7y6lc/bin/icc -march=corei7 -mtune=generic -mpopcnt -I/opt/release/linux-x86_64_v2/openmpi-4.0.2-4sz27c4hm2gfxrgklkvknzjngwi3gvbf/include -I/opt/release/linux-x86_64_v2/zlib-ng-2.3.2-frpuy2irgznanl26jax3ckunu6hdbywt/include -I/opt/release/linux-x86_64_v2/numactl-2.0.19-4c6o34fhrilx5ydtf4o426tczjvg7736/include -I/opt/release/linux-x86_64_v2/openmpi-4.0.2-4sz27c4hm2gfxrgklkvknzjngwi3gvbf/include -I/opt/release/linux-x86_64_v2/hdf5-1.10.11-gwi5iry4mwl5qmqvstnjtuku26kw4ond/include -L/opt/release/linux-x86_64_v2/hdf5-1.10.11-gwi5iry4mwl5qmqvstnjtuku26kw4ond/lib -L/opt/release/linux-x86_64_v2/openmpi-4.0.2-4sz27c4hm2gfxrgklkvknzjngwi3gvbf/lib -L/opt/release/linux-x86_64_v2/zlib-ng-2.3.2-frpuy2irgznanl26jax3ckunu6hdbywt/lib -L/opt/release/linux-x86_64_v2/numactl-2.0.19-4c6o34fhrilx5ydtf4o426tczjvg7736/lib -L/opt/release/linux-x86_64_v2/openmpi-4.0.2-4sz27c4hm2gfxrgklkvknzjngwi3gvbf/lib -L/opt/release/linux-x86_64_v2/hdf5-1.10.11-gwi5iry4mwl5qmqvstnjtuku26kw4ond/lib -Wl,--disable-new-dtags -Wl,-rpath,/opt/release/linux-x86_64_v2/netcdf-c-4.7.4-vse7qmw3vciqznlmvq34yd6ttchhmnkn/lib -Wl,-rpath,/opt/release/linux-x86_64_v2/openmpi-4.0.2-4sz27c4hm2gfxrgklkvknzjngwi3gvbf/lib -Wl,-rpath,/opt/release/linux-x86_64_v2/netcdf-c-4.7.4-vse7qmw3vciqznlmvq34yd6ttchhmnkn/lib -Wl,-rpath,/opt/release/linux-x86_64_v2/netcdf-c-4.7.4-vse7qmw3vciqznlmvq34yd6ttchhmnkn/lib64 -Wl,-rpath,/opt/release/linux-x86_64_v2/zlib-ng-2.3.2-frpuy2irgznanl26jax3ckunu6hdbywt/lib -Wl,-rpath,/opt/release/linux-x86_64_v2/numactl-2.0.19-4c6o34fhrilx5ydtf4o426tczjvg7736/lib -Wl,-rpath,/opt/release/linux-x86_64_v2/openmpi-4.0.2-4sz27c4hm2gfxrgklkvknzjngwi3gvbf/lib -Wl,-rpath,/opt/release/linux-x86_64_v2/hdf5-1.10.11-gwi5iry4mwl5qmqvstnjtuku26kw4ond/lib -Wl,-rpath,/opt/release/linux-x86_64/intel-oneapi-compilers-classic-2021.10.0-q2p33iufjfzwfjr7iqspe4luits7y6lc/compiler/lib/intel64_lin -o .libs/ncgen generate.o main.o cdata.o bindata.o genchar.o cvt.o data.o debug.o escapes.o genc.o genbin.o generr.o genlib.o getfill.o odom.o semantics.o dump.o util.o bytebuffer.o list.o genf77.o f77data.o genj.o jdata.o nc_iter.o ncgeny.o ../liblib/.libs/libnetcdf.so -lhdf5_hl -lhdf5 -lm -pthread -lmpi -diag-disable=10441 -fPIC -O2

harshula · 2026-03-04T00:57:53Z

Docker container (not on Gadi)

Update 06/03/2026: @becker33 suspects commit 8a613a7 is causing the march/mtune difference:

Spack v0.22

-march=skylake-avx512 -mtune=skylake-avx512

Spack v1.1

-march=corei7 -mtune=generic

Spack v1.1 after removing commit `8a613a7`

-march=skylake-avx512 -mtune=skylake-avx512

blimlim · 2026-03-04T22:55:43Z

Concretizer graph with spack 0.22 and 1.1:

0.22

 -   jkpm3a3  access-esm1p5@git.2024.12.1=2024.12.1%intel@19.0.3.199 build_system=bundle arch=linux-rocky8-x86_64_v4
 -   3gf6thf      ^cice4@git.access-esm1.5-2025.04.001=access-esm1.5%intel@19.0.3.199 build_system=makefile direct_ldflags=none arch=linux-rocky8-x86_64_v4
 -   j6yscmm          ^gmake@4.4.1%intel@19.0.3.199~guile build_system=generic arch=linux-rocky8-x86_64_v4
 -   lsig6tv          ^netcdf-fortran@4.5.2%intel@19.0.3.199~doc+pic+shared build_system=autotools patches=b050dbd arch=linux-rocky8-x86_64_v4
 -   572ga42          ^oasis3-mct@git.access-esm1.5_2024.05.24=access-esm1.5%intel@19.0.3.199~deterministic~optimisation_report build_system=makefile arch=linux-rocky8-x86_64_v4
[e]  mafwyvi          ^openmpi@4.0.2%intel@19.0.3.199~atomics~cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~internal-libevent~internal-pmix~java~legacylaunchers~lustre~memchecker~openshmem~orterunprefix~romio+rsh~singularity~static+vt+wrapper-rpath build_system=autotools fabrics=none patches=073477a,60ce20b romio-filesystem=none schedulers=none arch=linux-rocky8-x86_64_v4
[e]  mqjolvb      ^glibc@2.28%intel@19.0.3.199 build_system=autotools arch=linux-rocky8-x86_64_v4
 -   iuk76bo      ^mom5@git.access-esm1.5-2025.03.002=access-esm1.5%intel@19.0.3.199+restart_repro build_system=makefile arch=linux-rocky8-x86_64_v4
 -   mtmtrfl          ^netcdf-c@4.7.4%intel@19.0.3.199~blosc~byterange~dap~fsync~hdf4~jna+mpi~nczarr_zip+optimize~parallel-netcdf+pic+shared~szip~zstd build_system=autotools arch=linux-rocky8-x86_64_v4
 -   uygitju              ^hdf5@1.10.11%intel@19.0.3.199~cxx~fortran+hl~ipo~java+mpi+shared~szip~threadsafe+tools api=default build_system=cmake build_type=Release generator=make arch=linux-rocky8-x86_64_v4
[e]  ahfgfwe                  ^cmake@3.31.6%intel@19.0.3.199~doc+ncurses+ownlibs build_system=generic build_type=Release arch=linux-rocky8-x86_64_v4
 -   ugenh6g                  ^pkgconf@2.2.0%intel@19.0.3.199 build_system=autotools arch=linux-rocky8-x86_64_v4
 -   h45fvyw                  ^zlib-ng@2.1.6%intel@19.0.3.199+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-rocky8-x86_64_v4
 -   xvjigmg      ^um7@git.2024.10.17=access-esm1.5%intel@19.0.3.199+full build_system=generic optim=high arch=linux-rocky8-x86_64_v4
 -   qy5w2d7          ^dummygrib@1.0%intel@19.0.3.199 build_system=makefile arch=linux-rocky8-x86_64_v4
 -   ho2ie66          ^fcm@2021.05.0%intel@19.0.3.199 build_system=generic site=none arch=linux-rocky8-x86_64_v4
 -   sz4gvcr          ^gcom4@git.2024.05.28=access-esm1.5%intel@19.0.3.199+mpi build_system=generic arch=linux-rocky8-x86_64_v4

1.1

 -   siptkwj  access-esm1p5@latest build_system=bundle platform=linux os=rocky8 target=x86_64_v4
 -   rdo7ohi      ^cice4@git.access-esm1.5-2025.04.001=access-esm1.5 build_system=makefile commit=694a9fbd4ac29dc841b38aff002eb36da5b650f1 direct_ldflags=none platform=linux os=rocky8 target=x86_64_v4 %c,fortran=intel@19.0.3.199
[+]  vzuctxx          ^compiler-wrapper@1.0 build_system=generic platform=linux os=rocky8 target=x86_64_v4
[e]  vuczjrb          ^glibc@2.28 build_system=autotools platform=linux os=rocky8 target=x86_64
 -   wkhgvk3          ^gmake@4.4.1~guile build_system=generic platform=linux os=rocky8 target=x86_64_v4 %c=intel@19.0.3.199
[e]  6jvphra          ^intel-oneapi-compilers-classic@19.0.3.199 build_system=generic platform=linux os=rocky8 target=x86_64
 -   36z443p          ^netcdf-fortran@4.5.2~doc+pic+shared build_system=autotools patches:=b050dbd platform=linux os=rocky8 target=x86_64_v4 %c,fortran=intel@19.0.3.199
 -   ijhnniy          ^oasis3-mct@git.access-esm1.5_2024.05.24=access-esm1.5~deterministic~optimisation_report build_system=makefile commit=797bd2712d0dc133772c49ae6afbf10943ce3ded platform=linux os=rocky8 target=x86_64_v4 %c,fortran=intel@19.0.3.199
[e]  23im5ym          ^openmpi@4.0.2+atomics~cuda~cxx~cxx_exceptions~debug+fortran~gpfs~internal-hwloc~internal-libevent~internal-pmix~ipv6~java~lustre~memchecker~openshmem~orterunprefix~rocm+romio+rsh~static~two_level_namespace+vt+wrapper-rpath build_system=autotools fabrics:=none romio-filesystem:=none schedulers:=none platform=linux os=rocky8 target=x86_64
 -   y6lcskn      ^mom5@git.access-esm1.5-2025.03.002=access-esm1.5+restart_repro build_system=makefile commit=551e9e608de7f0390de42401100969c26b9eed0c platform=linux os=rocky8 target=x86_64_v4 %c,fortran=intel@19.0.3.199
 -   hlyuvib          ^netcdf-c@4.7.4~blosc~byterange~dap~fsync~hdf4~jna~logging+mpi~nczarr_zip+optimize~parallel-netcdf+pic+shared~szip~zstd build_system=autotools platform=linux os=rocky8 target=x86_64_v4 %c=intel@19.0.3.199
 -   ej3onoi              ^hdf5@1.10.11~cxx~fortran+hl~ipo~java+mpi+shared~szip~threadsafe+tools api=default build_system=cmake build_type=Release generator=make patches:=f42732a platform=linux os=rocky8 target=x86_64_v4 %c=intel@19.0.3.199
[e]  c6epq3u                  ^cmake@3.31.6~doc+ncurses+ownlibs~qtgui build_system=generic build_type=Release platform=linux os=rocky8 target=x86_64
 -   y7j77ka                  ^pkgconf@2.5.1 build_system=autotools platform=linux os=rocky8 target=x86_64_v4 %c=intel@19.0.3.199
 -   jeamext                  ^zlib-ng@2.3.2+compat~new_strategies+opt+pic+shared build_system=autotools platform=linux os=rocky8 target=x86_64_v4 %c,cxx=intel@19.0.3.199
 -   2pz4gq4      ^um7@git.2024.10.17=access-esm1.5+full build_system=generic commit=1d71fe04e003d9df0a2d9318c2ada09a28edef6a optim=high platform=linux os=rocky8 target=x86_64_v4 %c,fortran=intel@19.0.3.199
 -   4durshp          ^dummygrib@1.0 build_system=makefile commit=b56eaf3560bd0653b35538f36b7746171d8f7cad platform=linux os=rocky8 target=x86_64_v4 %fortran=intel@19.0.3.199
 -   mblh23p          ^fcm@2021.05.0 build_system=generic site=none platform=linux os=rocky8 target=x86_64_v4 %c,fortran=intel@19.0.3.199
 -   phgrh7n          ^gcom4@git.2024.05.28=access-esm1.5+mpi build_system=generic commit=e123dea738209ab10e194a10e2f19017aac2b032 platform=linux os=rocky8 target=x86_64_v4 %c,fortran=intel@19.0.3.199

blimlim · 2026-03-05T01:43:30Z

Differences in a model run first show up in the UM. Comparing the data passed between CICE<->MOM, there are no differences for the first three timesteps between the 0.22 and 1.1 builds:

During the first 3 timesteps, the ocean/ice models have not yet received any data from the UM.

Meanwhile, the solver norms in the UM's logs differ at the first timestep, suggesting that the atmospheric state changes first:

UM solver norms for the first 3 timesteps:

0.22

  Final Absolute Norm :   9.735899063190541E-003
  Final Absolute Norm :   9.907127310377030E-003
  Final Absolute Norm :   8.415570900815270E-003

1.1

  Final Absolute Norm :   9.735757890905522E-003
  Final Absolute Norm :   9.903919591002636E-003
  Final Absolute Norm :   8.443503694723147E-003

harshula · 2026-03-10T04:12:41Z

@blimlim & I were chatting. We are going to put the steps to reproduce the three different types of builds in this comment.

Reproducer steps

Spack v1.1 new build

Follow https://docs.access-hive.org.au/getting_started/spack/
git clone https://github.com/ACCESS-NRI/ACCESS-ESM1.5 --branch infra-update-v8
spack env activate -p ./ACCESS-ESM1.5
spack install

Spack v0.22 new build

git clone https://github.com/ACCESS-NRI/spack.git --branch releases/v0.22
git clone https://github.com/ACCESS-NRI/spack-config.git --branch main
git clone https://github.com/ACCESS-NRI/spack-packages.git --branch api-v1
ln -s -r -v spack-config/v0.22/gadi/* spack/etc/spack/

git clone https://github.com/ACCESS-NRI/ACCESS-ESM1.5 --branch main
spack env activate -p ./ACCESS-ESM1.5
spack install

Spack v0.22 release

Follow https://docs.access-hive.org.au/models/run_a_model/run_access-esm/

Run using a local build:

1) payu clone https://github.com/ACCESS-NRI/access-esm1.5-configs -B release-preindustrial+concentrations <control-directory-name>
2) Replace the following lines in the config.yaml file with the full paths to the locally built executables:
      exe: um_hg3.exe
      exe: fms_ACCESS-CM.x
      exe: cice_access_360x300_12x1_12p.exe
3) Change the following
manifest:
  reproduce:
    exe: True
to 
manifest:
  reproduce:
    exe: false

4) 1 month runs should be sufficient. Change 
    runtime:
        years: 1
        months: 0
        days: 0
to 
    runtime:
        years: 0
        months: 1
        days: 0

5) Run the model:
module use /g/data/vk83/modules
module load payu
payu setup
payu run -f

blimlim · 2026-03-11T02:59:40Z

Three way comparison

Great idea @harshula to do a three way comparison between the spack 0.22 released version, a new spack 0.22 build (also reproduced locally following the instructions here), and the spack 1.1 build

Running 1 day simulations using a UM only configuration

we have:

spack 0.22 release ≠ spack 0.22 new build

> nccmp -dg 0.22-release/archive/output000/atmosphere/netCDF/aiihca.pe-197801_dai.nc 0.22-apiv1head/archive/output000/atmosphere/netCDF/aiihca.pe-197801_dai.nc 
> DIFFER : VARIABLE : fld_s00i023 : POSITION : [0,1,20] : VALUES : 1100 <> 1100

spack 0.22 new build = spack 1.1

> nccmp -dg  0.22-apiv1head/archive/output000/atmosphere/netCDF/aiihca.pe-197801_dai.nc 1.1-all_x86_64_v4/archive/output000/atmosphere/netCDF/aiihca.pe-197801_dai.nc 
>

It looks like there are no issues with spack 1.1. Instead the non-reproducibility comes from changes between spack packages release 2025.03.006 and the head of the api-v1 branch

aidanheerdegen · 2026-03-11T05:24:44Z

It looks like there are no issues with spack 1.1. Instead the non-reproducibility comes from changes between spack packages release 2025.03.006 and the head of the api-v1 branch

Are you happy with those changes and it is understood why reproducibility is broken?

blimlim · 2026-03-11T05:40:54Z

I'm unsure what broke reproducibility, @harshula @manodeep, are there any changes in the um7 spack package (or packages for dependencies) which stand out as culprits: https://github.com/ACCESS-NRI/access-spack-packages/compare/2025.03.006..api-v1/

aidanheerdegen · 2026-03-11T06:05:28Z

I took a look. The um7 package changes focus mainly on ESM1.6. There is a lot of changes in the OASIS3-mct build, but it's not clear to me if they would be expected to make a change.

The substantive changes are here:

https://github.com/ACCESS-NRI/access-spack-packages/commits/354ff51459c25186ba75599cb5c87d093227048d/packages/oasis3-mct/package.py?browsing_rename_history=true&new_path=spack_repo/access/nri/packages/oasis3_mct/package.py&original_branch=api-v2

There are only five commits, so you could go backwards until you get back 0.22 build repro with new build?

Note: testing for breaking existing releases is something we had planned as a "nice to have" since the beginning of the org. This example shows why it is useful, and could save a lot of time.

harshula · 2026-03-11T06:41:03Z

Hi @blimlim , re: "spack 0.22 release ≠ spack 0.22 new build"
Is it still UM that introduces the difference?

harshula · 2026-03-11T06:44:09Z

We also need to decide whether we drop 8a613a7 before we merge.

CodeGat · 2026-03-11T21:58:10Z

I'll also have to drop beddb0a before merging, too

blimlim · 2026-03-11T22:22:31Z

Hi @blimlim , re: "spack 0.22 release ≠ spack 0.22 new build"
Is it still UM that introduces the difference?

Yes that's still the case. I checked reproducibility using a UM only configuration which doesn't use MOM or CICE. Hence it also seems less likely to be oasis3-mct that's breaking reproducibility here

manodeep · 2026-03-11T23:21:59Z

I'm unsure what broke reproducibility, @harshula @manodeep, are there any changes in the um7 spack package (or packages for dependencies) which stand out as culprits: https://github.com/ACCESS-NRI/access-spack-packages/compare/2025.03.006..api-v1/

I have looked through that diff and with the exception of FMS disabling gfs_phys by default, nothing else stood out as problematic. The only relevant change to the access-esm1.5 branch of UM7 recently is the orbital namelist PR.

A tangential thought - could this be a case of the spack bug affecting building from a branch, which reused the last successful build rather than using the latest commit? (my memory is vague, but this was something we saw in early 2025 for ESM 1.6, and, to sidestep the issue, we switched to building from commit hashes rather than branches)

blimlim · 2026-03-12T04:47:27Z

Reproducibility appears to have broken due to changes in the UM spack package specifically.

I tried starting with the HEAD of the api-v1 spack-packages branch, and reverting only the packages/um7/package.py package to its state in 2025.03.006 in the test-apiv1-use-2025.03.006-um7 branch. Building with this change, we recover reproducibility with the 2024.12.1 release of ACCESS-ESM1.5:

> nccmp -dg 0.22-release/archive/output000/atmosphere/netCDF/aiihca.pe-197801_dai.nc  0.22-apiv1head-revertumpackage-CIbuild/archive/output000/atmosphere/netCDF/aiihca.pe-197801_dai.nc 
>

The particular build used is here

Compiler flags from `-grecord-gcc-switches`

The executables built using the head of api-v1 , and the head of api-v1 with the um7 package reverted both had the -grecord-gcc-switches flags: https://github.com/ACCESS-NRI/ACCESS-ESM1.5/pull/48/changes

Running objdump -g um_hg3.exe | grep atmos_physics2 shows a difference (atmos_physics2.F90 is one of the source files)

`api-v1` HEAD (not reproducible)

<4000f>   Unknown AT value: 3b01: (indirect string, offset: 0x24cf8): -I/scratch/tm70/tm70_ci/tmp/restricted/spack-stage/spack-stage-um7-git.2024.10.17_access-esm1.5-2vjintzrzo5sqmg2sc5hnyp2efroek6c/spack-src/ummodel_hg3/inc -I/scratch/tm70/tm70_ci/tmp/restricted/spack-stage/spack-stage-um7-git.2024.10.17_access-esm1.5-2vjintzrzo5sqmg2sc5hnyp2efroek6c/spack-src/umbase_hg3/inc -I/g/data/vk83/prerelease/apps/spack/0.22/restricted/ukmo/release/linux-rocky8-x86_64_v4/intel-19.0.3.199/oasis3-mct-git.access-esm1.5_2024.05.24_access-esm1.5-aeqb2sz3gzixklatsuhp76j24znafsl2/include -I/g/data/vk83/prerelease/apps/spack/0.22/restricted/ukmo/release/linux-rocky8-x86_64_v4/intel-19.0.3.199/netcdf-fortran-4.5.2-lsig6tvoad3seiodkcmkp6o6za36zoaz/include -I/g/data/vk83/prerelease/apps/spack/0.22/restricted/ukmo/release/linux-rocky8-x86_64_v4/intel-19.0.3.199/netcdf-c-4.7.4-mtmtrflign6cy55x62bsgtlrbuymeypl/include -I/g/data/vk83/prerelease/apps/spack/0.22/restricted/ukmo/release/linux-rocky8-x86_64_v4/intel-19.0.3.199/hdf5-1.10.11-uygitju6jev5ze6z63lchr7uqm5nhg2m/include -I/g/data/vk83/prerelease/apps/spack/0.22/restricted/ukmo/release/linux-rocky8-x86_64_v4/intel-19.0.3.199/zlib-ng-2.1.6-h45fvywj47wc4uwa37mfzkdsqrgcqxux/include -I/g/data/vk83/prerelease/apps/spack/0.22/restricted/ukmo/release/linux-rocky8-x86_64_v4/intel-19.0.3.199/gcom4-git.2024.05.28_access-esm1.5-sz4gvcrkf35p3464asqujle4q5bev6rq/include -I/apps/openmpi/4.0.2/include -I/apps/openmpi/4.0.2/include -I/apps/openmpi/4.0.2/include/Intel -I/apps/openmpi/4.0.2/include -I/opt/Modules/v4.3.0/init/ksh-functions -march=skylake-avx512 -mtune=skylake-avx512 -o atmos_physics2.o -O2 -g -traceback -i8 -r8 -fp-model precise -ftz -what -fno-alias -stack-temps -safe-cray-ptr -c -pthread -grecord-gcc-switches -g3 -debug all

`api-v1` HEAD revert um7 package (reproducible)

<3e3cb>   Unknown AT value: 3b01: (indirect string, offset: 0x17ebd): -I/scratch/tm70/tm70_ci/tmp/restricted/spack-stage/spack-stage-um7-git.2024.10.17_access-esm1.5-v6b4wazcews42yk6d27q2dov5iusphca/spack-src/ummodel_hg3/inc -I/scratch/tm70/tm70_ci/tmp/restricted/spack-stage/spack-stage-um7-git.2024.10.17_access-esm1.5-v6b4wazcews42yk6d27q2dov5iusphca/spack-src/umbase_hg3/inc -I/apps/openmpi/4.0.2/include -I/apps/openmpi/4.0.2/include/Intel -I/apps/openmpi/4.0.2/include -I/opt/Modules/v4.3.0/init/ksh-functions -o atmos_physics2.o -O2 -g -traceback -V -i8 -r8 -fp-model precise -ftz -what -fno-alias -stack-temps -safe-cray-ptr -c -pthread

I'm not sure how to interpret this difference

blimlim · 2026-03-12T22:26:10Z

Found by @manodeep: One difference between the um7 spack package in https://github.com/ACCESS-NRI/access-spack-packages/compare/2025.03.006..api-v1/ is the removal of the -V flag, which @penguian discovered causes other compiler flags to be dropped here.

This appears to be the change that broke reproducibility. I produced an extra build here. This matches the spack 0.22 2024.12.1 release, but with the -V flag removed from the um7 spack package.

We get:

spack 0.22 2024.12.1 release ≠ spack 0.22 2024.12.1 with `-V` removed:

> nccmp -dg 0.22-release/archive/output000/atmosphere/netCDF/aiihca.pe-197801_dai.nc 0.22-remove-V/archive/output000/atmosphere/netCDF/aiihca.pe-197801_dai.nc 
> DIFFER : VARIABLE : fld_s00i023 : POSITION : [0,1,20] : VALUES : 1100 <> 1100

spack 0.22 2024.12.1 with `-V` removed = spack 1.1 build

> nccmp -dg 0.22-remove-V/archive/output000/atmosphere/netCDF/aiihca.pe-197801_dai.nc 1.1-all_x86_64_v4archive/output000/atmosphere/netCDF/aiihca.pe-197801_dai.nc 
>

blimlim · 2026-03-12T23:12:49Z

Unfortunately that's not the whole story. The above simulations were 2 day runs using a UM only configuration.

I reran a similar comparison with the same builds in longer 1 month runs, using a coupled configuration which includes MOM and CICE. Comparing the monthly mean atmosphere outputs, 4 variables differ when using the spack 1.1 executable, and the "spack 0.22 2024.12.1 with -V removed" executable. The following show plots of each run for these variables, and the difference between the two runs:

Although MOM and CICE were included in these simulations, their outputs are identical between the two runs and so it still seems like a UM issue.

CodeGat · 2026-03-12T23:57:25Z

Wow @blimlim and @manodeep, thank you so much for the detailed investigation! After chatting with @blimlim I think we will merge without a release, with the caveat that the next release will break repro.

I've dropped a test commit and the oasis3-mct x86_64_v2 commit, so this should be ready to go once approved.

blimlim

Thanks @CodeGat, looks good!

CodeGat added the type:infra Related to CI/CD label Nov 18, 2025

CodeGat self-assigned this Nov 18, 2025

CodeGat added 6 commits January 8, 2026 16:19

[no ci] infra: Update to v8

4bcdbf4

[no ci] infra: Update spack version to v1.1, update manifest schema t…

5658fca

…o 2-0-0

[no ci] spack.yaml: Dedent list elements for spack-style manifests

198bb50

[no ci] spack.yaml: Add reserved definitions

b14df0b

[no ci] versions.json,spack.yaml: Add ukmo-restricted-scope to custom…

93097a1

…-scopes rather than environment

[no ci] spack.yaml: Use toolchains

a8faf46

CodeGat force-pushed the infra-update-v8 branch from 98bed82 to a8faf46 Compare January 8, 2026 05:20

CodeGat added 2 commits January 20, 2026 15:50

[no ci] Remove inputs.pr and inputs.root-sbd, updated versions in ci-…

ebb2e1e

…comment

[no ci] Update config/versions.json access-spack-packages version to …

3208ca9

…common api-v2 tag

CodeGat marked this pull request as ready for review February 16, 2026 23:04

[no ci] Add language constraints

8c3cdc1

CodeGat temporarily deployed to Gadi Prerelease February 16, 2026 23:10 — with GitHub Actions Inactive

access-bot mentioned this pull request Feb 17, 2026

Auto update dev-preindustrial+concentrations to use access-esm1p5/pr46-1 ACCESS-NRI/access-esm1.5-configs#142

Draft

CodeGat mentioned this pull request Feb 17, 2026

Spack v0.22 to v1.1.x migration plan ACCESS-NRI/access-spack-packages#345

Open

CodeGat temporarily deployed to Gadi Prerelease February 18, 2026 01:18 — with GitHub Actions Inactive

access-bot mentioned this pull request Feb 18, 2026

Auto update dev-preindustrial+concentrations to use access-esm1p5/pr46-2 ACCESS-NRI/access-esm1.5-configs#143

Draft

harshula mentioned this pull request Feb 23, 2026

build_source_code.md: Update from Spack v0.22 to Spack v1.1 ACCESS-NRI/ACCESS-Hive-Docs#1154

Draft

CodeGat temporarily deployed to Gadi Prerelease March 3, 2026 00:24 — with GitHub Actions Inactive

[no ci] Add target preference rather than requirement

223d1eb

CodeGat force-pushed the infra-update-v8 branch from 8a613a7 to 223d1eb Compare March 12, 2026 23:54

blimlim reviewed Mar 13, 2026

View reviewed changes

Comment thread spack.yaml

blimlim approved these changes Mar 13, 2026

View reviewed changes

CodeGat merged commit f257282 into main Mar 13, 2026

CodeGat deleted the infra-update-v8 branch March 13, 2026 00:35

Conversation

CodeGat commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Background

Features

Prerequisites for Merging

Uh oh!

github-actions Bot commented Feb 16, 2026

🖥️ Gadi Deployment ✔️

Uh oh!

CodeGat commented Feb 17, 2026

Uh oh!

github-actions Bot commented Feb 17, 2026

Uh oh!

CodeGat commented Feb 17, 2026

Uh oh!

github-actions Bot commented Feb 18, 2026

🖥️ Gadi Deployment ✔️

Uh oh!

CodeGat commented Feb 18, 2026

Uh oh!

github-actions Bot commented Feb 18, 2026

Uh oh!

github-actions Bot commented Mar 3, 2026

🖥️ Gadi Deployment ✔️

Uh oh!

CodeGat commented Mar 3, 2026

Uh oh!

github-actions Bot commented Mar 3, 2026

Uh oh!

harshula commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Docker container (not on Gadi)

Uh oh!

harshula commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Docker container (not on Gadi)

Uh oh!

harshula commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Docker container (not on Gadi)

Spack v0.22

Spack v1.1

Spack v1.1 after removing commit 8a613a7

Uh oh!

blimlim commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

0.22

1.1

Uh oh!

blimlim commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

UM solver norms for the first 3 timesteps:

Uh oh!

harshula commented Mar 10, 2026 • edited by blimlim Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reproducer steps

Spack v1.1 new build

Spack v0.22 new build

Spack v0.22 release

Run using a local build:

Uh oh!

blimlim commented Mar 11, 2026

Three way comparison

spack 0.22 release ≠ spack 0.22 new build

spack 0.22 new build = spack 1.1

Uh oh!

aidanheerdegen commented Mar 11, 2026

Uh oh!

blimlim commented Mar 11, 2026

Uh oh!

aidanheerdegen commented Mar 11, 2026

Uh oh!

harshula commented Mar 11, 2026

Uh oh!

harshula commented Mar 11, 2026

Uh oh!

CodeGat commented Mar 11, 2026

Uh oh!

blimlim commented Mar 11, 2026

CodeGat commented Nov 18, 2025 •

edited

Loading

🖥️ `Gadi` Deployment ✔️

🖥️ `Gadi` Deployment ✔️

🖥️ `Gadi` Deployment ✔️

harshula commented Mar 4, 2026 •

edited

Loading

harshula commented Mar 4, 2026 •

edited

Loading

harshula commented Mar 4, 2026 •

edited

Loading

Spack v1.1 after removing commit `8a613a7`

blimlim commented Mar 4, 2026 •

edited

Loading

blimlim commented Mar 5, 2026 •

edited

Loading

harshula commented Mar 10, 2026 •

edited by blimlim

Loading

manodeep commented Mar 11, 2026 •

edited

Loading

Compiler flags from `-grecord-gcc-switches`

`api-v1` HEAD (not reproducible)

`api-v1` HEAD revert um7 package (reproducible)

spack 0.22 2024.12.1 release ≠ spack 0.22 2024.12.1 with `-V` removed:

spack 0.22 2024.12.1 with `-V` removed = spack 1.1 build