Skip to content

Update module load path in config.yaml to test pre-release build#403

Closed
aidanheerdegen wants to merge 14 commits intodev-preindustrial+concentrationsfrom
402-OASIS3-mct-update
Closed

Update module load path in config.yaml to test pre-release build#403
aidanheerdegen wants to merge 14 commits intodev-preindustrial+concentrationsfrom
402-OASIS3-mct-update

Conversation

@aidanheerdegen
Copy link
Copy Markdown
Member

@aidanheerdegen aidanheerdegen commented Jan 26, 2026

1. Summary:

Update the ACCESS-ESM1.6 module to pick up changes to OASIS3-mct code provenance.

To close #402

2. Issues Addressed:

3. Dependencies (e.g. on payu, or model)

This change requires changes to (note pull request(s) where relevant):

  • workflow manager (payu):
  • model deployment (ACCESS-ESM1.6):
  • model component or library dependency:
  • input workflow:

4. Ad-hoc Testing

What ad-hoc testing was done? How are you convinced this change is correct (plots are good)?

5. CI Testing

  • !test repro has been run

6. Reproducibility

Is this reproducible with the previous commit? (If not, why not?)

  • Yes
  • No - !test repro commit has been run.

7. Performance

Has the model performance (say, throughput of model-years/wall-day) changed?

  • Yes
  • No
  • N/A (if selected, please add a brief explanation why performance testing is not necessary for this PR)

If yes, provide the numbers from your testing. Is the performance better or worse?

8. Manifests

Have you changed the executable, the input files and/or the restart files?

  • Yes
  • No

If yes, have you updated the manifests?

  • Yes
  • No

To update the manifests, run payu setup (in a cloned copy of your feature branch) with reproducibility tests turned off:

manifest:
  reproduce:
    exe: false
    input: false
    restart: false
runlog:
  enable: false

Then commit the newly created manifest files (under manifests/) only to the branch for this PR.

9. Documentation

Is the documentation updated?

  • Yes
  • N/A

10. Merge Strategy

  • Merge commit
  • Rebase and merge
  • Squash

@aidanheerdegen
Copy link
Copy Markdown
Member Author

!test repro

@github-actions
Copy link
Copy Markdown

⚠️ The Bitwise Reproducibility Check Had Errors - Check https://github.com/ACCESS-NRI/access-esm1.6-configs/actions/runs/21378073877 ⚠️
❌ The Bitwise Reproducibility Check Failed ❌

When comparing:

  • 402-OASIS3-mct-update (checksums created using commit 595754a), against
  • dev-preindustrial+concentrations (checksums in commit d5be642)
Further information

The experiment can be found on Gadi at /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/595754a108976da9395522516b4066ab0ccf3a22, and the test results at https://github.com/ACCESS-NRI/access-esm1.6-configs/runs/61538986688.

The checksums generated by this !test command are found in the testing/checksum directory of https://github.com/ACCESS-NRI/access-esm1.6-configs/actions/runs/21378073877/artifacts/5264793479.

The checksums compared against are found here https://github.com/ACCESS-NRI/access-esm1.6-configs/tree/d5be642259e4b502b0cf79ed91ee3a21e0ce20e4/testing/checksum

Test summary:
🔥 test_repro_historical
🔥 test_repro_determinism

@aidanheerdegen
Copy link
Copy Markdown
Member Author

Got a negative on the repro test because I didn't change the exe manifest check. It is misleading to report a repro error when it just failed to payu setup. Would be good to report this error correctly.

@aidanheerdegen
Copy link
Copy Markdown
Member Author

!test repro

@github-actions
Copy link
Copy Markdown

⚠️ The Bitwise Reproducibility Check Had Errors - Check https://github.com/ACCESS-NRI/access-esm1.6-configs/actions/runs/21378675615 ⚠️
❌ The Bitwise Reproducibility Check Failed ❌

When comparing:

  • 402-OASIS3-mct-update (checksums created using commit c24c59e), against
  • dev-preindustrial+concentrations (checksums in commit d5be642)
Further information

The experiment can be found on Gadi at /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/c24c59ef2c6a2f25e5d5fc6511634cb846080a84, and the test results at https://github.com/ACCESS-NRI/access-esm1.6-configs/runs/61540952316.

The checksums generated by this !test command are found in the testing/checksum directory of https://github.com/ACCESS-NRI/access-esm1.6-configs/actions/runs/21378675615/artifacts/5265020344.

The checksums compared against are found here https://github.com/ACCESS-NRI/access-esm1.6-configs/tree/d5be642259e4b502b0cf79ed91ee3a21e0ce20e4/testing/checksum

Test summary:
🔥 test_repro_historical
🔥 test_repro_determinism

manodeep and others added 3 commits January 27, 2026 11:19
Co-authored-by: Anton Steketee <79179784+anton-seaice@users.noreply.github.com>
@aidanheerdegen
Copy link
Copy Markdown
Member Author

aidanheerdegen commented Jan 27, 2026

Crashed because the namcouple file is inconsistent with v5 of OASIS3-mct.

Cherry-picked the 3 commits that modified the namcouple file from @manodeep's PR #298

@aidanheerdegen
Copy link
Copy Markdown
Member Author

!test repro

@github-actions
Copy link
Copy Markdown

⚠️ The Bitwise Reproducibility Check Had Errors - Check https://github.com/ACCESS-NRI/access-esm1.6-configs/actions/runs/21379473183 ⚠️
❌ The Bitwise Reproducibility Check Failed ❌

When comparing:

  • 402-OASIS3-mct-update (checksums created using commit d0aadfe), against
  • dev-preindustrial+concentrations (checksums in commit d5be642)
Further information

The experiment can be found on Gadi at /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/d0aadfef1b66e0e60a9f04da247f304911aa4879, and the test results at https://github.com/ACCESS-NRI/access-esm1.6-configs/runs/61543611288.

The checksums generated by this !test command are found in the testing/checksum directory of https://github.com/ACCESS-NRI/access-esm1.6-configs/actions/runs/21379473183/artifacts/5265338758.

The checksums compared against are found here https://github.com/ACCESS-NRI/access-esm1.6-configs/tree/d5be642259e4b502b0cf79ed91ee3a21e0ce20e4/testing/checksum

Test summary:
🔥 test_repro_historical
🔥 test_repro_determinism

@aidanheerdegen
Copy link
Copy Markdown
Member Author

There were still differences between the namcouple file after cherry-picking

$ git diff origin/final-opt-devPI -- coupler/namcouple
Details of diff

diff --git a/coupler/namcouple b/coupler/namcouple
index 878e5b5..fe96587 100644
--- a/coupler/namcouple
+++ b/coupler/namcouple
@@ -193,9 +193,8 @@ heatflux:pen_sol:lhflx:tmlt01:tmlt02:tmlt03:tmlt04:tmlt05:bmlt01:bmlt02:bmlt03:b
 192 145 360 300 um1t cice LAG=+1800 SEQ=+1
 P 0 P 0
 #
-LOCTRANS CHECKIN SCRIPR CHECKOUT
+LOCTRANS SCRIPR
 INSTANT
-INT=0
 CONSERV LR SCALAR LATLON 10 FRACNNEI FIRST
 INT=0
 # -- 19:20:21:22:23
@@ -203,9 +202,8 @@ runoff:wme:train:tsnow:evap2d runoff_i:wme_i:rain_i:snow_i:evap_i 1 10800 4 a2i.
 192 145 360 300 um1t cice LAG=+1800 SEQ=+1
 P 0 P 0
 #
-LOCTRANS CHECKIN SCRIPR CHECKOUT
+LOCTRANS SCRIPR
 INSTANT
-INT=0
 CONSERV LR SCALAR LATLON 10 DESTAREA FIRST
 INT=0
 # -- 19: 
@@ -213,19 +211,16 @@ taux taux_i 23 10800 4 a2i.nc EXPORTED
 192 145 360 300 um1u cice LAG=+1800 SEQ=+1
 P 0 P 0
 #
-LOCTRANS CHECKIN SCRIPR CHECKOUT
+LOCTRANS SCRIPR
 INSTANT
-INT=0
 CONSERV LR SCALAR LATLON 10 FRACNNEI FIRST
-INT=0
 # -- 20: 
 tauy tauy_i 24 10800 4 a2i.nc EXPORTED
 192 144 360 300 um1v cice LAG=+1800 SEQ=+1
 P 0 P 0
 #
-LOCTRANS CHECKIN SCRIPR CHECKOUT
+LOCTRANS SCRIPR
 INSTANT
-INT=0
 CONSERV LR SCALAR LATLON 10 FRACNNEI FIRST
 INT=0
 # -- 26: 
@@ -233,11 +228,9 @@ wnd_a wnd_ai 33 10800 4 a2i.nc EXPORTED
 192 144 360 300 um1v cice LAG=+1800 SEQ=+1
 P 0 P 0
 #
-LOCTRANS CHECKIN SCRIPR CHECKOUT
+LOCTRANS SCRIPR
 INSTANT
-INT=0
 CONSERV LR SCALAR LATLON 10 FRACNNEI FIRST
-INT=0
 ############################################################################
 #
 #                      ICE --->>> OCEAN
@@ -481,84 +474,68 @@ INSTANT
 #                      ICE --->>> ATMOSPHERE
 #                      ---------------------
 # -- 01:
-isst_ia ocn_sst 1 10800 4 i2a.nc EXPORTED
+isst_ia ocn_sst 1 10800 2 i2a.nc EXPORTED
 360 300 192 145 cice um1t LAG=+3600 SEQ=+1
 P 0 P 0
 #
-LOCTRANS CHECKIN SCRIPR CHECKOUT
+LOCTRANS SCRIPR
 INSTANT
-INT=0
 CONSERV LR SCALAR LATLON 10 FRACNNEI FIRST
-INT=0
 # -- 02:
 icecon01:icecon02:icecon03:icecon04:icecon05 ofrzn01:ofrzn02:ofrzn03:ofrzn04:ofrzn05 31 10800 4 i2a.nc EXPORTED
 360 300 192 145 cice um1t LAG=+3600 SEQ=+1
 P 0 P 0
 #
-LOCTRANS CHECKIN SCRIPR CHECKOUT
+LOCTRANS SCRIPR
 INSTANT
-INT=0
 CONSERV LR SCALAR LATLON 10 FRACNNEI FIRST
-INT=0
 # -- 07:
 snwthk01:snwthk02:snwthk03:snwthk04:snwthk05 osnwtn01:osnwtn02:osnwtn03:osnwtn04:osnwtn05 373 10800 4 i2a.nc EXPORTED
 360 300 192 145 cice um1t LAG=+3600 SEQ=+1
 P 0 P 0
 #
-LOCTRANS CHECKIN SCRIPR CHECKOUT
+LOCTRANS SCRIPR
 INSTANT
-INT=0
 CONSERV LR SCALAR LATLON 10 FRACNNEI FIRST
-INT=0
 # -- 12:
 icethk01:icethk02:icethk03:icethk04:icethk05 ohicn01:ohicn02:ohicn03:ohicn04:ohicn05 468 10800 4 i2a.nc EXPORTED
 360 300 192 145 cice um1t LAG=+3600 SEQ=+1
 P 0 P 0
 #
-LOCTRANS CHECKIN SCRIPR CHECKOUT
+LOCTRANS SCRIPR
 INSTANT
-INT=0
 CONSERV LR SCALAR LATLON 10 FRACNNEI FIRST
-INT=0
 # -- 17:
-uvel_ia sunocean 47 10800 4 i2a.nc EXPORTED
+uvel_ia sunocean 47 10800 2 i2a.nc EXPORTED
 360 300 192 145 cice um1u LAG=+3600 SEQ=+1
 P 0 P 0
 #
-LOCTRANS CHECKIN SCRIPR CHECKOUT
+LOCTRANS SCRIPR
 INSTANT
-INT=0
 CONSERV LR SCALAR LATLON 10 FRACNNEI FIRST
-INT=0
 # -- 18:
-vvel_ia svnocean 48 10800 4 i2a.nc EXPORTED
+vvel_ia svnocean 48 10800 2 i2a.nc EXPORTED
 360 300 192 144 cice um1v LAG=+3600 SEQ=+1
 P 0 P 0
 #
-LOCTRANS CHECKIN SCRIPR CHECKOUT
+LOCTRANS SCRIPR
 INSTANT
-INT=0
 CONSERV LR SCALAR LATLON 10 FRACNNEI FIRST
-INT=0
 # -- 19:
 co2_i2 co2_ia 1 10800 4 i2a.nc EXPORTED
 360 300 192 145 cice um1t LAG=+3600 SEQ=+1
 P 0 P 0
 #
-LOCTRANS CHECKIN SCRIPR CHECKOUT
+LOCTRANS SCRIPR
 INSTANT
-INT=0
 CONSERV LR SCALAR LATLON 10 FRACNNEI FIRST
-INT=0
 # -- 20:
 co2fx_i2 co2fx_ia 31 10800 4 i2a.nc EXPORTED
 360 300 192 145 cice um1t LAG=+3600 SEQ=+1
 P 0 P 0
 #
-LOCTRANS CHECKIN SCRIPR CHECKOUT
+LOCTRANS SCRIPR
 INSTANT
-INT=0
 CONSERV LR SCALAR LATLON 10 FRACNNEI FIRST
-INT=0
 ############################################################################
  $END

and was getting errors in reading the file, so just grabbed the version of namcouple from the functioning PR

git checkout origin/final-opt-devPI -- coupler/namcouple

@aidanheerdegen
Copy link
Copy Markdown
Member Author

!test repro

@github-actions
Copy link
Copy Markdown

✅ The Bitwise Reproducibility Check Succeeded ✅

When comparing:

  • 402-OASIS3-mct-update (checksums created using commit 5111e06), against
  • dev-preindustrial+concentrations (checksums in commit d5be642)
Further information

The experiment can be found on Gadi at /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/5111e06e2f9227503d71006f75bed6d65884d508, and the test results at https://github.com/ACCESS-NRI/access-esm1.6-configs/runs/61546283754.

The checksums generated by this !test command are found in the testing/checksum directory of https://github.com/ACCESS-NRI/access-esm1.6-configs/actions/runs/21380329600/artifacts/5265664781.

The checksums compared against are found here https://github.com/ACCESS-NRI/access-esm1.6-configs/tree/d5be642259e4b502b0cf79ed91ee3a21e0ce20e4/testing/checksum

Test summary:
test_repro_historical
test_repro_determinism

@aidanheerdegen
Copy link
Copy Markdown
Member Author

!test repro

@github-actions
Copy link
Copy Markdown

✅ The Bitwise Reproducibility Check Succeeded ✅

When comparing:

  • 402-OASIS3-mct-update (checksums created using commit d1da0b2), against
  • dev-preindustrial+concentrations (checksums in commit d5be642)
Further information

The experiment can be found on Gadi at /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/d1da0b2f0d227de96d622b20e4908cbbc8060de0, and the test results at https://github.com/ACCESS-NRI/access-esm1.6-configs/runs/61966029827.

The checksums generated by this !test command are found in the testing/checksum directory of https://github.com/ACCESS-NRI/access-esm1.6-configs/actions/runs/21507159700/artifacts/5314790913.

The checksums compared against are found here https://github.com/ACCESS-NRI/access-esm1.6-configs/tree/d5be642259e4b502b0cf79ed91ee3a21e0ce20e4/testing/checksum

Test summary:
test_repro_historical
test_repro_determinism

@aidanheerdegen
Copy link
Copy Markdown
Member Author

!test repro

@github-actions
Copy link
Copy Markdown

✅ The Bitwise Reproducibility Check Succeeded ✅

When comparing:

  • 402-OASIS3-mct-update (checksums created using commit 262aeef), against
  • dev-preindustrial+concentrations (checksums in commit d5be642)
Further information

The experiment can be found on Gadi at /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/262aeef6017a34623386d976152ba81ff6bbcad8, and the test results at https://github.com/ACCESS-NRI/access-esm1.6-configs/runs/61973649099.

The checksums generated by this !test command are found in the testing/checksum directory of https://github.com/ACCESS-NRI/access-esm1.6-configs/actions/runs/21509597912/artifacts/5315642620.

The checksums compared against are found here https://github.com/ACCESS-NRI/access-esm1.6-configs/tree/d5be642259e4b502b0cf79ed91ee3a21e0ce20e4/testing/checksum

Test summary:
test_repro_historical
test_repro_determinism

@anton-seaice anton-seaice marked this pull request as draft February 1, 2026 21:44
@anton-seaice
Copy link
Copy Markdown
Collaborator

I switched cice5 to use the CMake build

@anton-seaice
Copy link
Copy Markdown
Collaborator

!test repro

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Feb 1, 2026

⚠️ The Bitwise Reproducibility Check Had Errors - Check https://github.com/ACCESS-NRI/access-esm1.6-configs/actions/runs/21570955208 ⚠️
❌ The Bitwise Reproducibility Check Failed ❌

When comparing:

  • 402-OASIS3-mct-update (checksums created using commit 61dfcfe), against
  • dev-preindustrial+concentrations (checksums in commit d5be642)
Further information

The experiment can be found on Gadi at /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/61dfcfe71d7a20cbdaecbe3e856e64bfeb148cf3, and the test results at https://github.com/ACCESS-NRI/access-esm1.6-configs/runs/62149978303.

The checksums generated by this !test command are found in the testing/checksum directory of https://github.com/ACCESS-NRI/access-esm1.6-configs/actions/runs/21570955208/artifacts/5336147397.

The checksums compared against are found here https://github.com/ACCESS-NRI/access-esm1.6-configs/tree/d5be642259e4b502b0cf79ed91ee3a21e0ce20e4/testing/checksum

Test summary:
🔥 test_repro_historical
🔥 test_repro_determinism

@anton-seaice
Copy link
Copy Markdown
Collaborator

!test repro

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Feb 1, 2026

⚠️ The Bitwise Reproducibility Check Had Errors - Check https://github.com/ACCESS-NRI/access-esm1.6-configs/actions/runs/21571024531 ⚠️
❌ The Bitwise Reproducibility Check Failed ❌

When comparing:

  • 402-OASIS3-mct-update (checksums created using commit 1325982), against
  • dev-preindustrial+concentrations (checksums in commit d5be642)
Further information

The experiment can be found on Gadi at /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae, and the test results at https://github.com/ACCESS-NRI/access-esm1.6-configs/runs/62150239289.

The checksums generated by this !test command are found in the testing/checksum directory of https://github.com/ACCESS-NRI/access-esm1.6-configs/actions/runs/21571024531/artifacts/5336175561.

The checksums compared against are found here https://github.com/ACCESS-NRI/access-esm1.6-configs/tree/d5be642259e4b502b0cf79ed91ee3a21e0ce20e4/testing/checksum

Test summary:
🔥 test_repro_historical
🔥 test_repro_determinism

@anton-seaice
Copy link
Copy Markdown
Collaborator

huh ?

gc_abort (Processor    96): over-writing due to dim_e_out size
gc_abort (Processor   224): over-writing due to dim_e_out size
gc_abort (Processor   176): over-writing due to dim_e_out size
gc_abort (Processor    64): over-writing due to dim_e_out size
gc_abort (Processor    16): over-writing due to dim_e_out size
gc_abort (Processor    80): over-writing due to dim_e_out size
gc_abort (Processor   208): over-writing due to dim_e_out size
gc_abort (Processor   112): over-writing due to dim_e_out size
gc_abort (Processor   240): over-writing due to dim_e_out size
gc_abort (Processor   192): over-writing due to dim_e_out size
gc_abort (Processor    32): over-writing due to dim_e_out size
 *********************************************************************************
 UM ERROR (Model aborting) :
 Routine generating error: Bi_linear_h
 Error code:                     10
 Error message: 
over-writing due to dim_e_out size
 *********************************************************************************
gc_abort (Processor     0): over-writing due to dim_e_out size
gc_abort (Processor   160): over-writing due to dim_e_out size
gc_abort (Processor   144): over-writing due to dim_e_out size
gc_abort (Processor   128): over-writing due to dim_e_out size
gc_abort (Processor    48): over-writing due to dim_e_out size
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 208 in communicator MPI COMM 9 SPLIT FROM 0
  Proc: [[39998,1],208]
  Errorcode: 9

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
�[gadi-cpu-spr-0218:3261911] [0] func:/apps/openmpi/5.0.8/lib/libopen-pal.so.80(opal_backtrace_buffer+0x21) [0x14b25c2ae7e1]
[gadi-cpu-spr-0218:3261911] [1] func:/apps/openmpi/5.0.8/lib/libmpi.so.40(+0x39eaa) [0x14b25c362eaa]
[gadi-cpu-spr-0218:3261911] [2] func:/apps/openmpi/5.0.8/lib/libmpi.so.40(PMPI_Abort+0xdb) [0x14b25c37868b]
[gadi-cpu-spr-0218:3261911] [3] func:/apps/openmpi/5.0.8/lib/libmpi_mpifh_Intel.so.40(pmpi_abort+0x20) [0x14b25fad9650]
[gadi-cpu-spr-0218:3261911] [4] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x12e1eb9]
[gadi-cpu-spr-0218:3261911] [5] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x12de910]
[gadi-cpu-spr-0218:3261911] [6] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x41ce07]
[gadi-cpu-spr-0218:3261911] [7] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x7def9c]
[gadi-cpu-spr-0218:3261911] [8] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0xac0420]
[gadi-cpu-spr-0218:3261911] [9] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x813d00]
[gadi-cpu-spr-0218:3261911] [10] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x7414a1]
[gadi-cpu-spr-0218:3261911] [11] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x61b8d2]
[gadi-cpu-spr-0218:3261911] [12] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x4ac81e]
[gadi-cpu-spr-0218:3261911] [13] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x440de3]
[gadi-cpu-spr-0218:3261911] [14] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x416627]
[gadi-cpu-spr-0218:3261911] [15] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x4116e8]
[gadi-cpu-spr-0218:3261911] [16] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x41164d]
[gadi-cpu-spr-0218:3261911] [17] func:/lib64/libc.so.6(__libc_start_main+0xe5) [0x14b25e94b7e5]
[gadi-cpu-spr-0218:3261911] [18] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x41156e]
[gadi-cpu-spr-0218:3261925] [0] func:/apps/openmpi/5.0.8/lib/libopen-pal.so.80(opal_backtrace_buffer+0x21) [0x1494b7ba97e1]
[gadi-cpu-spr-0218:3261973] [0] func:/apps/openmpi/5.0.8/lib/libopen-pal.so.80(opal_backtrace_buffer+0x21) [0x15363b0847e1]
[gadi-cpu-spr-0218:3261973] [1] func:/apps/openmpi/5.0.8/lib/libmpi.so.40(+0x39eaa) [0x15363b138eaa]
[gadi-cpu-spr-0218:3261973] [2] func:/apps/openmpi/5.0.8/lib/libmpi.so.40(PMPI_Abort+0xdb) [0x15363b14e68b]
[gadi-cpu-spr-0218:3261973] [3] func:/apps/openmpi/5.0.8/lib/libmpi_mpifh_Intel.so.40(pmpi_abort+0x20) [0x15363e8af650]
[gadi-cpu-spr-0218:3261973] [4] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x12e1eb9]
[gadi-cpu-spr-0218:3261973] [5] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x12de910]
[gadi-cpu-spr-0218:3261973] [6] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x41ce07]
[gadi-cpu-spr-0218:3261973] [7] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x7def9c]
[gadi-cpu-spr-0218:3261973] [8] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0xac0420]
[gadi-cpu-spr-0218:3261973] [9] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x813d00]
[gadi-cpu-spr-0218:3261973] [10] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x7414a1]
[gadi-cpu-spr-0218:3261973] [11] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x61b8d2]
[gadi-cpu-spr-0218:3261973] [12] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x4ac81e]
[gadi-cpu-spr-0218:3261973] [13] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x440de3]
[gadi-cpu-spr-0218:3261973] [14] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x416627]
[gadi-cpu-spr-0218:3261973] [15] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x4116e8]
[gadi-cpu-spr-0218:3261973] [16] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x41164d]
[gadi-cpu-spr-0218:3261973] [17] func:/lib64/libc.so.6(__libc_start_main+0xe5) [0x15363d7217e5]
[gadi-cpu-spr-0218:3261973] [18] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x41156e]
[gadi-cpu-spr-0218:3262007] [0] func:/apps/openmpi/5.0.8/lib/libopen-pal.so.80(opal_backtrace_buffer+0x21) [0x151932ae07e1]
[gadi-cpu-spr-0218:3262007] [1] func:/apps/openmpi/5.0.8/lib/libmpi.so.40(+0x39eaa) [0x151932b94eaa]
[gadi-cpu-spr-0218:3262007] [2] func:/apps/openmpi/5.0.8/lib/libmpi.so.40(PMPI_Abort+0xdb) [0x151932baa68b]
[gadi-cpu-spr-0218:3262007] [3] func:/apps/openmpi/5.0.8/lib/libmpi_mpifh_Intel.so.40(pmpi_abort+0x20) [0x15193630b650]
[gadi-cpu-spr-0218:3262007] [4] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x12e1eb9]
[gadi-cpu-spr-0218:3262007] [5] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x12de910]
[gadi-cpu-spr-0218:3262007] [6] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x41ce07]
[gadi-cpu-spr-0218:3262007] [7] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x7def9c]
[gadi-cpu-spr-0218:3262007] [8] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0xac0420]
[gadi-cpu-spr-0218:3262007] [9] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x813d00]
[gadi-cpu-spr-0218:3262007] [10] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x7414a1]
[gadi-cpu-spr-0218:3262007] [11] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x61b8d2]
[gadi-cpu-spr-0218:3262007] [12] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x4ac81e]
[gadi-cpu-spr-0218:3262007] [13] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x440de3]
[gadi-cpu-spr-0218:3262007] [14] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x416627]
[gadi-cpu-spr-0218:3262007] [15] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x4116e8]
[gadi-cpu-spr-0218:3262007] [16] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x41164d]
[gadi-cpu-spr-0218:3262007] [17] func:/lib64/libc.so.6(__libc_start_main+0xe5) [0x15193517d7e5]
[gadi-cpu-spr-0218:3262007] [18] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x41156e]
[gadi-cpu-spr-0218:3261943] [0] func:/apps/openmpi/5.0.8/lib/libopen-pal.so.80(opal_backtrace_buffer+0x21) [0x14f8b26d97e1]
[gadi-cpu-spr-0218:3261943] [1] func:/apps/openmpi/5.0.8/lib/libmpi.so.40(+0x39eaa) [0x14f8b278deaa]
[gadi-cpu-spr-0218:3261943] [2] func:/apps/openmpi/5.0.8/lib/libmpi.so.40(PMPI_Abort+0xdb) [0x14f8b27a368b]
[gadi-cpu-spr-0218:3261943] [3] func:/apps/openmpi/5.0.8/lib/libmpi_mpifh_Intel.so.40(pmpi_abort+0x20) [0x14f8b5f04650]
[gadi-cpu-spr-0218:3261943] [4] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x12e1eb9]
[gadi-cpu-spr-0218:3261943] [5] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x12de910]
[gadi-cpu-spr-0218:3261943] [6] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x41ce07]
[gadi-cpu-spr-0218:3261943] [7] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x7def9c]
[gadi-cpu-spr-0218:3261943] [8] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0xac0420]
[gadi-cpu-spr-0218:3261943] [9] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x813d00]
[gadi-cpu-spr-0218:3261943] [10] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x7414a1]
[gadi-cpu-spr-0218:3261943] [11] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x61b8d2]
[gadi-cpu-spr-0218:3261943] [12] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x4ac81e]
[gadi-cpu-spr-0218:3261943] [13] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x440de3]
[gadi-cpu-spr-0218:3261943] [14] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x416627]
[gadi-cpu-spr-0218:3261943] [15] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x4116e8]
[gadi-cpu-spr-0218:3261943] [16] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x41164d]
[gadi-cpu-spr-0218:3261943] [17] func:/lib64/libc.so.6(__libc_start_main+0xe5) [0x14f8b4d767e5]
[gadi-cpu-spr-0218:3261943] [18] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x41156e]
[gadi-cpu-spr-0218:3261991] [0] func:/apps/openmpi/5.0.8/lib/libopen-pal.so.80(opal_backtrace_buffer+0x21) [0x148bbea677e1]
[gadi-cpu-spr-0218:3261991] [1] func:/apps/openmpi/5.0.8/lib/libmpi.so.40(+0x39eaa) [0x148bbeb1beaa]
[gadi-cpu-spr-0218:3261991] [2] func:/apps/openmpi/5.0.8/lib/libmpi.so.40(PMPI_Abort+0xdb) [0x148bbeb3168b]
[gadi-cpu-spr-0218:3261991] [3] func:/apps/openmpi/5.0.8/lib/libmpi_mpifh_Intel.so.40(pmpi_abort+0x20) [0x148bc2292650]
[gadi-cpu-spr-0218:3261991] [4] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x12e1eb9]
[gadi-cpu-spr-0218:3261991] [5] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x12de910]
[gadi-cpu-spr-0218:3261991] [6] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x41ce07]
[gadi-cpu-spr-0218:3261991] [7] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x7def9c]
[gadi-cpu-spr-0218:3261991] [8] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0xac0420]
[gadi-cpu-spr-0218:3261991] [9] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x813d00]
[gadi-cpu-spr-0218:3261991] [10] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x7414a1]
[gadi-cpu-spr-0218:3261991] [11] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x61b8d2]
[gadi-cpu-spr-0218:3261991] [12] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x4ac81e]
[gadi-cpu-spr-0218:3261991] [13] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x440de3]
[gadi-cpu-spr-0218:3261991] [14] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x416627]
[gadi-cpu-spr-0218:3261991] [15] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x4116e8]
[gadi-cpu-spr-0218:3261991] [16] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x41164d]
[gadi-cpu-spr-0218:3261991] [17] func:/lib64/libc.so.6(__libc_start_main+0xe5) [0x148bc11047e5]
[gadi-cpu-spr-0218:3261991] [18] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x41156e]
[gadi-cpu-spr-0218:3261961] [0] func:/apps/openmpi/5.0.8/lib/libopen-pal.so.80(opal_backtrace_buffer+0x21) [0x149dc77cf7e1]
[gadi-cpu-spr-0218:3261961] [1] func:/apps/openmpi/5.0.8/lib/libmpi.so.40(+0x39eaa) [0x149dc7883eaa]
[gadi-cpu-spr-0218:3261961] [2] func:/apps/openmpi/5.0.8/lib/libmpi.so.40(PMPI_Abort+0xdb) [0x149dc789968b]
[gadi-cpu-spr-0218:3261961] [3] func:/apps/openmpi/5.0.8/lib/libmpi_mpifh_Intel.so.40(pmpi_abort+0x20) [0x149dcaffa650]
[gadi-cpu-spr-0218:3261961] [4] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x12e1eb9]
[gadi-cpu-spr-0218:3261961] [5] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x12de910]
[gadi-cpu-spr-0218:3261961] [6] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x41ce07]
[gadi-cpu-spr-0218:3261961] [7] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x7def9c]
[gadi-cpu-spr-0218:3261961] [8] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0xac0420]
[gadi-cpu-spr-0218:3261961] [9] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x813d00]
[gadi-cpu-spr-0218:3261961] [10] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x7414a1]
[gadi-cpu-spr-0218:3261961] [11] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x61b8d2]
[gadi-cpu-spr-0218:3261961] [12] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x4ac81e]
[gadi-cpu-spr-0218:3261961] [13] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x440de3]
[gadi-cpu-spr-0218:3261961] [14] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x416627]
[gadi-cpu-spr-0218:3261961] [15] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x4116e8]
[gadi-cpu-spr-0218:3261961] [16] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x41164d]
[gadi-cpu-spr-0218:3261961] [17] func:/lib64/libc.so.6(__libc_start_main+0xe5) [0x149dc9e6c7e5]
[gadi-cpu-spr-0218:3261961] [18] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x41156e]
[gadi-cpu-spr-0218:3261925] [1] func:/apps/openmpi/5.0.8/lib/libmpi.so.40(+0x39eaa) [0x1494b7c5deaa]
[gadi-cpu-spr-0218:3261925] [2] func:/apps/openmpi/5.0.8/lib/libmpi.so.40(PMPI_Abort+0xdb) [0x1494b7c7368b]
[gadi-cpu-spr-0218:3261925] [3] func:/apps/openmpi/5.0.8/lib/libmpi_mpifh_Intel.so.40(pmpi_abort+0x20) [0x1494bb3d4650]
[gadi-cpu-spr-0218:3261925] [4] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x12e1eb9]
[gadi-cpu-spr-0218:3261925] [5] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x12de910]
[gadi-cpu-spr-0218:3261925] [6] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x41ce07]
[gadi-cpu-spr-0218:3261925] [7] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x7def9c]
[gadi-cpu-spr-0218:3261925] [8] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0xac0420]
[gadi-cpu-spr-0218:3261925] [9] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x813d00]
[gadi-cpu-spr-0218:3261925] [10] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x7414a1]
[gadi-cpu-spr-0218:3261925] [11] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x61b8d2]
[gadi-cpu-spr-0218:3261925] [12] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x4ac81e]
[gadi-cpu-spr-0218:3261925] [13] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x440de3]
[gadi-cpu-spr-0218:3261925] [14] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x416627]
[gadi-cpu-spr-0218:3261925] [15] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x4116e8]
[gadi-cpu-spr-0218:3261925] [16] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x41164d]
[gadi-cpu-spr-0218:3261925] [17] func:/lib64/libc.so.6(__libc_start_main+0xe5) [0x1494ba2467e5]
[gadi-cpu-spr-0218:3261925] [18] func:/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe() [0x41156e]
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source             
libpthread-2.28.s  00001494BA7F9990  Unknown               Unknown  Unknown
libpthread-2.28.s  00001494BA7F536F  pthread_cond_wait     Unknown  Unknown
libpmix.so.2.18.0  00001494B51EAB00  PMIx_Abort            Unknown  Unknown
libmpi.so.40.40.7  00001494B7C625CB  ompi_rte_abort        Unknown  Unknown
libmpi.so.40.40.7  00001494B7C5DE91  Unknown               Unknown  Unknown
libmpi.so.40.40.7  00001494B7C7368B  PMPI_Abort            Unknown  Unknown
libmpi_mpifh_Inte  00001494BB3D4650  pmpi_abort            Unknown  Unknown
um_hg3.exe         00000000012E1EB9  mpl_abort                  53  mpl_abort.F90
um_hg3.exe         00000000012DE910  gc_abort                  131  gc_abort.F90
um_hg3.exe         000000000041CE07  ereport                   384  ereport.f90
um_hg3.exe         00000000007DEF9C  bi_linear_h               581  bi_linear_h.f90
um_hg3.exe         0000000000AC0420  ritchie                  2557  ritchie.f90
um_hg3.exe         0000000000813D00  departure_point           378  departure_point.f90
um_hg3.exe         00000000007414A1  sl_thermo                 678  sl_thermo.f90
um_hg3.exe         000000000061B8D2  ni_sl_thermo              664  ni_sl_thermo.f90
um_hg3.exe         00000000004AC81E  Unknown               Unknown  Unknown
um_hg3.exe         0000000000440DE3  Unknown               Unknown  Unknown
um_hg3.exe         0000000000416627  um_shell                 3920  um_shell.f90
um_hg3.exe         00000000004116E8  flumemain                  29  flumeMain.f90
um_hg3.exe         000000000041164D  Unknown               Unknown  Unknown
libc-2.28.so       00001494BA2467E5  __libc_start_main     Unknown  Unknown
um_hg3.exe         000000000041156E  Unknown               Unknown  Unknown

@anton-seaice
Copy link
Copy Markdown
Collaborator

!test repro

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Feb 1, 2026

⚠️ The Bitwise Reproducibility Check Had Errors - Check https://github.com/ACCESS-NRI/access-esm1.6-configs/actions/runs/21571118892 ⚠️
❌ The Bitwise Reproducibility Check Failed ❌

When comparing:

  • 402-OASIS3-mct-update (checksums created using commit 1325982), against
  • dev-preindustrial+concentrations (checksums in commit d5be642)
Further information

The experiment can be found on Gadi at /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae, and the test results at https://github.com/ACCESS-NRI/access-esm1.6-configs/runs/62150503441.

The checksums generated by this !test command are found in the testing/checksum directory of https://github.com/ACCESS-NRI/access-esm1.6-configs/actions/runs/21571118892/artifacts/5336205234.

The checksums compared against are found here https://github.com/ACCESS-NRI/access-esm1.6-configs/tree/d5be642259e4b502b0cf79ed91ee3a21e0ce20e4/testing/checksum

Test summary:
🔥 test_repro_historical
🔥 test_repro_determinism

@anton-seaice
Copy link
Copy Markdown
Collaborator

!test repro

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Feb 1, 2026

⚠️ The Bitwise Reproducibility Check Had Errors - Check https://github.com/ACCESS-NRI/access-esm1.6-configs/actions/runs/21571881972 ⚠️
❌ The Bitwise Reproducibility Check Failed ❌

When comparing:

  • 402-OASIS3-mct-update (checksums created using commit 2464299), against
  • dev-preindustrial+concentrations (checksums in commit d5be642)
Further information

The experiment can be found on Gadi at /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/2464299f815e8d3de003d1f8eca6eaded8904fd3, and the test results at https://github.com/ACCESS-NRI/access-esm1.6-configs/runs/62152325415.

The checksums generated by this !test command are found in the testing/checksum directory of https://github.com/ACCESS-NRI/access-esm1.6-configs/actions/runs/21571881972/artifacts/5336422335.

The checksums compared against are found here https://github.com/ACCESS-NRI/access-esm1.6-configs/tree/d5be642259e4b502b0cf79ed91ee3a21e0ce20e4/testing/checksum

Test summary:
🔥 test_repro_historical
🔥 test_repro_determinism

@anton-seaice
Copy link
Copy Markdown
Collaborator

!test repro

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Feb 1, 2026

✅ The Bitwise Reproducibility Check Succeeded ✅

When comparing:

  • 402-OASIS3-mct-update (checksums created using commit 0192ef5), against
  • dev-preindustrial+concentrations (checksums in commit d5be642)
Further information

The experiment can be found on Gadi at /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/0192ef5451f3814bd7d3590564452f56b6bdbee6, and the test results at https://github.com/ACCESS-NRI/access-esm1.6-configs/runs/62152675539.

The checksums generated by this !test command are found in the testing/checksum directory of https://github.com/ACCESS-NRI/access-esm1.6-configs/actions/runs/21571966317/artifacts/5336461680.

The checksums compared against are found here https://github.com/ACCESS-NRI/access-esm1.6-configs/tree/d5be642259e4b502b0cf79ed91ee3a21e0ce20e4/testing/checksum

Test summary:
test_repro_historical
test_repro_determinism

@aidanheerdegen
Copy link
Copy Markdown
Member Author

!test repro

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Feb 1, 2026

✅ The Bitwise Reproducibility Check Succeeded ✅

When comparing:

  • 402-OASIS3-mct-update (checksums created using commit 390d573), against
  • dev-preindustrial+concentrations (checksums in commit d5be642)
Further information

The experiment can be found on Gadi at /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/390d5731e8c615f3535160fec167ba36852446f3, and the test results at https://github.com/ACCESS-NRI/access-esm1.6-configs/runs/62154389904.

The checksums generated by this !test command are found in the testing/checksum directory of https://github.com/ACCESS-NRI/access-esm1.6-configs/actions/runs/21572585531/artifacts/5336663976.

The checksums compared against are found here https://github.com/ACCESS-NRI/access-esm1.6-configs/tree/d5be642259e4b502b0cf79ed91ee3a21e0ce20e4/testing/checksum

Test summary:
test_repro_historical
test_repro_determinism

@aidanheerdegen
Copy link
Copy Markdown
Member Author

!test repro

@aidanheerdegen
Copy link
Copy Markdown
Member Author

!test repro

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Feb 3, 2026

✅ The Bitwise Reproducibility Check Succeeded ✅

When comparing:

  • 402-OASIS3-mct-update (checksums created using commit 71e57fd), against
  • dev-preindustrial+concentrations (checksums in commit d5be642)
Further information

The experiment can be found on Gadi at /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/71e57fd8afba2dcf25db2f60bf3e6b67f3222dbc, and the test results at https://github.com/ACCESS-NRI/access-esm1.6-configs/runs/62417700387.

The checksums generated by this !test command are found in the testing/checksum directory of https://github.com/ACCESS-NRI/access-esm1.6-configs/actions/runs/21651604894/artifacts/5366438520.

The checksums compared against are found here https://github.com/ACCESS-NRI/access-esm1.6-configs/tree/d5be642259e4b502b0cf79ed91ee3a21e0ce20e4/testing/checksum

Test summary:
test_repro_historical
test_repro_determinism

@aidanheerdegen aidanheerdegen deleted the 402-OASIS3-mct-update branch February 3, 2026 23:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants