failed on setup with "RuntimeError: There was an error running experiment exp_default_runtime: Payu run job failed with exit status 1:
Job_ID: 160198384.gadi-pbs
Output files: ['/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/control/exp_default_runtime/pre-industrial.o160198384', '/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/control/exp_default_runtime/pre-industrial.e160198384']
--- stdout ---
laboratory path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab
binary path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/bin
input path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/input
work path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/work
archive path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/archive
Experiment name is configured in config.yaml:  exp_default_runtime
nruns: 1 nruns_per_submit: 1 subrun: 1
payu: Found modules in /opt/Modules/v4.3.0
Loading input manifest: manifests/input.yaml
Loading restart manifest: manifests/restart.yaml
Loading exe manifest: manifests/exe.yaml
Setting up atmosphere
Setting up ocean
Setting up ice
Setting up coupler
Checking exe, input and restart manifests
mod oasis3-mct/5.2-dzwdenl
mod cice5/2026.01.000-ikbjpb4
mod access-fms/mom5-2025.08.000-ltn3e2q
mod access-mocsy/2025.07.002-ejsylbt
mod access-generic-tracers/2025.09.000-x2czqdx
mod mom5/2025.05.000-jotcb6x
mod cable/2025.11.000-7ey6m4f
mod gcom4/2025.08.000-qhkz6ut
mod um7/2025.12.000-d54mlw4
mod access-esm1p6/2026.01.000
mpirun --mca mpi_yield_when_idle true -wdir /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/work/exp_default_runtime/atmosphere -np 256  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/work/exp_default_runtime/atmosphere/um_hg3.exe : -wdir /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/work/exp_default_runtime/ocean -np 240  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/work/exp_default_runtime/ocean/mom5_access_cm : -wdir /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/work/exp_default_runtime/ice -np 12  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/work/exp_default_runtime/ice/cice_access_360x300_30x300.exe
/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/control/exp_default_runtime/access-esm1.6.err /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/archive/exp_default_runtime/error_logs/access-esm1.6.160198384.gadi-pbs.err
/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/control/exp_default_runtime/access-esm1.6.out /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/archive/exp_default_runtime/error_logs/access-esm1.6.160198384.gadi-pbs.out
/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/control/exp_default_runtime/env.yaml /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/archive/exp_default_runtime/error_logs/env.160198384.gadi-pbs.yaml

======================================================================================
                  Resource Usage on 2026-02-09 11:15:30:
   Job Id:             160198384.gadi-pbs
   Project:            tm70
   Exit Status:        1
   Service Units:      10.98
   NCPUs Requested:    520                 CPU Time Used: 01:51:54        
   Memory Requested:   2.5TB                 Memory Used: 62.85GB         
   Walltime Requested: 02:30:00            Walltime Used: 00:00:38        
   JobFS Requested:    1.46GB                 JobFS Used: 0B              
======================================================================================

--- stderr ---
Loading access-esm1p6/2026.01.000
  Loading requirement: oasis3-mct/5.2-dzwdenl cice5/2026.01.000-ikbjpb4
    access-fms/mom5-2025.08.000-ltn3e2q access-mocsy/2025.07.002-ejsylbt
    access-generic-tracers/2025.09.000-x2czqdx mom5/2025.05.000-jotcb6x
    cable/2025.11.000-7ey6m4f gcom4/2025.08.000-qhkz6ut um7/2025.12.000-d54mlw4
Currently Loaded Modulefiles:
 1) oasis3-mct/5.2-dzwdenl                       7) cable/2025.11.000-7ey6m4f  
 2) cice5/2026.01.000-ikbjpb4                    8) gcom4/2025.08.000-qhkz6ut  
 3) access-fms/mom5-2025.08.000-ltn3e2q          9) um7/2025.12.000-d54mlw4    
 4) access-mocsy/2025.07.002-ejsylbt            10) access-esm1p6/2026.01.000  
 5) access-generic-tracers/2025.09.000-x2czqdx  11) pbs                        
 6) mom5/2025.05.000-jotcb6x                    12) openmpi/5.0.8              
payu: Model exited with error code 64; aborting."
request = <SubRequest 'requested_experiments' for <Function test_repro_historical>>
experiments = <model_config_tests.exp_test_helper.Experiments object at 0x7f6de8580d30>

    @pytest.fixture
    def requested_experiments(request, experiments: Experiments):
        """Fixture to check that requested experiments have run successfully
        and return a dictionary of ExpTestHelper instances for each experiment."""
        exp_marker = request.node.get_closest_marker("experiments").args[0]
        requested_exps = {}
        for exp_name in exp_marker:
            # Check experiment has run successfully - this will raise an
            # error if there are any non-zero exit codes in the outputs
>           experiments.check_experiment(exp_name)

../test-venv/lib/python3.10/site-packages/model_config_tests/config_tests/test_bit_reproducibility.py:159: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <model_config_tests.exp_test_helper.Experiments object at 0x7f6de8580d30>
exp_name = 'exp_default_runtime'

    def check_experiment(self, exp_name: str) -> None:
        """
        Check whether given experiment name has run successfully
        """
        if exp_name in self.experiment_errors:
>           raise RuntimeError(
                f"There was an error running experiment {exp_name}:"
                f" {self.experiment_errors[exp_name]}"
            )
E           RuntimeError: There was an error running experiment exp_default_runtime: Payu run job failed with exit status 1:
E           Job_ID: 160198384.gadi-pbs
E           Output files: ['/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/control/exp_default_runtime/pre-industrial.o160198384', '/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/control/exp_default_runtime/pre-industrial.e160198384']
E           --- stdout ---
E           laboratory path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab
E           binary path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/bin
E           input path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/input
E           work path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/work
E           archive path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/archive
E           Experiment name is configured in config.yaml:  exp_default_runtime
E           nruns: 1 nruns_per_submit: 1 subrun: 1
E           payu: Found modules in /opt/Modules/v4.3.0
E           Loading input manifest: manifests/input.yaml
E           Loading restart manifest: manifests/restart.yaml
E           Loading exe manifest: manifests/exe.yaml
E           Setting up atmosphere
E           Setting up ocean
E           Setting up ice
E           Setting up coupler
E           Checking exe, input and restart manifests
E           mod oasis3-mct/5.2-dzwdenl
E           mod cice5/2026.01.000-ikbjpb4
E           mod access-fms/mom5-2025.08.000-ltn3e2q
E           mod access-mocsy/2025.07.002-ejsylbt
E           mod access-generic-tracers/2025.09.000-x2czqdx
E           mod mom5/2025.05.000-jotcb6x
E           mod cable/2025.11.000-7ey6m4f
E           mod gcom4/2025.08.000-qhkz6ut
E           mod um7/2025.12.000-d54mlw4
E           mod access-esm1p6/2026.01.000
E           mpirun --mca mpi_yield_when_idle true -wdir /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/work/exp_default_runtime/atmosphere -np 256  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/work/exp_default_runtime/atmosphere/um_hg3.exe : -wdir /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/work/exp_default_runtime/ocean -np 240  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/work/exp_default_runtime/ocean/mom5_access_cm : -wdir /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/work/exp_default_runtime/ice -np 12  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/work/exp_default_runtime/ice/cice_access_360x300_30x300.exe
E           /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/control/exp_default_runtime/access-esm1.6.err /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/archive/exp_default_runtime/error_logs/access-esm1.6.160198384.gadi-pbs.err
E           /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/control/exp_default_runtime/access-esm1.6.out /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/archive/exp_default_runtime/error_logs/access-esm1.6.160198384.gadi-pbs.out
E           /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/control/exp_default_runtime/env.yaml /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/archive/exp_default_runtime/error_logs/env.160198384.gadi-pbs.yaml
E           
E           ======================================================================================
E                             Resource Usage on 2026-02-09 11:15:30:
E              Job Id:             160198384.gadi-pbs
E              Project:            tm70
E              Exit Status:        1
E              Service Units:      10.98
E              NCPUs Requested:    520                 CPU Time Used: 01:51:54        
E              Memory Requested:   2.5TB                 Memory Used: 62.85GB         
E              Walltime Requested: 02:30:00            Walltime Used: 00:00:38        
E              JobFS Requested:    1.46GB                 JobFS Used: 0B              
E           ======================================================================================
E           
E           --- stderr ---
E           Loading access-esm1p6/2026.01.000
E             Loading requirement: oasis3-mct/5.2-dzwdenl cice5/2026.01.000-ikbjpb4
E               access-fms/mom5-2025.08.000-ltn3e2q access-mocsy/2025.07.002-ejsylbt
E               access-generic-tracers/2025.09.000-x2czqdx mom5/2025.05.000-jotcb6x
E               cable/2025.11.000-7ey6m4f gcom4/2025.08.000-qhkz6ut um7/2025.12.000-d54mlw4
E           Currently Loaded Modulefiles:
E            1) oasis3-mct/5.2-dzwdenl                       7) cable/2025.11.000-7ey6m4f  
E            2) cice5/2026.01.000-ikbjpb4                    8) gcom4/2025.08.000-qhkz6ut  
E            3) access-fms/mom5-2025.08.000-ltn3e2q          9) um7/2025.12.000-d54mlw4    
E            4) access-mocsy/2025.07.002-ejsylbt            10) access-esm1p6/2026.01.000  
E            5) access-generic-tracers/2025.09.000-x2czqdx  11) pbs                        
E            6) mom5/2025.05.000-jotcb6x                    12) openmpi/5.0.8              
E           payu: Model exited with error code 64; aborting.

../test-venv/lib/python3.10/site-packages/model_config_tests/exp_test_helper.py:318: RuntimeError

failed on setup with "RuntimeError: There was an error running experiment exp_1d_runtime: Payu run job failed with exit status 1:
Job_ID: 160198391.gadi-pbs
Output files: ['/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/control/exp_1d_runtime/pre-industrial.o160198391', '/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/control/exp_1d_runtime/pre-industrial.e160198391']
--- stdout ---
laboratory path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab
binary path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/bin
input path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/input
work path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/work
archive path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/archive
Experiment name is configured in config.yaml:  exp_1d_runtime
nruns: 1 nruns_per_submit: 1 subrun: 1
payu: Found modules in /opt/Modules/v4.3.0
Loading input manifest: manifests/input.yaml
Loading restart manifest: manifests/restart.yaml
Loading exe manifest: manifests/exe.yaml
Setting up atmosphere
Setting up ocean
Setting up ice
Setting up coupler
Checking exe, input and restart manifests
mod oasis3-mct/5.2-dzwdenl
mod cice5/2026.01.000-ikbjpb4
mod access-fms/mom5-2025.08.000-ltn3e2q
mod access-mocsy/2025.07.002-ejsylbt
mod access-generic-tracers/2025.09.000-x2czqdx
mod mom5/2025.05.000-jotcb6x
mod cable/2025.11.000-7ey6m4f
mod gcom4/2025.08.000-qhkz6ut
mod um7/2025.12.000-d54mlw4
mod access-esm1p6/2026.01.000
mpirun --mca mpi_yield_when_idle true -wdir /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/work/exp_1d_runtime/atmosphere -np 256  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/work/exp_1d_runtime/atmosphere/um_hg3.exe : -wdir /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/work/exp_1d_runtime/ocean -np 240  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/work/exp_1d_runtime/ocean/mom5_access_cm : -wdir /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/work/exp_1d_runtime/ice -np 12  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/work/exp_1d_runtime/ice/cice_access_360x300_30x300.exe
/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/control/exp_1d_runtime/access-esm1.6.err /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/archive/exp_1d_runtime/error_logs/access-esm1.6.160198391.gadi-pbs.err
/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/control/exp_1d_runtime/access-esm1.6.out /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/archive/exp_1d_runtime/error_logs/access-esm1.6.160198391.gadi-pbs.out
/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/control/exp_1d_runtime/env.yaml /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/archive/exp_1d_runtime/error_logs/env.160198391.gadi-pbs.yaml

======================================================================================
                  Resource Usage on 2026-02-09 11:15:30:
   Job Id:             160198391.gadi-pbs
   Project:            tm70
   Exit Status:        1
   Service Units:      11.27
   NCPUs Requested:    520                 CPU Time Used: 01:50:46        
   Memory Requested:   2.5TB                 Memory Used: 63.38GB         
   Walltime Requested: 02:30:00            Walltime Used: 00:00:39        
   JobFS Requested:    1.46GB                 JobFS Used: 0B              
======================================================================================

--- stderr ---
Loading access-esm1p6/2026.01.000
  Loading requirement: oasis3-mct/5.2-dzwdenl cice5/2026.01.000-ikbjpb4
    access-fms/mom5-2025.08.000-ltn3e2q access-mocsy/2025.07.002-ejsylbt
    access-generic-tracers/2025.09.000-x2czqdx mom5/2025.05.000-jotcb6x
    cable/2025.11.000-7ey6m4f gcom4/2025.08.000-qhkz6ut um7/2025.12.000-d54mlw4
Currently Loaded Modulefiles:
 1) oasis3-mct/5.2-dzwdenl                       7) cable/2025.11.000-7ey6m4f  
 2) cice5/2026.01.000-ikbjpb4                    8) gcom4/2025.08.000-qhkz6ut  
 3) access-fms/mom5-2025.08.000-ltn3e2q          9) um7/2025.12.000-d54mlw4    
 4) access-mocsy/2025.07.002-ejsylbt            10) access-esm1p6/2026.01.000  
 5) access-generic-tracers/2025.09.000-x2czqdx  11) pbs                        
 6) mom5/2025.05.000-jotcb6x                    12) openmpi/5.0.8              
payu: Model exited with error code 64; aborting."
request = <SubRequest 'requested_experiments' for <Function test_repro_determinism>>
experiments = <model_config_tests.exp_test_helper.Experiments object at 0x7f6de8580d30>

    @pytest.fixture
    def requested_experiments(request, experiments: Experiments):
        """Fixture to check that requested experiments have run successfully
        and return a dictionary of ExpTestHelper instances for each experiment."""
        exp_marker = request.node.get_closest_marker("experiments").args[0]
        requested_exps = {}
        for exp_name in exp_marker:
            # Check experiment has run successfully - this will raise an
            # error if there are any non-zero exit codes in the outputs
>           experiments.check_experiment(exp_name)

../test-venv/lib/python3.10/site-packages/model_config_tests/config_tests/test_bit_reproducibility.py:159: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <model_config_tests.exp_test_helper.Experiments object at 0x7f6de8580d30>
exp_name = 'exp_1d_runtime'

    def check_experiment(self, exp_name: str) -> None:
        """
        Check whether given experiment name has run successfully
        """
        if exp_name in self.experiment_errors:
>           raise RuntimeError(
                f"There was an error running experiment {exp_name}:"
                f" {self.experiment_errors[exp_name]}"
            )
E           RuntimeError: There was an error running experiment exp_1d_runtime: Payu run job failed with exit status 1:
E           Job_ID: 160198391.gadi-pbs
E           Output files: ['/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/control/exp_1d_runtime/pre-industrial.o160198391', '/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/control/exp_1d_runtime/pre-industrial.e160198391']
E           --- stdout ---
E           laboratory path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab
E           binary path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/bin
E           input path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/input
E           work path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/work
E           archive path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/archive
E           Experiment name is configured in config.yaml:  exp_1d_runtime
E           nruns: 1 nruns_per_submit: 1 subrun: 1
E           payu: Found modules in /opt/Modules/v4.3.0
E           Loading input manifest: manifests/input.yaml
E           Loading restart manifest: manifests/restart.yaml
E           Loading exe manifest: manifests/exe.yaml
E           Setting up atmosphere
E           Setting up ocean
E           Setting up ice
E           Setting up coupler
E           Checking exe, input and restart manifests
E           mod oasis3-mct/5.2-dzwdenl
E           mod cice5/2026.01.000-ikbjpb4
E           mod access-fms/mom5-2025.08.000-ltn3e2q
E           mod access-mocsy/2025.07.002-ejsylbt
E           mod access-generic-tracers/2025.09.000-x2czqdx
E           mod mom5/2025.05.000-jotcb6x
E           mod cable/2025.11.000-7ey6m4f
E           mod gcom4/2025.08.000-qhkz6ut
E           mod um7/2025.12.000-d54mlw4
E           mod access-esm1p6/2026.01.000
E           mpirun --mca mpi_yield_when_idle true -wdir /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/work/exp_1d_runtime/atmosphere -np 256  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/work/exp_1d_runtime/atmosphere/um_hg3.exe : -wdir /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/work/exp_1d_runtime/ocean -np 240  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/work/exp_1d_runtime/ocean/mom5_access_cm : -wdir /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/work/exp_1d_runtime/ice -np 12  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/work/exp_1d_runtime/ice/cice_access_360x300_30x300.exe
E           /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/control/exp_1d_runtime/access-esm1.6.err /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/archive/exp_1d_runtime/error_logs/access-esm1.6.160198391.gadi-pbs.err
E           /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/control/exp_1d_runtime/access-esm1.6.out /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/archive/exp_1d_runtime/error_logs/access-esm1.6.160198391.gadi-pbs.out
E           /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/control/exp_1d_runtime/env.yaml /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/103f4710e3c3bcda4ea0322857da41dcb4a89387/lab/archive/exp_1d_runtime/error_logs/env.160198391.gadi-pbs.yaml
E           
E           ======================================================================================
E                             Resource Usage on 2026-02-09 11:15:30:
E              Job Id:             160198391.gadi-pbs
E              Project:            tm70
E              Exit Status:        1
E              Service Units:      11.27
E              NCPUs Requested:    520                 CPU Time Used: 01:50:46        
E              Memory Requested:   2.5TB                 Memory Used: 63.38GB         
E              Walltime Requested: 02:30:00            Walltime Used: 00:00:39        
E              JobFS Requested:    1.46GB                 JobFS Used: 0B              
E           ======================================================================================
E           
E           --- stderr ---
E           Loading access-esm1p6/2026.01.000
E             Loading requirement: oasis3-mct/5.2-dzwdenl cice5/2026.01.000-ikbjpb4
E               access-fms/mom5-2025.08.000-ltn3e2q access-mocsy/2025.07.002-ejsylbt
E               access-generic-tracers/2025.09.000-x2czqdx mom5/2025.05.000-jotcb6x
E               cable/2025.11.000-7ey6m4f gcom4/2025.08.000-qhkz6ut um7/2025.12.000-d54mlw4
E           Currently Loaded Modulefiles:
E            1) oasis3-mct/5.2-dzwdenl                       7) cable/2025.11.000-7ey6m4f  
E            2) cice5/2026.01.000-ikbjpb4                    8) gcom4/2025.08.000-qhkz6ut  
E            3) access-fms/mom5-2025.08.000-ltn3e2q          9) um7/2025.12.000-d54mlw4    
E            4) access-mocsy/2025.07.002-ejsylbt            10) access-esm1p6/2026.01.000  
E            5) access-generic-tracers/2025.09.000-x2czqdx  11) pbs                        
E            6) mom5/2025.05.000-jotcb6x                    12) openmpi/5.0.8              
E           payu: Model exited with error code 64; aborting.

../test-venv/lib/python3.10/site-packages/model_config_tests/exp_test_helper.py:318: RuntimeError