failed on setup with "RuntimeError: There was an error running experiment exp_default_runtime: Payu run job failed with exit status 1:
Job_ID: 159728614.gadi-pbs
Output files: ['/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/control/exp_default_runtime/pre-industrial.o159728614', '/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/control/exp_default_runtime/pre-industrial.e159728614']
--- stdout ---
laboratory path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab
binary path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/bin
input path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/input
work path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work
archive path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/archive
Experiment name is configured in config.yaml:  exp_default_runtime
nruns: 1 nruns_per_submit: 1 subrun: 1
payu: Found modules in /opt/Modules/v4.3.0
Loading input manifest: manifests/input.yaml
Loading restart manifest: manifests/restart.yaml
Loading exe manifest: manifests/exe.yaml
Setting up atmosphere
Setting up ocean
Setting up ice
Setting up coupler
Checking exe, input and restart manifests
mod access-esm1p6/dependencies/pr181-9/oasis3-mct/5.2-dzwdenl
mod access-esm1p6/dependencies/pr181-9/cice5/2026.01.000-3ecdmyk
mod access-esm1p6/dependencies/pr181-9/access-fms/mom5-2025.08.000-ltn3e2q
mod access-esm1p6/dependencies/pr181-9/access-mocsy/2025.07.002-ejsylbt
mod access-esm1p6/dependencies/pr181-9/access-generic-tracers/2025.09.000-x2czqdx
mod access-esm1p6/dependencies/pr181-9/mom5/2025.05.000-jotcb6x
mod access-esm1p6/dependencies/pr181-9/cable/2025.11.000-7ey6m4f
mod access-esm1p6/dependencies/pr181-9/gcom4/2025.08.000-qhkz6ut
mod access-esm1p6/dependencies/pr181-9/um7/2025.12.000-oz4papa
mod access-esm1p6/pr181-9
mpirun --mca mpi_yield_when_idle true -wdir /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere -np 256  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe : -wdir /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/ocean -np 240  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/ocean/mom5_access_cm : -wdir /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/ice -np 12  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/ice/cice_access_360x300_36x300.exe
/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/control/exp_default_runtime/access-esm1.6.err /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/archive/exp_default_runtime/error_logs/access-esm1.6.159728614.gadi-pbs.err
/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/control/exp_default_runtime/access-esm1.6.out /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/archive/exp_default_runtime/error_logs/access-esm1.6.159728614.gadi-pbs.out
/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/control/exp_default_runtime/env.yaml /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/archive/exp_default_runtime/error_logs/env.159728614.gadi-pbs.yaml

======================================================================================
                  Resource Usage on 2026-02-02 09:03:48:
   Job Id:             159728614.gadi-pbs
   Project:            tm70
   Exit Status:        1
   Service Units:      27.16
   NCPUs Requested:    520                 CPU Time Used: 07:12:43        
   Memory Requested:   2.5TB                 Memory Used: 190.14GB        
   Walltime Requested: 02:30:00            Walltime Used: 00:01:34        
   JobFS Requested:    1.46GB                 JobFS Used: 0B              
======================================================================================

--- stderr ---
Loading access-esm1p6/pr181-9
  Loading requirement: access-esm1p6/dependencies/pr181-9/oasis3-mct/5.2-dzwdenl
    access-esm1p6/dependencies/pr181-9/cice5/2026.01.000-3ecdmyk
    access-esm1p6/dependencies/pr181-9/access-fms/mom5-2025.08.000-ltn3e2q
    access-esm1p6/dependencies/pr181-9/access-mocsy/2025.07.002-ejsylbt
    access-esm1p6/dependencies/pr181-9/access-generic-tracers/2025.09.000-x2czqdx access-esm1p6/dependencies/pr181-9/mom5/2025.05.000-jotcb6x access-esm1p6/dependencies/pr181-9/cable/2025.11.000-7ey6m4f access-esm1p6/dependencies/pr181-9/gcom4/2025.08.000-qhkz6ut access-esm1p6/dependencies/pr181-9/um7/2025.12.000-oz4papa
Currently Loaded Modulefiles:
 1) access-esm1p6/dependencies/pr181-9/oasis3-mct/5.2-dzwdenl                      
 2) access-esm1p6/dependencies/pr181-9/cice5/2026.01.000-3ecdmyk                   
 3) access-esm1p6/dependencies/pr181-9/access-fms/mom5-2025.08.000-ltn3e2q         
 4) access-esm1p6/dependencies/pr181-9/access-mocsy/2025.07.002-ejsylbt            
 5) access-esm1p6/dependencies/pr181-9/access-generic-tracers/2025.09.000-x2czqdx  
 6) access-esm1p6/dependencies/pr181-9/mom5/2025.05.000-jotcb6x                    
 7) access-esm1p6/dependencies/pr181-9/cable/2025.11.000-7ey6m4f                   
 8) access-esm1p6/dependencies/pr181-9/gcom4/2025.08.000-qhkz6ut                   
 9) access-esm1p6/dependencies/pr181-9/um7/2025.12.000-oz4papa                     
10) access-esm1p6/pr181-9                                                          
11) pbs                                                                            
12) openmpi/5.0.8                                                                  
payu: Model exited with error code 9; aborting."
request = <SubRequest 'requested_experiments' for <Function test_repro_historical>>
experiments = <model_config_tests.exp_test_helper.Experiments object at 0x7f6ca0ba8ee0>

    @pytest.fixture
    def requested_experiments(request, experiments: Experiments):
        """Fixture to check that requested experiments have run successfully
        and return a dictionary of ExpTestHelper instances for each experiment."""
        exp_marker = request.node.get_closest_marker("experiments").args[0]
        requested_exps = {}
        for exp_name in exp_marker:
            # Check experiment has run successfully - this will raise an
            # error if there are any non-zero exit codes in the outputs
>           experiments.check_experiment(exp_name)

../test-venv/lib/python3.10/site-packages/model_config_tests/config_tests/test_bit_reproducibility.py:159: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <model_config_tests.exp_test_helper.Experiments object at 0x7f6ca0ba8ee0>
exp_name = 'exp_default_runtime'

    def check_experiment(self, exp_name: str) -> None:
        """
        Check whether given experiment name has run successfully
        """
        if exp_name in self.experiment_errors:
>           raise RuntimeError(
                f"There was an error running experiment {exp_name}:"
                f" {self.experiment_errors[exp_name]}"
            )
E           RuntimeError: There was an error running experiment exp_default_runtime: Payu run job failed with exit status 1:
E           Job_ID: 159728614.gadi-pbs
E           Output files: ['/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/control/exp_default_runtime/pre-industrial.o159728614', '/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/control/exp_default_runtime/pre-industrial.e159728614']
E           --- stdout ---
E           laboratory path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab
E           binary path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/bin
E           input path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/input
E           work path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work
E           archive path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/archive
E           Experiment name is configured in config.yaml:  exp_default_runtime
E           nruns: 1 nruns_per_submit: 1 subrun: 1
E           payu: Found modules in /opt/Modules/v4.3.0
E           Loading input manifest: manifests/input.yaml
E           Loading restart manifest: manifests/restart.yaml
E           Loading exe manifest: manifests/exe.yaml
E           Setting up atmosphere
E           Setting up ocean
E           Setting up ice
E           Setting up coupler
E           Checking exe, input and restart manifests
E           mod access-esm1p6/dependencies/pr181-9/oasis3-mct/5.2-dzwdenl
E           mod access-esm1p6/dependencies/pr181-9/cice5/2026.01.000-3ecdmyk
E           mod access-esm1p6/dependencies/pr181-9/access-fms/mom5-2025.08.000-ltn3e2q
E           mod access-esm1p6/dependencies/pr181-9/access-mocsy/2025.07.002-ejsylbt
E           mod access-esm1p6/dependencies/pr181-9/access-generic-tracers/2025.09.000-x2czqdx
E           mod access-esm1p6/dependencies/pr181-9/mom5/2025.05.000-jotcb6x
E           mod access-esm1p6/dependencies/pr181-9/cable/2025.11.000-7ey6m4f
E           mod access-esm1p6/dependencies/pr181-9/gcom4/2025.08.000-qhkz6ut
E           mod access-esm1p6/dependencies/pr181-9/um7/2025.12.000-oz4papa
E           mod access-esm1p6/pr181-9
E           mpirun --mca mpi_yield_when_idle true -wdir /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere -np 256  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/atmosphere/um_hg3.exe : -wdir /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/ocean -np 240  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/ocean/mom5_access_cm : -wdir /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/ice -np 12  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_default_runtime/ice/cice_access_360x300_36x300.exe
E           /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/control/exp_default_runtime/access-esm1.6.err /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/archive/exp_default_runtime/error_logs/access-esm1.6.159728614.gadi-pbs.err
E           /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/control/exp_default_runtime/access-esm1.6.out /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/archive/exp_default_runtime/error_logs/access-esm1.6.159728614.gadi-pbs.out
E           /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/control/exp_default_runtime/env.yaml /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/archive/exp_default_runtime/error_logs/env.159728614.gadi-pbs.yaml
E           
E           ======================================================================================
E                             Resource Usage on 2026-02-02 09:03:48:
E              Job Id:             159728614.gadi-pbs
E              Project:            tm70
E              Exit Status:        1
E              Service Units:      27.16
E              NCPUs Requested:    520                 CPU Time Used: 07:12:43        
E              Memory Requested:   2.5TB                 Memory Used: 190.14GB        
E              Walltime Requested: 02:30:00            Walltime Used: 00:01:34        
E              JobFS Requested:    1.46GB                 JobFS Used: 0B              
E           ======================================================================================
E           
E           --- stderr ---
E           Loading access-esm1p6/pr181-9
E             Loading requirement: access-esm1p6/dependencies/pr181-9/oasis3-mct/5.2-dzwdenl
E               access-esm1p6/dependencies/pr181-9/cice5/2026.01.000-3ecdmyk
E               access-esm1p6/dependencies/pr181-9/access-fms/mom5-2025.08.000-ltn3e2q
E               access-esm1p6/dependencies/pr181-9/access-mocsy/2025.07.002-ejsylbt
E               access-esm1p6/dependencies/pr181-9/access-generic-tracers/2025.09.000-x2czqdx access-esm1p6/dependencies/pr181-9/mom5/2025.05.000-jotcb6x access-esm1p6/dependencies/pr181-9/cable/2025.11.000-7ey6m4f access-esm1p6/dependencies/pr181-9/gcom4/2025.08.000-qhkz6ut access-esm1p6/dependencies/pr181-9/um7/2025.12.000-oz4papa
E           Currently Loaded Modulefiles:
E            1) access-esm1p6/dependencies/pr181-9/oasis3-mct/5.2-dzwdenl                      
E            2) access-esm1p6/dependencies/pr181-9/cice5/2026.01.000-3ecdmyk                   
E            3) access-esm1p6/dependencies/pr181-9/access-fms/mom5-2025.08.000-ltn3e2q         
E            4) access-esm1p6/dependencies/pr181-9/access-mocsy/2025.07.002-ejsylbt            
E            5) access-esm1p6/dependencies/pr181-9/access-generic-tracers/2025.09.000-x2czqdx  
E            6) access-esm1p6/dependencies/pr181-9/mom5/2025.05.000-jotcb6x                    
E            7) access-esm1p6/dependencies/pr181-9/cable/2025.11.000-7ey6m4f                   
E            8) access-esm1p6/dependencies/pr181-9/gcom4/2025.08.000-qhkz6ut                   
E            9) access-esm1p6/dependencies/pr181-9/um7/2025.12.000-oz4papa                     
E           10) access-esm1p6/pr181-9                                                          
E           11) pbs                                                                            
E           12) openmpi/5.0.8                                                                  
E           payu: Model exited with error code 9; aborting.

../test-venv/lib/python3.10/site-packages/model_config_tests/exp_test_helper.py:318: RuntimeError

failed on setup with "RuntimeError: There was an error running experiment exp_1d_runtime: Payu run job failed with exit status 1:
Job_ID: 159728621.gadi-pbs
Output files: ['/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/control/exp_1d_runtime/pre-industrial.o159728621', '/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/control/exp_1d_runtime/pre-industrial.e159728621']
--- stdout ---
laboratory path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab
binary path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/bin
input path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/input
work path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work
archive path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/archive
Experiment name is configured in config.yaml:  exp_1d_runtime
nruns: 1 nruns_per_submit: 1 subrun: 1
payu: Found modules in /opt/Modules/v4.3.0
Loading input manifest: manifests/input.yaml
Loading restart manifest: manifests/restart.yaml
Loading exe manifest: manifests/exe.yaml
Setting up atmosphere
Setting up ocean
Setting up ice
Setting up coupler
Checking exe, input and restart manifests
mod access-esm1p6/dependencies/pr181-9/oasis3-mct/5.2-dzwdenl
mod access-esm1p6/dependencies/pr181-9/cice5/2026.01.000-3ecdmyk
mod access-esm1p6/dependencies/pr181-9/access-fms/mom5-2025.08.000-ltn3e2q
mod access-esm1p6/dependencies/pr181-9/access-mocsy/2025.07.002-ejsylbt
mod access-esm1p6/dependencies/pr181-9/access-generic-tracers/2025.09.000-x2czqdx
mod access-esm1p6/dependencies/pr181-9/mom5/2025.05.000-jotcb6x
mod access-esm1p6/dependencies/pr181-9/cable/2025.11.000-7ey6m4f
mod access-esm1p6/dependencies/pr181-9/gcom4/2025.08.000-qhkz6ut
mod access-esm1p6/dependencies/pr181-9/um7/2025.12.000-oz4papa
mod access-esm1p6/pr181-9
mpirun --mca mpi_yield_when_idle true -wdir /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_1d_runtime/atmosphere -np 256  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_1d_runtime/atmosphere/um_hg3.exe : -wdir /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_1d_runtime/ocean -np 240  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_1d_runtime/ocean/mom5_access_cm : -wdir /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_1d_runtime/ice -np 12  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_1d_runtime/ice/cice_access_360x300_36x300.exe
/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/control/exp_1d_runtime/access-esm1.6.err /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/archive/exp_1d_runtime/error_logs/access-esm1.6.159728621.gadi-pbs.err
/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/control/exp_1d_runtime/access-esm1.6.out /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/archive/exp_1d_runtime/error_logs/access-esm1.6.159728621.gadi-pbs.out
/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/control/exp_1d_runtime/env.yaml /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/archive/exp_1d_runtime/error_logs/env.159728621.gadi-pbs.yaml

======================================================================================
                  Resource Usage on 2026-02-02 09:03:46:
   Job Id:             159728621.gadi-pbs
   Project:            tm70
   Exit Status:        1
   Service Units:      24.84
   NCPUs Requested:    520                 CPU Time Used: 06:02:26        
   Memory Requested:   2.5TB                 Memory Used: 186.46GB        
   Walltime Requested: 02:30:00            Walltime Used: 00:01:26        
   JobFS Requested:    1.46GB                 JobFS Used: 4.39MB          
======================================================================================

--- stderr ---
Loading access-esm1p6/pr181-9
  Loading requirement: access-esm1p6/dependencies/pr181-9/oasis3-mct/5.2-dzwdenl
    access-esm1p6/dependencies/pr181-9/cice5/2026.01.000-3ecdmyk
    access-esm1p6/dependencies/pr181-9/access-fms/mom5-2025.08.000-ltn3e2q
    access-esm1p6/dependencies/pr181-9/access-mocsy/2025.07.002-ejsylbt
    access-esm1p6/dependencies/pr181-9/access-generic-tracers/2025.09.000-x2czqdx access-esm1p6/dependencies/pr181-9/mom5/2025.05.000-jotcb6x access-esm1p6/dependencies/pr181-9/cable/2025.11.000-7ey6m4f access-esm1p6/dependencies/pr181-9/gcom4/2025.08.000-qhkz6ut access-esm1p6/dependencies/pr181-9/um7/2025.12.000-oz4papa
Currently Loaded Modulefiles:
 1) access-esm1p6/dependencies/pr181-9/oasis3-mct/5.2-dzwdenl                      
 2) access-esm1p6/dependencies/pr181-9/cice5/2026.01.000-3ecdmyk                   
 3) access-esm1p6/dependencies/pr181-9/access-fms/mom5-2025.08.000-ltn3e2q         
 4) access-esm1p6/dependencies/pr181-9/access-mocsy/2025.07.002-ejsylbt            
 5) access-esm1p6/dependencies/pr181-9/access-generic-tracers/2025.09.000-x2czqdx  
 6) access-esm1p6/dependencies/pr181-9/mom5/2025.05.000-jotcb6x                    
 7) access-esm1p6/dependencies/pr181-9/cable/2025.11.000-7ey6m4f                   
 8) access-esm1p6/dependencies/pr181-9/gcom4/2025.08.000-qhkz6ut                   
 9) access-esm1p6/dependencies/pr181-9/um7/2025.12.000-oz4papa                     
10) access-esm1p6/pr181-9                                                          
11) pbs                                                                            
12) openmpi/5.0.8                                                                  
payu: Model exited with error code 1; aborting."
request = <SubRequest 'requested_experiments' for <Function test_repro_determinism>>
experiments = <model_config_tests.exp_test_helper.Experiments object at 0x7f6ca0ba8ee0>

    @pytest.fixture
    def requested_experiments(request, experiments: Experiments):
        """Fixture to check that requested experiments have run successfully
        and return a dictionary of ExpTestHelper instances for each experiment."""
        exp_marker = request.node.get_closest_marker("experiments").args[0]
        requested_exps = {}
        for exp_name in exp_marker:
            # Check experiment has run successfully - this will raise an
            # error if there are any non-zero exit codes in the outputs
>           experiments.check_experiment(exp_name)

../test-venv/lib/python3.10/site-packages/model_config_tests/config_tests/test_bit_reproducibility.py:159: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <model_config_tests.exp_test_helper.Experiments object at 0x7f6ca0ba8ee0>
exp_name = 'exp_1d_runtime'

    def check_experiment(self, exp_name: str) -> None:
        """
        Check whether given experiment name has run successfully
        """
        if exp_name in self.experiment_errors:
>           raise RuntimeError(
                f"There was an error running experiment {exp_name}:"
                f" {self.experiment_errors[exp_name]}"
            )
E           RuntimeError: There was an error running experiment exp_1d_runtime: Payu run job failed with exit status 1:
E           Job_ID: 159728621.gadi-pbs
E           Output files: ['/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/control/exp_1d_runtime/pre-industrial.o159728621', '/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/control/exp_1d_runtime/pre-industrial.e159728621']
E           --- stdout ---
E           laboratory path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab
E           binary path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/bin
E           input path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/input
E           work path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work
E           archive path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/archive
E           Experiment name is configured in config.yaml:  exp_1d_runtime
E           nruns: 1 nruns_per_submit: 1 subrun: 1
E           payu: Found modules in /opt/Modules/v4.3.0
E           Loading input manifest: manifests/input.yaml
E           Loading restart manifest: manifests/restart.yaml
E           Loading exe manifest: manifests/exe.yaml
E           Setting up atmosphere
E           Setting up ocean
E           Setting up ice
E           Setting up coupler
E           Checking exe, input and restart manifests
E           mod access-esm1p6/dependencies/pr181-9/oasis3-mct/5.2-dzwdenl
E           mod access-esm1p6/dependencies/pr181-9/cice5/2026.01.000-3ecdmyk
E           mod access-esm1p6/dependencies/pr181-9/access-fms/mom5-2025.08.000-ltn3e2q
E           mod access-esm1p6/dependencies/pr181-9/access-mocsy/2025.07.002-ejsylbt
E           mod access-esm1p6/dependencies/pr181-9/access-generic-tracers/2025.09.000-x2czqdx
E           mod access-esm1p6/dependencies/pr181-9/mom5/2025.05.000-jotcb6x
E           mod access-esm1p6/dependencies/pr181-9/cable/2025.11.000-7ey6m4f
E           mod access-esm1p6/dependencies/pr181-9/gcom4/2025.08.000-qhkz6ut
E           mod access-esm1p6/dependencies/pr181-9/um7/2025.12.000-oz4papa
E           mod access-esm1p6/pr181-9
E           mpirun --mca mpi_yield_when_idle true -wdir /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_1d_runtime/atmosphere -np 256  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_1d_runtime/atmosphere/um_hg3.exe : -wdir /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_1d_runtime/ocean -np 240  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_1d_runtime/ocean/mom5_access_cm : -wdir /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_1d_runtime/ice -np 12  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/work/exp_1d_runtime/ice/cice_access_360x300_36x300.exe
E           /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/control/exp_1d_runtime/access-esm1.6.err /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/archive/exp_1d_runtime/error_logs/access-esm1.6.159728621.gadi-pbs.err
E           /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/control/exp_1d_runtime/access-esm1.6.out /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/archive/exp_1d_runtime/error_logs/access-esm1.6.159728621.gadi-pbs.out
E           /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/control/exp_1d_runtime/env.yaml /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/13259829add9ab6f328eafa807859733124340ae/lab/archive/exp_1d_runtime/error_logs/env.159728621.gadi-pbs.yaml
E           
E           ======================================================================================
E                             Resource Usage on 2026-02-02 09:03:46:
E              Job Id:             159728621.gadi-pbs
E              Project:            tm70
E              Exit Status:        1
E              Service Units:      24.84
E              NCPUs Requested:    520                 CPU Time Used: 06:02:26        
E              Memory Requested:   2.5TB                 Memory Used: 186.46GB        
E              Walltime Requested: 02:30:00            Walltime Used: 00:01:26        
E              JobFS Requested:    1.46GB                 JobFS Used: 4.39MB          
E           ======================================================================================
E           
E           --- stderr ---
E           Loading access-esm1p6/pr181-9
E             Loading requirement: access-esm1p6/dependencies/pr181-9/oasis3-mct/5.2-dzwdenl
E               access-esm1p6/dependencies/pr181-9/cice5/2026.01.000-3ecdmyk
E               access-esm1p6/dependencies/pr181-9/access-fms/mom5-2025.08.000-ltn3e2q
E               access-esm1p6/dependencies/pr181-9/access-mocsy/2025.07.002-ejsylbt
E               access-esm1p6/dependencies/pr181-9/access-generic-tracers/2025.09.000-x2czqdx access-esm1p6/dependencies/pr181-9/mom5/2025.05.000-jotcb6x access-esm1p6/dependencies/pr181-9/cable/2025.11.000-7ey6m4f access-esm1p6/dependencies/pr181-9/gcom4/2025.08.000-qhkz6ut access-esm1p6/dependencies/pr181-9/um7/2025.12.000-oz4papa
E           Currently Loaded Modulefiles:
E            1) access-esm1p6/dependencies/pr181-9/oasis3-mct/5.2-dzwdenl                      
E            2) access-esm1p6/dependencies/pr181-9/cice5/2026.01.000-3ecdmyk                   
E            3) access-esm1p6/dependencies/pr181-9/access-fms/mom5-2025.08.000-ltn3e2q         
E            4) access-esm1p6/dependencies/pr181-9/access-mocsy/2025.07.002-ejsylbt            
E            5) access-esm1p6/dependencies/pr181-9/access-generic-tracers/2025.09.000-x2czqdx  
E            6) access-esm1p6/dependencies/pr181-9/mom5/2025.05.000-jotcb6x                    
E            7) access-esm1p6/dependencies/pr181-9/cable/2025.11.000-7ey6m4f                   
E            8) access-esm1p6/dependencies/pr181-9/gcom4/2025.08.000-qhkz6ut                   
E            9) access-esm1p6/dependencies/pr181-9/um7/2025.12.000-oz4papa                     
E           10) access-esm1p6/pr181-9                                                          
E           11) pbs                                                                            
E           12) openmpi/5.0.8                                                                  
E           payu: Model exited with error code 1; aborting.

../test-venv/lib/python3.10/site-packages/model_config_tests/exp_test_helper.py:318: RuntimeError