failed on setup with "RuntimeError: Failed to run payu setup:
Return code: 1
--- stdout ---
laboratory path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/7baa80efc461f20f598c2255a8c928e29e781b43/lab
binary path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/7baa80efc461f20f598c2255a8c928e29e781b43/lab/bin
input path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/7baa80efc461f20f598c2255a8c928e29e781b43/lab/input
work path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/7baa80efc461f20f598c2255a8c928e29e781b43/lab/work
archive path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/7baa80efc461f20f598c2255a8c928e29e781b43/lab/archive
Experiment name is configured in config.yaml:  exp_default_runtime
payu: Found modules in /opt/Modules/v4.3.0
Loading input manifest: manifests/input.yaml
Loading restart manifest: manifests/restart.yaml
Loading exe manifest: manifests/exe.yaml
Setting up atmosphere

--- stderr ---
Loading access-esm1p6/dependencies/pr168-4/access-generic-tracers/2025.09.000-daonig7
  ERROR: Unable to locate a modulefile for
    'access-esm1p6/dependencies/pr168-4/access-mocsy/2025.07.002-pk66vb6'
  ERROR: Requirement
    access-esm1p6/dependencies/pr168-4/access-mocsy/2025.07.002-pk66vb6 is not
    loaded

Loading access-esm1p6/dependencies/pr168-4/mom5/2025.05.000-bpjxcnl
  ERROR: Requirement
    access-esm1p6/dependencies/pr168-4/access-generic-tracers/2025.09.000-daonig7 is not loaded

Loading access-esm1p6/pr168-4
  ERROR: Load of requirement
    access-esm1p6/dependencies/pr168-4/access-mocsy/2025.07.002-pk66vb6 failed
  ERROR: Load of requirement
    access-esm1p6/dependencies/pr168-4/access-generic-tracers/2025.09.000-daonig7 failed
  ERROR: Load of requirement
    access-esm1p6/dependencies/pr168-4/mom5/2025.05.000-bpjxcnl failed
Traceback (most recent call last):
  File "/opt/conda/payu-dev-20260213T043728Z-0359e0d/bin/payu", line 6, in <module>
    sys.exit(parse())
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260213T043728Z-0359e0d/lib/python3.10/site-packages/payu/cli.py", line 52, in parse
    run_cmd(**args)
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260213T043728Z-0359e0d/lib/python3.10/site-packages/payu/subcommands/setup_cmd.py", line 28, in runcmd
    expt.setup(force_archive=force_archive)
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260213T043728Z-0359e0d/lib/python3.10/site-packages/payu/experiment.py", line 63, in wrapper
    result = func(self, *args, **kwargs)
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260213T043728Z-0359e0d/lib/python3.10/site-packages/payu/experiment.py", line 523, in setup
    model.setup()
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260213T043728Z-0359e0d/lib/python3.10/site-packages/payu/models/um.py", line 129, in setup
    super(UnifiedModel, self).setup()
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260213T043728Z-0359e0d/lib/python3.10/site-packages/payu/models/model.py", line 315, in setup
    raise FileNotFoundError(
FileNotFoundError: Executable for atmosphere model not found on path: /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/7baa80efc461f20f598c2255a8c928e29e781b43/lab/bin/um_hg3.exe"
request = <SubRequest 'experiments' for <Function test_repro_historical>>
output_path = PosixPath('/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/7baa80efc461f20f598c2255a8c928e29e781b43')
control_path = PosixPath('/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/7baa80efc461f20f598c2255a8c928e29e781b43/base-experiment')
keep_archive = False

    @pytest.fixture(scope="class")
    def experiments(
        request, output_path: Path, control_path: Path, keep_archive: Optional[bool]
    ):
        """
        Parse the experiments markers from the requested tests and
        submit all necessary experiments at the same time.
    
        The scope is class so the experiments are only run once before all repro
        tests.
        """
    
        # Parse the experiments markers from the requested tests
        experiments_markers = []
        for item in request.session.items:
            if item.parent == request.node:
                marker = item.get_closest_marker("experiments")
                if marker:
                    experiments_markers.append(marker.args[0])
    
>       return _experiments(experiments_markers, output_path, control_path, keep_archive)

../test-venv/lib/python3.10/site-packages/model_config_tests/config_tests/test_bit_reproducibility.py:147: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../test-venv/lib/python3.10/site-packages/model_config_tests/config_tests/test_bit_reproducibility.py:113: in _experiments
    experiments.setup_and_submit(
../test-venv/lib/python3.10/site-packages/model_config_tests/exp_test_helper.py:279: in setup_and_submit
    exp.submit_payu_run(n_runs=n_runs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <model_config_tests.exp_test_helper.ExpTestHelper object at 0x7f3bd35d0d60>
n_runs = 1

    def submit_payu_run(self, n_runs: int = None) -> str:
        """
        Submit a payu run job.
    
        Parameters
        ----------
        n_runs: int
            The number of runs to submit with --nruns.
    
        Returns
        ----------
        str
            The job ID of the submitted payu run job
        """
        if self.disable_payu_run:
            return
    
        owd = Path.cwd()
        try:
            # Change to experiment directory and run.
            os.chdir(self.control_path)
    
            print("Running payu setup")
            result = sp.run(
                ["payu", "setup", "--lab", str(self.lab_path)],
                capture_output=True,
                text=True,
            )
            if result.returncode != 0:
                # Add additional error messaging for debugging
                error_msg = (
                    "Failed to run payu setup:\n"
                    f"Return code: {result.returncode}\n"
                    f"--- stdout ---\n{result.stdout}\n"
                    f"--- stderr ---\n{result.stderr}"
                )
                print(error_msg)
>               raise RuntimeError(error_msg)
E               RuntimeError: Failed to run payu setup:
E               Return code: 1
E               --- stdout ---
E               laboratory path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/7baa80efc461f20f598c2255a8c928e29e781b43/lab
E               binary path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/7baa80efc461f20f598c2255a8c928e29e781b43/lab/bin
E               input path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/7baa80efc461f20f598c2255a8c928e29e781b43/lab/input
E               work path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/7baa80efc461f20f598c2255a8c928e29e781b43/lab/work
E               archive path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/7baa80efc461f20f598c2255a8c928e29e781b43/lab/archive
E               Experiment name is configured in config.yaml:  exp_default_runtime
E               payu: Found modules in /opt/Modules/v4.3.0
E               Loading input manifest: manifests/input.yaml
E               Loading restart manifest: manifests/restart.yaml
E               Loading exe manifest: manifests/exe.yaml
E               Setting up atmosphere
E               
E               --- stderr ---
E               Loading access-esm1p6/dependencies/pr168-4/access-generic-tracers/2025.09.000-daonig7
E                 ERROR: Unable to locate a modulefile for
E                   'access-esm1p6/dependencies/pr168-4/access-mocsy/2025.07.002-pk66vb6'
E                 ERROR: Requirement
E                   access-esm1p6/dependencies/pr168-4/access-mocsy/2025.07.002-pk66vb6 is not
E                   loaded
E               
E               Loading access-esm1p6/dependencies/pr168-4/mom5/2025.05.000-bpjxcnl
E                 ERROR: Requirement
E                   access-esm1p6/dependencies/pr168-4/access-generic-tracers/2025.09.000-daonig7 is not loaded
E               
E               Loading access-esm1p6/pr168-4
E                 ERROR: Load of requirement
E                   access-esm1p6/dependencies/pr168-4/access-mocsy/2025.07.002-pk66vb6 failed
E                 ERROR: Load of requirement
E                   access-esm1p6/dependencies/pr168-4/access-generic-tracers/2025.09.000-daonig7 failed
E                 ERROR: Load of requirement
E                   access-esm1p6/dependencies/pr168-4/mom5/2025.05.000-bpjxcnl failed
E               Traceback (most recent call last):
E                 File "/opt/conda/payu-dev-20260213T043728Z-0359e0d/bin/payu", line 6, in <module>
E                   sys.exit(parse())
E                 File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260213T043728Z-0359e0d/lib/python3.10/site-packages/payu/cli.py", line 52, in parse
E                   run_cmd(**args)
E                 File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260213T043728Z-0359e0d/lib/python3.10/site-packages/payu/subcommands/setup_cmd.py", line 28, in runcmd
E                   expt.setup(force_archive=force_archive)
E                 File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260213T043728Z-0359e0d/lib/python3.10/site-packages/payu/experiment.py", line 63, in wrapper
E                   result = func(self, *args, **kwargs)
E                 File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260213T043728Z-0359e0d/lib/python3.10/site-packages/payu/experiment.py", line 523, in setup
E                   model.setup()
E                 File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260213T043728Z-0359e0d/lib/python3.10/site-packages/payu/models/um.py", line 129, in setup
E                   super(UnifiedModel, self).setup()
E                 File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260213T043728Z-0359e0d/lib/python3.10/site-packages/payu/models/model.py", line 315, in setup
E                   raise FileNotFoundError(
E               FileNotFoundError: Executable for atmosphere model not found on path: /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/7baa80efc461f20f598c2255a8c928e29e781b43/lab/bin/um_hg3.exe

../test-venv/lib/python3.10/site-packages/model_config_tests/exp_test_helper.py:154: RuntimeError

failed on setup with "RuntimeError: Failed to run payu setup:
Return code: 1
--- stdout ---
laboratory path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/7baa80efc461f20f598c2255a8c928e29e781b43/lab
binary path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/7baa80efc461f20f598c2255a8c928e29e781b43/lab/bin
input path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/7baa80efc461f20f598c2255a8c928e29e781b43/lab/input
work path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/7baa80efc461f20f598c2255a8c928e29e781b43/lab/work
archive path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/7baa80efc461f20f598c2255a8c928e29e781b43/lab/archive
Experiment name is configured in config.yaml:  exp_default_runtime
payu: Found modules in /opt/Modules/v4.3.0
Loading input manifest: manifests/input.yaml
Loading restart manifest: manifests/restart.yaml
Loading exe manifest: manifests/exe.yaml
Setting up atmosphere

--- stderr ---
Loading access-esm1p6/dependencies/pr168-4/access-generic-tracers/2025.09.000-daonig7
  ERROR: Unable to locate a modulefile for
    'access-esm1p6/dependencies/pr168-4/access-mocsy/2025.07.002-pk66vb6'
  ERROR: Requirement
    access-esm1p6/dependencies/pr168-4/access-mocsy/2025.07.002-pk66vb6 is not
    loaded

Loading access-esm1p6/dependencies/pr168-4/mom5/2025.05.000-bpjxcnl
  ERROR: Requirement
    access-esm1p6/dependencies/pr168-4/access-generic-tracers/2025.09.000-daonig7 is not loaded

Loading access-esm1p6/pr168-4
  ERROR: Load of requirement
    access-esm1p6/dependencies/pr168-4/access-mocsy/2025.07.002-pk66vb6 failed
  ERROR: Load of requirement
    access-esm1p6/dependencies/pr168-4/access-generic-tracers/2025.09.000-daonig7 failed
  ERROR: Load of requirement
    access-esm1p6/dependencies/pr168-4/mom5/2025.05.000-bpjxcnl failed
Traceback (most recent call last):
  File "/opt/conda/payu-dev-20260213T043728Z-0359e0d/bin/payu", line 6, in <module>
    sys.exit(parse())
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260213T043728Z-0359e0d/lib/python3.10/site-packages/payu/cli.py", line 52, in parse
    run_cmd(**args)
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260213T043728Z-0359e0d/lib/python3.10/site-packages/payu/subcommands/setup_cmd.py", line 28, in runcmd
    expt.setup(force_archive=force_archive)
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260213T043728Z-0359e0d/lib/python3.10/site-packages/payu/experiment.py", line 63, in wrapper
    result = func(self, *args, **kwargs)
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260213T043728Z-0359e0d/lib/python3.10/site-packages/payu/experiment.py", line 523, in setup
    model.setup()
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260213T043728Z-0359e0d/lib/python3.10/site-packages/payu/models/um.py", line 129, in setup
    super(UnifiedModel, self).setup()
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260213T043728Z-0359e0d/lib/python3.10/site-packages/payu/models/model.py", line 315, in setup
    raise FileNotFoundError(
FileNotFoundError: Executable for atmosphere model not found on path: /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/7baa80efc461f20f598c2255a8c928e29e781b43/lab/bin/um_hg3.exe"
request = <SubRequest 'experiments' for <Function test_repro_historical>>
output_path = PosixPath('/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/7baa80efc461f20f598c2255a8c928e29e781b43')
control_path = PosixPath('/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/7baa80efc461f20f598c2255a8c928e29e781b43/base-experiment')
keep_archive = False

    @pytest.fixture(scope="class")
    def experiments(
        request, output_path: Path, control_path: Path, keep_archive: Optional[bool]
    ):
        """
        Parse the experiments markers from the requested tests and
        submit all necessary experiments at the same time.
    
        The scope is class so the experiments are only run once before all repro
        tests.
        """
    
        # Parse the experiments markers from the requested tests
        experiments_markers = []
        for item in request.session.items:
            if item.parent == request.node:
                marker = item.get_closest_marker("experiments")
                if marker:
                    experiments_markers.append(marker.args[0])
    
>       return _experiments(experiments_markers, output_path, control_path, keep_archive)

../test-venv/lib/python3.10/site-packages/model_config_tests/config_tests/test_bit_reproducibility.py:147: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../test-venv/lib/python3.10/site-packages/model_config_tests/config_tests/test_bit_reproducibility.py:113: in _experiments
    experiments.setup_and_submit(
../test-venv/lib/python3.10/site-packages/model_config_tests/exp_test_helper.py:279: in setup_and_submit
    exp.submit_payu_run(n_runs=n_runs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <model_config_tests.exp_test_helper.ExpTestHelper object at 0x7f3bd35d0d60>
n_runs = 1

    def submit_payu_run(self, n_runs: int = None) -> str:
        """
        Submit a payu run job.
    
        Parameters
        ----------
        n_runs: int
            The number of runs to submit with --nruns.
    
        Returns
        ----------
        str
            The job ID of the submitted payu run job
        """
        if self.disable_payu_run:
            return
    
        owd = Path.cwd()
        try:
            # Change to experiment directory and run.
            os.chdir(self.control_path)
    
            print("Running payu setup")
            result = sp.run(
                ["payu", "setup", "--lab", str(self.lab_path)],
                capture_output=True,
                text=True,
            )
            if result.returncode != 0:
                # Add additional error messaging for debugging
                error_msg = (
                    "Failed to run payu setup:\n"
                    f"Return code: {result.returncode}\n"
                    f"--- stdout ---\n{result.stdout}\n"
                    f"--- stderr ---\n{result.stderr}"
                )
                print(error_msg)
>               raise RuntimeError(error_msg)
E               RuntimeError: Failed to run payu setup:
E               Return code: 1
E               --- stdout ---
E               laboratory path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/7baa80efc461f20f598c2255a8c928e29e781b43/lab
E               binary path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/7baa80efc461f20f598c2255a8c928e29e781b43/lab/bin
E               input path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/7baa80efc461f20f598c2255a8c928e29e781b43/lab/input
E               work path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/7baa80efc461f20f598c2255a8c928e29e781b43/lab/work
E               archive path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/7baa80efc461f20f598c2255a8c928e29e781b43/lab/archive
E               Experiment name is configured in config.yaml:  exp_default_runtime
E               payu: Found modules in /opt/Modules/v4.3.0
E               Loading input manifest: manifests/input.yaml
E               Loading restart manifest: manifests/restart.yaml
E               Loading exe manifest: manifests/exe.yaml
E               Setting up atmosphere
E               
E               --- stderr ---
E               Loading access-esm1p6/dependencies/pr168-4/access-generic-tracers/2025.09.000-daonig7
E                 ERROR: Unable to locate a modulefile for
E                   'access-esm1p6/dependencies/pr168-4/access-mocsy/2025.07.002-pk66vb6'
E                 ERROR: Requirement
E                   access-esm1p6/dependencies/pr168-4/access-mocsy/2025.07.002-pk66vb6 is not
E                   loaded
E               
E               Loading access-esm1p6/dependencies/pr168-4/mom5/2025.05.000-bpjxcnl
E                 ERROR: Requirement
E                   access-esm1p6/dependencies/pr168-4/access-generic-tracers/2025.09.000-daonig7 is not loaded
E               
E               Loading access-esm1p6/pr168-4
E                 ERROR: Load of requirement
E                   access-esm1p6/dependencies/pr168-4/access-mocsy/2025.07.002-pk66vb6 failed
E                 ERROR: Load of requirement
E                   access-esm1p6/dependencies/pr168-4/access-generic-tracers/2025.09.000-daonig7 failed
E                 ERROR: Load of requirement
E                   access-esm1p6/dependencies/pr168-4/mom5/2025.05.000-bpjxcnl failed
E               Traceback (most recent call last):
E                 File "/opt/conda/payu-dev-20260213T043728Z-0359e0d/bin/payu", line 6, in <module>
E                   sys.exit(parse())
E                 File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260213T043728Z-0359e0d/lib/python3.10/site-packages/payu/cli.py", line 52, in parse
E                   run_cmd(**args)
E                 File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260213T043728Z-0359e0d/lib/python3.10/site-packages/payu/subcommands/setup_cmd.py", line 28, in runcmd
E                   expt.setup(force_archive=force_archive)
E                 File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260213T043728Z-0359e0d/lib/python3.10/site-packages/payu/experiment.py", line 63, in wrapper
E                   result = func(self, *args, **kwargs)
E                 File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260213T043728Z-0359e0d/lib/python3.10/site-packages/payu/experiment.py", line 523, in setup
E                   model.setup()
E                 File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260213T043728Z-0359e0d/lib/python3.10/site-packages/payu/models/um.py", line 129, in setup
E                   super(UnifiedModel, self).setup()
E                 File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260213T043728Z-0359e0d/lib/python3.10/site-packages/payu/models/model.py", line 315, in setup
E                   raise FileNotFoundError(
E               FileNotFoundError: Executable for atmosphere model not found on path: /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/7baa80efc461f20f598c2255a8c928e29e781b43/lab/bin/um_hg3.exe

../test-venv/lib/python3.10/site-packages/model_config_tests/exp_test_helper.py:154: RuntimeError