failed on setup with "RuntimeError: Failed to run payu setup:
Return code: 1
--- stdout ---
laboratory path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/61dfcfe71d7a20cbdaecbe3e856e64bfeb148cf3/lab
binary path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/61dfcfe71d7a20cbdaecbe3e856e64bfeb148cf3/lab/bin
input path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/61dfcfe71d7a20cbdaecbe3e856e64bfeb148cf3/lab/input
work path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/61dfcfe71d7a20cbdaecbe3e856e64bfeb148cf3/lab/work
archive path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/61dfcfe71d7a20cbdaecbe3e856e64bfeb148cf3/lab/archive
Experiment name is configured in config.yaml:  exp_default_runtime
payu: Found modules in /opt/Modules/v4.3.0
Loading input manifest: manifests/input.yaml
Loading restart manifest: manifests/restart.yaml
Loading exe manifest: manifests/exe.yaml
Setting up atmosphere
Setting up ocean
Setting up ice

--- stderr ---
Loading access-esm1p6/pr181-9
  Loading requirement: access-esm1p6/dependencies/pr181-9/oasis3-mct/5.2-dzwdenl
    access-esm1p6/dependencies/pr181-9/cice5/2026.01.000-3ecdmyk
    access-esm1p6/dependencies/pr181-9/access-fms/mom5-2025.08.000-ltn3e2q
    access-esm1p6/dependencies/pr181-9/access-mocsy/2025.07.002-ejsylbt
    access-esm1p6/dependencies/pr181-9/access-generic-tracers/2025.09.000-x2czqdx access-esm1p6/dependencies/pr181-9/mom5/2025.05.000-jotcb6x access-esm1p6/dependencies/pr181-9/cable/2025.11.000-7ey6m4f access-esm1p6/dependencies/pr181-9/gcom4/2025.08.000-qhkz6ut access-esm1p6/dependencies/pr181-9/um7/2025.12.000-oz4papa
Traceback (most recent call last):
  File "/opt/conda/payu-dev-20260130T053924Z-1346ed4/bin/payu", line 7, in <module>
    sys.exit(parse())
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260130T053924Z-1346ed4/lib/python3.10/site-packages/payu/cli.py", line 52, in parse
    run_cmd(**args)
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260130T053924Z-1346ed4/lib/python3.10/site-packages/payu/subcommands/setup_cmd.py", line 28, in runcmd
    expt.setup(force_archive=force_archive)
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260130T053924Z-1346ed4/lib/python3.10/site-packages/payu/experiment.py", line 63, in wrapper
    result = func(self, *args, **kwargs)
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260130T053924Z-1346ed4/lib/python3.10/site-packages/payu/experiment.py", line 523, in setup
    model.setup()
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260130T053924Z-1346ed4/lib/python3.10/site-packages/payu/models/cice5.py", line 71, in setup
    super(Cice5, self).setup()
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260130T053924Z-1346ed4/lib/python3.10/site-packages/payu/models/cice.py", line 164, in setup
    super(Cice, self).setup()
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260130T053924Z-1346ed4/lib/python3.10/site-packages/payu/models/model.py", line 315, in setup
    raise FileNotFoundError(
FileNotFoundError: Executable for ice model not found on path: /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/61dfcfe71d7a20cbdaecbe3e856e64bfeb148cf3/lab/bin/cice_access-esm1.6_360x300_12x1_12p.exe"
request = <SubRequest 'experiments' for <Function test_repro_historical>>
output_path = PosixPath('/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/61dfcfe71d7a20cbdaecbe3e856e64bfeb148cf3')
control_path = PosixPath('/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/61dfcfe71d7a20cbdaecbe3e856e64bfeb148cf3/base-experiment')
keep_archive = False

    @pytest.fixture(scope="class")
    def experiments(
        request, output_path: Path, control_path: Path, keep_archive: Optional[bool]
    ):
        """
        Parse the experiments markers from the requested tests and
        submit all necessary experiments at the same time.
    
        The scope is class so the experiments are only run once before all repro
        tests.
        """
    
        # Parse the experiments markers from the requested tests
        experiments_markers = []
        for item in request.session.items:
            if item.parent == request.node:
                marker = item.get_closest_marker("experiments")
                if marker:
                    experiments_markers.append(marker.args[0])
    
>       return _experiments(experiments_markers, output_path, control_path, keep_archive)

../test-venv/lib/python3.10/site-packages/model_config_tests/config_tests/test_bit_reproducibility.py:147: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../test-venv/lib/python3.10/site-packages/model_config_tests/config_tests/test_bit_reproducibility.py:113: in _experiments
    experiments.setup_and_submit(
../test-venv/lib/python3.10/site-packages/model_config_tests/exp_test_helper.py:279: in setup_and_submit
    exp.submit_payu_run(n_runs=n_runs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <model_config_tests.exp_test_helper.ExpTestHelper object at 0x7f85347d4f10>
n_runs = 1

    def submit_payu_run(self, n_runs: int = None) -> str:
        """
        Submit a payu run job.
    
        Parameters
        ----------
        n_runs: int
            The number of runs to submit with --nruns.
    
        Returns
        ----------
        str
            The job ID of the submitted payu run job
        """
        if self.disable_payu_run:
            return
    
        owd = Path.cwd()
        try:
            # Change to experiment directory and run.
            os.chdir(self.control_path)
    
            print("Running payu setup")
            result = sp.run(
                ["payu", "setup", "--lab", str(self.lab_path)],
                capture_output=True,
                text=True,
            )
            if result.returncode != 0:
                # Add additional error messaging for debugging
                error_msg = (
                    "Failed to run payu setup:\n"
                    f"Return code: {result.returncode}\n"
                    f"--- stdout ---\n{result.stdout}\n"
                    f"--- stderr ---\n{result.stderr}"
                )
                print(error_msg)
>               raise RuntimeError(error_msg)
E               RuntimeError: Failed to run payu setup:
E               Return code: 1
E               --- stdout ---
E               laboratory path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/61dfcfe71d7a20cbdaecbe3e856e64bfeb148cf3/lab
E               binary path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/61dfcfe71d7a20cbdaecbe3e856e64bfeb148cf3/lab/bin
E               input path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/61dfcfe71d7a20cbdaecbe3e856e64bfeb148cf3/lab/input
E               work path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/61dfcfe71d7a20cbdaecbe3e856e64bfeb148cf3/lab/work
E               archive path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/61dfcfe71d7a20cbdaecbe3e856e64bfeb148cf3/lab/archive
E               Experiment name is configured in config.yaml:  exp_default_runtime
E               payu: Found modules in /opt/Modules/v4.3.0
E               Loading input manifest: manifests/input.yaml
E               Loading restart manifest: manifests/restart.yaml
E               Loading exe manifest: manifests/exe.yaml
E               Setting up atmosphere
E               Setting up ocean
E               Setting up ice
E               
E               --- stderr ---
E               Loading access-esm1p6/pr181-9
E                 Loading requirement: access-esm1p6/dependencies/pr181-9/oasis3-mct/5.2-dzwdenl
E                   access-esm1p6/dependencies/pr181-9/cice5/2026.01.000-3ecdmyk
E                   access-esm1p6/dependencies/pr181-9/access-fms/mom5-2025.08.000-ltn3e2q
E                   access-esm1p6/dependencies/pr181-9/access-mocsy/2025.07.002-ejsylbt
E                   access-esm1p6/dependencies/pr181-9/access-generic-tracers/2025.09.000-x2czqdx access-esm1p6/dependencies/pr181-9/mom5/2025.05.000-jotcb6x access-esm1p6/dependencies/pr181-9/cable/2025.11.000-7ey6m4f access-esm1p6/dependencies/pr181-9/gcom4/2025.08.000-qhkz6ut access-esm1p6/dependencies/pr181-9/um7/2025.12.000-oz4papa
E               Traceback (most recent call last):
E                 File "/opt/conda/payu-dev-20260130T053924Z-1346ed4/bin/payu", line 7, in <module>
E                   sys.exit(parse())
E                 File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260130T053924Z-1346ed4/lib/python3.10/site-packages/payu/cli.py", line 52, in parse
E                   run_cmd(**args)
E                 File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260130T053924Z-1346ed4/lib/python3.10/site-packages/payu/subcommands/setup_cmd.py", line 28, in runcmd
E                   expt.setup(force_archive=force_archive)
E                 File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260130T053924Z-1346ed4/lib/python3.10/site-packages/payu/experiment.py", line 63, in wrapper
E                   result = func(self, *args, **kwargs)
E                 File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260130T053924Z-1346ed4/lib/python3.10/site-packages/payu/experiment.py", line 523, in setup
E                   model.setup()
E                 File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260130T053924Z-1346ed4/lib/python3.10/site-packages/payu/models/cice5.py", line 71, in setup
E                   super(Cice5, self).setup()
E                 File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260130T053924Z-1346ed4/lib/python3.10/site-packages/payu/models/cice.py", line 164, in setup
E                   super(Cice, self).setup()
E                 File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260130T053924Z-1346ed4/lib/python3.10/site-packages/payu/models/model.py", line 315, in setup
E                   raise FileNotFoundError(
E               FileNotFoundError: Executable for ice model not found on path: /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/61dfcfe71d7a20cbdaecbe3e856e64bfeb148cf3/lab/bin/cice_access-esm1.6_360x300_12x1_12p.exe

../test-venv/lib/python3.10/site-packages/model_config_tests/exp_test_helper.py:154: RuntimeError

failed on setup with "RuntimeError: Failed to run payu setup:
Return code: 1
--- stdout ---
laboratory path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/61dfcfe71d7a20cbdaecbe3e856e64bfeb148cf3/lab
binary path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/61dfcfe71d7a20cbdaecbe3e856e64bfeb148cf3/lab/bin
input path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/61dfcfe71d7a20cbdaecbe3e856e64bfeb148cf3/lab/input
work path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/61dfcfe71d7a20cbdaecbe3e856e64bfeb148cf3/lab/work
archive path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/61dfcfe71d7a20cbdaecbe3e856e64bfeb148cf3/lab/archive
Experiment name is configured in config.yaml:  exp_default_runtime
payu: Found modules in /opt/Modules/v4.3.0
Loading input manifest: manifests/input.yaml
Loading restart manifest: manifests/restart.yaml
Loading exe manifest: manifests/exe.yaml
Setting up atmosphere
Setting up ocean
Setting up ice

--- stderr ---
Loading access-esm1p6/pr181-9
  Loading requirement: access-esm1p6/dependencies/pr181-9/oasis3-mct/5.2-dzwdenl
    access-esm1p6/dependencies/pr181-9/cice5/2026.01.000-3ecdmyk
    access-esm1p6/dependencies/pr181-9/access-fms/mom5-2025.08.000-ltn3e2q
    access-esm1p6/dependencies/pr181-9/access-mocsy/2025.07.002-ejsylbt
    access-esm1p6/dependencies/pr181-9/access-generic-tracers/2025.09.000-x2czqdx access-esm1p6/dependencies/pr181-9/mom5/2025.05.000-jotcb6x access-esm1p6/dependencies/pr181-9/cable/2025.11.000-7ey6m4f access-esm1p6/dependencies/pr181-9/gcom4/2025.08.000-qhkz6ut access-esm1p6/dependencies/pr181-9/um7/2025.12.000-oz4papa
Traceback (most recent call last):
  File "/opt/conda/payu-dev-20260130T053924Z-1346ed4/bin/payu", line 7, in <module>
    sys.exit(parse())
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260130T053924Z-1346ed4/lib/python3.10/site-packages/payu/cli.py", line 52, in parse
    run_cmd(**args)
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260130T053924Z-1346ed4/lib/python3.10/site-packages/payu/subcommands/setup_cmd.py", line 28, in runcmd
    expt.setup(force_archive=force_archive)
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260130T053924Z-1346ed4/lib/python3.10/site-packages/payu/experiment.py", line 63, in wrapper
    result = func(self, *args, **kwargs)
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260130T053924Z-1346ed4/lib/python3.10/site-packages/payu/experiment.py", line 523, in setup
    model.setup()
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260130T053924Z-1346ed4/lib/python3.10/site-packages/payu/models/cice5.py", line 71, in setup
    super(Cice5, self).setup()
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260130T053924Z-1346ed4/lib/python3.10/site-packages/payu/models/cice.py", line 164, in setup
    super(Cice, self).setup()
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260130T053924Z-1346ed4/lib/python3.10/site-packages/payu/models/model.py", line 315, in setup
    raise FileNotFoundError(
FileNotFoundError: Executable for ice model not found on path: /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/61dfcfe71d7a20cbdaecbe3e856e64bfeb148cf3/lab/bin/cice_access-esm1.6_360x300_12x1_12p.exe"
request = <SubRequest 'experiments' for <Function test_repro_historical>>
output_path = PosixPath('/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/61dfcfe71d7a20cbdaecbe3e856e64bfeb148cf3')
control_path = PosixPath('/scratch/tm70/repro-ci/experiments/access-esm1.6-configs/61dfcfe71d7a20cbdaecbe3e856e64bfeb148cf3/base-experiment')
keep_archive = False

    @pytest.fixture(scope="class")
    def experiments(
        request, output_path: Path, control_path: Path, keep_archive: Optional[bool]
    ):
        """
        Parse the experiments markers from the requested tests and
        submit all necessary experiments at the same time.
    
        The scope is class so the experiments are only run once before all repro
        tests.
        """
    
        # Parse the experiments markers from the requested tests
        experiments_markers = []
        for item in request.session.items:
            if item.parent == request.node:
                marker = item.get_closest_marker("experiments")
                if marker:
                    experiments_markers.append(marker.args[0])
    
>       return _experiments(experiments_markers, output_path, control_path, keep_archive)

../test-venv/lib/python3.10/site-packages/model_config_tests/config_tests/test_bit_reproducibility.py:147: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../test-venv/lib/python3.10/site-packages/model_config_tests/config_tests/test_bit_reproducibility.py:113: in _experiments
    experiments.setup_and_submit(
../test-venv/lib/python3.10/site-packages/model_config_tests/exp_test_helper.py:279: in setup_and_submit
    exp.submit_payu_run(n_runs=n_runs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <model_config_tests.exp_test_helper.ExpTestHelper object at 0x7f85347d4f10>
n_runs = 1

    def submit_payu_run(self, n_runs: int = None) -> str:
        """
        Submit a payu run job.
    
        Parameters
        ----------
        n_runs: int
            The number of runs to submit with --nruns.
    
        Returns
        ----------
        str
            The job ID of the submitted payu run job
        """
        if self.disable_payu_run:
            return
    
        owd = Path.cwd()
        try:
            # Change to experiment directory and run.
            os.chdir(self.control_path)
    
            print("Running payu setup")
            result = sp.run(
                ["payu", "setup", "--lab", str(self.lab_path)],
                capture_output=True,
                text=True,
            )
            if result.returncode != 0:
                # Add additional error messaging for debugging
                error_msg = (
                    "Failed to run payu setup:\n"
                    f"Return code: {result.returncode}\n"
                    f"--- stdout ---\n{result.stdout}\n"
                    f"--- stderr ---\n{result.stderr}"
                )
                print(error_msg)
>               raise RuntimeError(error_msg)
E               RuntimeError: Failed to run payu setup:
E               Return code: 1
E               --- stdout ---
E               laboratory path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/61dfcfe71d7a20cbdaecbe3e856e64bfeb148cf3/lab
E               binary path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/61dfcfe71d7a20cbdaecbe3e856e64bfeb148cf3/lab/bin
E               input path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/61dfcfe71d7a20cbdaecbe3e856e64bfeb148cf3/lab/input
E               work path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/61dfcfe71d7a20cbdaecbe3e856e64bfeb148cf3/lab/work
E               archive path:  /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/61dfcfe71d7a20cbdaecbe3e856e64bfeb148cf3/lab/archive
E               Experiment name is configured in config.yaml:  exp_default_runtime
E               payu: Found modules in /opt/Modules/v4.3.0
E               Loading input manifest: manifests/input.yaml
E               Loading restart manifest: manifests/restart.yaml
E               Loading exe manifest: manifests/exe.yaml
E               Setting up atmosphere
E               Setting up ocean
E               Setting up ice
E               
E               --- stderr ---
E               Loading access-esm1p6/pr181-9
E                 Loading requirement: access-esm1p6/dependencies/pr181-9/oasis3-mct/5.2-dzwdenl
E                   access-esm1p6/dependencies/pr181-9/cice5/2026.01.000-3ecdmyk
E                   access-esm1p6/dependencies/pr181-9/access-fms/mom5-2025.08.000-ltn3e2q
E                   access-esm1p6/dependencies/pr181-9/access-mocsy/2025.07.002-ejsylbt
E                   access-esm1p6/dependencies/pr181-9/access-generic-tracers/2025.09.000-x2czqdx access-esm1p6/dependencies/pr181-9/mom5/2025.05.000-jotcb6x access-esm1p6/dependencies/pr181-9/cable/2025.11.000-7ey6m4f access-esm1p6/dependencies/pr181-9/gcom4/2025.08.000-qhkz6ut access-esm1p6/dependencies/pr181-9/um7/2025.12.000-oz4papa
E               Traceback (most recent call last):
E                 File "/opt/conda/payu-dev-20260130T053924Z-1346ed4/bin/payu", line 7, in <module>
E                   sys.exit(parse())
E                 File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260130T053924Z-1346ed4/lib/python3.10/site-packages/payu/cli.py", line 52, in parse
E                   run_cmd(**args)
E                 File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260130T053924Z-1346ed4/lib/python3.10/site-packages/payu/subcommands/setup_cmd.py", line 28, in runcmd
E                   expt.setup(force_archive=force_archive)
E                 File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260130T053924Z-1346ed4/lib/python3.10/site-packages/payu/experiment.py", line 63, in wrapper
E                   result = func(self, *args, **kwargs)
E                 File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260130T053924Z-1346ed4/lib/python3.10/site-packages/payu/experiment.py", line 523, in setup
E                   model.setup()
E                 File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260130T053924Z-1346ed4/lib/python3.10/site-packages/payu/models/cice5.py", line 71, in setup
E                   super(Cice5, self).setup()
E                 File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260130T053924Z-1346ed4/lib/python3.10/site-packages/payu/models/cice.py", line 164, in setup
E                   super(Cice, self).setup()
E                 File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20260130T053924Z-1346ed4/lib/python3.10/site-packages/payu/models/model.py", line 315, in setup
E                   raise FileNotFoundError(
E               FileNotFoundError: Executable for ice model not found on path: /scratch/tm70/repro-ci/experiments/access-esm1.6-configs/61dfcfe71d7a20cbdaecbe3e856e64bfeb148cf3/lab/bin/cice_access-esm1.6_360x300_12x1_12p.exe

../test-venv/lib/python3.10/site-packages/model_config_tests/exp_test_helper.py:154: RuntimeError