Add repro payu setup check - this should pick up changes to manifests

[jo-basevi]

This PR adds a "repro_payu_setup" test marker, which selects a test that runs "payu setup" with "--reproduce" flag - This is equivalent to setting the following in config.yaml:

manifests:
  reproduce:
    inputs: true
    exe: true
    restart: true

So setup will raise an error if any full hashes in the generated manifests do not match.

Tested this with running model-config-tests -m repro_payu_setup on a configuration with unchanged manifests, and with a changed restart manifest (removed values from a binhash and md5), the error is:

E           Failed: Failed to run payu setup with --reproduce. Error: Loading access-esm1p6/2025.11.002
E             Loading requirement: cice5/access-esm1.6-2025.11.001-6umbsy5 mom5/2025.05.000-tzlv4hw um7/2025.11.000-nwpmrxg
E           Run cannot reproduce: manifest manifests/restart.yaml is not correct
E
E           Full output: laboratory path:  /scratch/tm70/jb4202/tmp/test-model-repro/lab
E           binary path:  /scratch/tm70/jb4202/tmp/test-model-repro/lab/bin
E           input path:  /scratch/tm70/jb4202/tmp/test-model-repro/lab/input
E           work path:  /scratch/tm70/jb4202/tmp/test-model-repro/lab/work
E           archive path:  /scratch/tm70/jb4202/tmp/test-model-repro/lab/archive
E           Experiment name is configured in config.yaml:  dev-preindustrial+concentrations-repro_payu_setup
E           payu: Found modules in /opt/Modules/v4.3.0
E           Loading input manifest: manifests/input.yaml
E           Loading restart manifest: manifests/restart.yaml
E           Loading exe manifest: manifests/exe.yaml
E           Setting up atmosphere
E           Setting up ocean
E           Setting up ice
E           Setting up coupler
E           Checking exe, input and restart manifests
E           Manifest path: stored hash != calculated hash
E             work/atmosphere/restart_dump.astart: md5: f65a5a631c516907330e7f2d4479d5e != f65a5a631c516907330e7f2d4479d5e2

An issue is that Payu raises an error when restart manifests are empty, e.g. ACCESS-OM2:

 Failed: Failed to run payu setup with --reproduce. Error: Restart manifest must exist and be populated if reproduce is configured to True

I wonder if payu could not error here and only error if there was no pre-existing manifest and a new restart manifest was generated?

Related to #9

[codecov]

Codecov Report

❌ Patch coverage is 79.31034% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.38%. Comparing base (b881d91) to head (2998e68).
⚠️ Report is 25 commits behind head on main.

Files with missing lines	Patch %	Lines
...fig_tests/config_tests/test_bit_reproducibility.py	44.44%	10 Missing ⚠️
src/model_config_tests/exp_test_helper.py	94.59%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main     #190       +/-   ##
===========================================
+ Coverage   68.39%   85.38%   +16.98%     
===========================================
  Files          14       22        +8     
  Lines         829     1300      +471     
===========================================
+ Hits          567     1110      +543     
+ Misses        262      190       -72     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[jo-basevi]

Thinking about this more, it could just check if any manifests/input.yaml, manifests/restart.yaml, manifests/exe.yaml had changed..

Originally I thought it is not a good idea as it'll pick up changes to binhashes which can change even if the file is exactly the same. But I think it'll still be useful to update the binhashes on the configurations as it can lead to longer payu setup time in future, as it needs to recalculate the md5 hashes.

[Qian-HuiChen]

Thanks @jo-basevi for the previous work and suggestions!
I change the setup_reproduce to run payu setup (no --reproduce flag) and then check git diff. If the output is not empty, then returns an error.

In this case, empty manifests will not raise an issue.

Also test model-config-tests -m repro_payu_setup on unchanged manifests --> pass.
I changed manifests on git and run model-config-tests -m repro_payu_setup again, an error is raised as expected:

Manifests have been modified. The modified files include: restart.yaml

I am not sure if this is what we want. Happy to chat and change it.

[jo-basevi]

Thanks for adding this change in!!

Thinking about this more, I am wondering whether this should be two tests:

test_setup_reproduce_manifests

• Uses payu setup --repro (will need to fix payu failing on empty manifests first...). This will pick up if md5 hashes have changed (so the files are different).

test_setup_unchanged_manifests

• Catches if paths or binhashes have changed using the git diff logic you have added. If the md5's are the same, it will just need the manifests to be updated with payu setup. This wouldn't be as large of an issue as the configurations are still using the same files but the initial payu setup would take longer as it needs to recalculate all the md5 hashes.

For scheduled tests for tags, only test_setup_reproduce_manifests should run, as if it fails, it will indicate something on NCI has changed.
But for PRs or scheduled tests for branches, both manifest tests should run so the manifests are always kept up to date.

[Qian-HuiChen]

For scheduled tests for tags, only test_setup_reproduce_manifests should run, as if it fails, it will indicate something on NCI has changed.
But for PRs or scheduled tests for branches, both manifest tests should run so the manifests are always kept up to date.

I agree that it is good to split test_setup_reproduce_manifests and test_setup_unchanged_manifests for different usages and triggered differently. I have opened a payu issue to solve the empty manifest handling issue first, and then come back to modify the model-config-tests.

[jo-basevi]

Thanks for adding these changes in!

[jo-basevi]

It could be good add the new markers to the list of markers in the README.md:

https://github.com/ACCESS-NRI/model-config-tests?tab=readme-ov-file#selecting-tests-using-markers

[Qian-HuiChen]

Thanks @jo-basevi for reminding me of updating readme :)

[jo-basevi]

Changes look good! Just a couple very minor comments

[jo-basevi]

Thanks @Qian-HuiChen, changes look good to me! I've run a couple tests and it seems to run as expected

I'm not able to approve this PR as I opened the PR initially. As there doesn't seem to be a way to change authorship, it might make sense to close this PR and re-open a new PR, which I can then approve?

[Qian-HuiChen]

Since Jo cannot approve it on her side, I will approve myself haha

[Qian-HuiChen]

Turns out my approval cannot allow merging because I pushed the last commit. I re-opened this PR #208.