Usability improvement for `test repro` messages

[manodeep]

Currently !test repro command only outputs "Error" or "Success", depending on whether the PR produced identical checksums or not. However, there is an additional case - where the model fails to run/aborts, and that falls into the "Error" category, but should really be flagged differently. I am proposing/requesting that there be two modes of "fail":

1. When the model fails to run, ie the job exitcode is non-zero -> this should be treated as "Error" and marked accordingly
2. When the model runs successfully, but the checksum does not match -> this should be treated as "Checksum Failed"

A tiny bit more context is around here

[jo-basevi]

This feature should now be added in #173, which checks if the model ran without errors as part of test setup - so any exceptions raised will be treated as test errors (rather than failures), and so the automated comment on the pull request, will report a test error vs a repro failure.

[manodeep]

Fantastic - thanks Jo! Is this now immediately available in build-cd?

[jo-basevi]

In the case of the PR linked above, yes, because it looks like the version of model-config-tests used for all tests in access-esm1.6-configs uses the main branch in this repository (https://github.com/ACCESS-NRI/access-esm1.6-configs/blob/7e3215866c37d197f4b60ab9b841cdc99b86433c/config/ci.json#L19). However for other config repos, they might use a fix version of model-config-tests for released branches. So once we are sure it's working ok, then can add a new version of model-config-tests and update other repositories.

[manodeep]

@jo-basevi I was not expecting a failure/error for this bitwise repro test - will you please check whether I have missed some step?

(It's not clear to me how to go about manually comparing the checksums from the 1-day run against the existing checksum)

[jo-basevi]

Hmm it looks like the model is exiting with an error - payu: Model exited with error code 19; aborting. On the issue comment, if you click Further Information drop-down and it should have "check the test results at https://github.com/ACCESS-NRI/access-esm1.6-configs/runs/58034755448". If you follow that link and click the Raw output for a test, it should now show the payu job standard output and error logs.

It looks like it's errored out with the same error for both of the historical and determinism repro test runs, so makes me think it wasn't a gadi issue. Was this configuration running fine locally?

[manodeep]

Thanks for your patience Jo. Not really sure what is going down - just added another test to see if anything is different. If this fails, then I will go to one where I have run and know that the results from a 1-year run are bitwise identical with the base config

[jo-basevi]

Closing this issue as models exiting with an error should result in an error result (rather than failure) in #173