Is `repro` keyword use confusing?

[anton-seaice]

We use the word repro with two different meanings:

1. It is a pytest marker:

Current markers include:

repro: All available reproducibility tests (all repro_ test markers but repro_determinism_restart).

2. It is a !test keyword:

Usage is as follows:

!test TYPE [commit]

Where TYPE is a test suite that we support. Currently, this consists of repro.

However, the meaning is different. In the second case, it runs the tests configured in that repository as reproducability tests, e.g. the ci.json could say:

"reproducibility": {
        "default": {
            "markers": "repro and (not slow)"
        },
        "dev-MC_100km_jra_ryf": {
            "markers": "repro"
        },
        "dev-MC_100km_jra_ryf+wombatlite": {
            "markers": "repro_historical and repro_determinism"
        },
}

So - when users run the !test repro command in a comment, they might run the tests which use the pytest marker repro or they might not. Users can't run !test repro_determinism.

I think this is confusing!

@CodeGat @minghangli-uni @aidanheerdegen

[CodeGat]

Fair enough, that is a bit confusing!

The original reason for using !test repro specifically was because I envisaged a future where performance tests would be run in a similar way in model-config-tests (eg. with !test performance or something.

Currently yes, it does take what pytest markers are used to test reproducibility via the config/ci.json - I remember that there was a possibility of specifying markers outside of what was defined in config/ci.json (for example, !test repro markers='repro_historical and repro_determinism' or something) but it hasn't been too high on the priority list - this might bump it up.

In terms of what could be changed now, the pytest markers could always be edited. In future though - would you prefer the !test TYPE to be something other than repro? What could that be, without obscuring the meaning of what we are testing?

[jo-basevi]

Oops sorry I should've thought about the name clash of repro when changing the test markers a couple months ago.. It is possible to change them again but a question will be what is a good name to make it more clear.

I remember that there was a possibility of specifying markers outside of what was defined in config/ci.json. (for example, !test repro markers='repro_historical and repro_determinism' or something)

This could be useful option for running that one reproducibility test that isn't a part of the "repro" marker - repro_determinism_restart. Could it also override the other options in the ci.json file, like versions of tests/payu?