Test diagnostics and diag system

[nichannah]

Issue #62 will result in some changes to the diagnostic infrastructure. Before doing this we need a way to test the diagnostics to ensure that no regressions are introduced. Tests for diagnostic output would be very useful for other reasons also.

Three kinds of tests are proposed:

1. make checksums of diagnostic output fields and test against a baseline to find regressions.
2. some 'absolute' checks, for example checking that there are no NaNs in the output, or comparing against some expected field statistics.
3. checking the that all diagnostics actually work. i.e. ensure that we can get output for all diagnostics in the code.

[nichannah]

Introduction

Presently the model is tested for regressions by comparing checksums of key values between code changes. This allows scientific changes to be detected and then verified. The outputs of the model are not tested. In particular this means that incorrect calculation of diagnostic outputs will go unnoticed.

The purpose of this task is to support testing of diagnostics. One motivation to do this now is so that changes to the diagnostic manager within MOM can be made.

This task will introduce three kinds of system-level tests:
1. Compare checksums on diagnostic output against previous checksums. Call these relative system tests.
2. Do arbitrary pass/fail checks on particular fields. For example checking agains expected statistics. Call these absolute system tests.
3. Check that all diagnostics in the code do work.

Requirements

• Test that all available diagnostics are in fact output.
• Cover ocean and ice diagnostics.
• Detect changes to diagnostic output.
• It should be easy to update the baseline if legitimate changes to diagnostics do occur. i.e. just do a 'cp' or 'mv'.
• Easy to extend and add new absolute system tests.

Design

Steps:
1. Run the test case and get a list of all available diagnostics for that test case.
2. Put each of the available diagnostics into the diag_table with it's own output file and frequency 0.
3. Run the test case again. It should output all available diagnostics at the maximum frequency.
4. Do checks on output from above.

This process is generic, i.e. it is not tied to a particular test case. It could be repeated for several test cases until we know that we've covered absolutely all the diagnostics in the source code. Presently it only runs on the ice_ocean_SIS2/Baltic test.

Implementation

See MOM-examples/test/test_diagnostic_output.py

Presently, in order to run the tests a GNU repro executable should already exist. Then run:

cd MOM-examples
nosetests

On gaea is may be necessary to do module load python_nose first.