Verification files

Bris-inference can generate verification files compatible with the Verif program (https://github.com/WFRT/verif). These files contain both forecasts and observations for point locations and together with the Verif tools let's you do extensive analysis of the performance of the model.

This can be configured by adding a Verif output in the config file:

outputs:
  - verif:
      filename: output_filename.nc
      variable: 2t
      obs_sources:
        - verif:
            filename: observation_file.nc

A Verif file can be produced for a single variable, which is defined by variable. This must be the name of a variable that the model produces. This can also be used for supported derived variables (currently only "ws" for wind-speed).

filename is the name of the output file. obs_sources is a list of observation sources, described next.

Observations

Verif NetCDF files typically contain forecasts and observations. In many cases, the user has available a Verif file containing observations. These files can be used to provide observations for this output.

However, Bris inference can also parse NetCDF files with a simpler format, containing only the observations. So if you do not already have Verif files we recommend that you dump observations and station metadata onto a NetCDF file with this structure:

   netcdf format {
   dimensions:
      time = UNLIMITED;
      location = 10;
   variables:
      int time(time);               // Valid time of forecast initialization in
                                    // number of seconds since 1970-01-01 00:00:00 +00:00
      float lat(location);          // Decimal degrees latitude
      float lon(location);          // Decimal degrees longitude
      float altitude(location);     // Altitude in meters
      float obs(time, location);    // Observations
      }

Verification against analysis

Bris inference can also be used to generate Verif files that compare forecast to the analysis (validation/test data). This is done by specifying an Anemoi zarr dataset as source to the verif output:

verif_dataset:
  dataset:
    dataset: PATH_TO_DATASET
    trim_edge: 160
  thinning: 20

outputs:
  - verif:
      filename: output_filename.nc
      variable: 2t
      obs_sources:
        - anemoidataset:
            dataset: ${verif_dataset}            
            variable: 2t

In the above example we sub-sample the verif dataset using the anemoi-datasets functions thinning and trim_edge. Bris inference will interpolate the prediction to the points in verif_dataset (or compare directly if the coordinates are in the prediction output).

Maximum interpolation distance

This output will interpolate the forecast fields to each observation point. In some cases, these observation points can be way outside the forecast field (for example if global observations are used to verify regional forecasts). This is not ideal for verifying the forecasts. This can be alleviated by setting max_distance to a distance in meters, which will throw out observation points that are too far away from the nearest forecast grid point.

Consensus method

In this context, consensus method means the field used as forecast. For ensemble verification, this can be set to control (member 0) or mean (ensemble mean). For single member forecasts, this is always control. The default is control. Note that the consensus method can be overwritten in verif by setting, for instance by switching to ensemble mean: -fcst=ens_mean

Fair quantiles, thresholds and CRPS

Bris inference adjusts quantiles and threshold probabilities for sampling error by default. To turn this off, set fair_quantile=false and fair_threshold=false. Also, Continuous Ranked Probability Score (CRPS) is fair by default, but can be turned off by setting fair_crps=false:

outputs:
  - verif:
      filename: output_filename.nc
      variable: 2t
      obs_sources:
        - verif:
            filename: observation_file.nc
      consensus_method: control
      fair_quantile: false
      fair_threshold: false
      fair_crps: false