Potential non-reproducibility of branch run with noresm3_0_beta12

[matsbn]

I was looking closer at my recent challenges to start sensitivity experiments with RUN_TYPE=branch and instead being forced to use RUN_TYPE=hybrid. It appears the reason for my branch run crashing on startup was because I tried to make a noresm3_0_beta12 based branch run from a noresm3_0_beta11 simulation.

When trying branch and hybrid continuations on Betzy from NorESMhub/noresm3_dev_simulations#377, all based on noresm3_0_beta12, both technically worked. For the branch and hybrid continuations I used identical configuration to NorESMhub/noresm3_dev_simulations#377. As expected the hybrid continuation was not bit-identical. However, the branch continuation was also not bit-identical, contrary to expectation.

Case, run and archive folders for hybrid run:

/cluster/projects/nn9560k/matsbn/NorESM/cases/n1850.ne16pg3_tn14.noresm3_0_beta12.Run8_hybrid.2026-03-13
/cluster/work/users/matsbn/noresm/n1850.ne16pg3_tn14.noresm3_0_beta12.Run8_hybrid.2026-03-13/run
/cluster/work/users/matsbn/archive/n1850.ne16pg3_tn14.noresm3_0_beta12.Run8_hybrid.2026-03-13

Case, run and archive folders for branch run:

/cluster/projects/nn9560k/matsbn/NorESM/cases/n1850.ne16pg3_tn14.noresm3_0_beta12.Run8_branch.2026-03-13
/cluster/work/users/matsbn/noresm/n1850.ne16pg3_tn14.noresm3_0_beta12.Run8_branch.2026-03-13/run
/cluster/work/users/matsbn/archive/n1850.ne16pg3_tn14.noresm3_0_beta12.Run8_branch.2026-03-13

To reduce that chance of having messed up the configuration in the branch run, I made a clone of NorESMhub/noresm3_dev_simulations#377, and configured it as a branch continuation. Case, run and archive folders of this cloned branch run:

/cluster/projects/nn9560k/matsbn/NorESM/cases/n1850.ne16pg3_tn14.noresm3_0_beta12.Run8_clone.2026-03-13
/cluster/work/users/matsbn/noresm/n1850.ne16pg3_tn14.noresm3_0_beta12.Run8_clone.2026-03-13/run
/cluster/work/users/matsbn/archive/n1850.ne16pg3_tn14.noresm3_0_beta12.Run8_clone.2026-03-13

All simulations have RUN_REFDATE=0596-01-01 and set to run one month. The branch run and cloned branch runs gives bit-identical results (both differing from NorESMhub/noresm3_dev_simulations#377).

I'm hesitant to file this as a bug, since NorESMhub/noresm3_dev_simulations#377 is not based on a "clean" noresm3_0_beta12 code base, since the BLOM tag v1.12.26 is replaced with v1.12.30 to enable recent CMIP7 diagnostic output that I needed for other sensitivity experiments. Anyway, I'm doubtful this has an impact on non-reproducibility. I have not had time to redo these tests with a "clean" codebase.

I wonder if our standard regression testing, close to cold starts of components, might not be picking up reproducibility issues that could arise when the simulated state is more evolved.

Pinging the tag team here: @mvertens, @gold2718, @TomasTorsvik, @monsieuralok, @mvdebolskiy. I would be grateful if someone could test this independently based on recent noresm3_0 dev simulations.

[mvertens]

@matsbn - thanks for the careful summary of this problem. I've had a look at your log files from both the original run and the branch run - and the surprising result from looking at the global integrals in the atm.log files is that answers start changing at time step 175228 although the branch starts at time step 175201. I've never seen this kind of time lag in differences showing up. I'll try to reproduce your case - but also to see if restarts using the branch run are bfb.

[mvertens]

@matsbn - I have confirmed your results. However, I am now wondering if there is something wrong with the initial run NorESMhub/noresm3_dev_simulations#377 (I'll refer to this as run_377 below)

I have done the following:

• Ran a 2 month branch run starting from run_377 with RUN_REFDATE=0596-01-01 and wrote restart files every month. I''ll call this run_377_branch.
• Restarted run_377_branch from the beginning of the second month and compared the result to run_377_branch at the end of the second month.
  The results are bfb on restart. So I am not finding any restart problem at this point with noresm3_0_beta12.

However, if there was a problem with run_377 that somehow occurred at time step 175201 - this would then explain why we are suddenly seeing differences part way through the day. If my hypothesis is correct, we have a real problem with betzy.
I think another test that is worth doing is branching from anther point with respect to run_377 and seeing if we continue to have divergences.

My case directory is:
/cluster/work/users/mvertens/noresm/n1850.ne16pg3_tn14.noresm3_0_beta12.Run8_branch.2026-03-13
My run directory is
/cluster/work/users/mvertens/noresm/n1850.ne16pg3_tn14.noresm3_0_beta12.Run8_branch.2026-03-13/run
No short term archiving for these tests:
DOUT_S=FALSE

[matsbn]

Thanks for testing further, @mvertens. It sounds like a good idea to try a branch run from another restart. There are several restarts here:

/cluster/work/users/matsbn/archive/n1850.ne16pg3_tn14.noresm3_0_beta12.Run8.2026-03-11/rest

[TomasTorsvik]

Maybe related. I did a bunch of tests with noresm3_0_beta03b in October. Somehow, the runs on 20251021 deviated from all later runs, but I was not able to reproduce the original errors.

I have not been able to reproduce the errors from the first 4 runs. I have now run a total of 14 runs, which all give the same output as run5. I suppose there could have been some Betzy system issue on 2025-10-21, since this was the date when runs were inconsistent. However, there was no error message to indicate that something went wrong in these runs, which is worrying.

I created some difference files in

/cluster/work/users/tomast/archive/n1850.ne16_tn14.diff_run1_run5_yr2/

Looking at the blom.hd.0002-01.nc I see changes in SST from January 1. In blom.hm.0002-01, templvl show differences at the surface, but not at depth. So from this it looks like the ocean-atmosphere interface could be a possible source of errors.

Originally posted by @TomasTorsvik in #670

[mvertens]

@TomasTorsvik @matsbn - branching from 0621-02-01-00000 - the results are bfb for the whole of January from the log files IF you compare to the earlier log file where the run was stopped - but the, cam history files are different a month after branching.

/nird/datalake/NS9560K/noresm3/cases/n1850.ne16pg3_tn14.noresm3_0_beta12.Run8.2026-03-11/logs/atm.log.1465364.260321-101311   : nstep, te   615264   0.26077595243196125E+10   0.26077597673645787E+10   0.13468857844655732E-04   0.98512002769676066E+05   0.20548077300190923E+03
/nird/datalake/NS9560K/noresm3/cases/n1850.ne16pg3_tn14.noresm3_0_beta12.Run8.2026-03-11/logs/atm.log.1466901.260323-060809.gz: nstep, te   615264   0.26070487188632631E+10   0.26070481428597698E+10  -0.31919998942082696E-04   0.98513448664088137E+05   0.20548077300190923E+03
branch   atm.log                                                                                                                nstep, te   615264   0.26077595243196125E+10   0.26077597673645787E+10   0.13468857844655732E-04   0.98512002769676066E+05   0.20548077300190923E+03

This indicates a system problem on betzy to me. I plan to do a similar comparison on olivia using the following:
Run to duplicate branch problem on olivia:

Case folder: /cluster/projects/nn9560k/kjetisaa/n1850.ne16pg3_tn14.noresm3_0_beta12.Run9.Olivia.2026-03-12
Restar files: /cluster/work/projects/nn9560k/kjetisaa/archive/n1850.ne16pg3_tn14.noresm3_0_beta12.Run9.Olivia.2026-03-12/rest

[mvertens]

@matsbn - I did a one one month branch run on olivia from 0831-01-01 of /cluster/work/projects/nn9560k/kjetisaa/archive/n1850.ne16pg3_tn14.noresm3_0_beta12.Run9.Olivia.2026-03-12/rest and after a month the cam history files were bfb identical between my branch run and original run from @kjetilaas.
My casedir is
/cluster/work/projects/nn9560k/mvertens/noresm/n1850.ne16pg3_tn14.noresm3_0_beta12.Run9.Olivia.2026-03-12_branch
My rundir is
/cluster/work/projects/nn9560k/mvertens/noresm/n1850.ne16pg3_tn14.noresm3_0_beta12.Run9.Olivia.2026-03-12_branch/run.