Emission driven run crashing in FATES

[kjetilaas]

Brief summary of bug

(copying email from Jerry here)

My esm-hist crashes, what it seems like related to Fates, with the following error:

1228: FATES is having difficulties fusing very small patches.
1228: It is possible that a either a secondary or primary
1228: patch has become the only patch of its kind, and it is
1228: is very very small. You can test your luck by
1228: disabling the endrun statement following this message.
1228: FATES may or may not continue to operate within error
1228: tolerances, but will generate another fail if it does not.

Any suggestions how to fix this?

The case and run directories are:
/cluster/projects/nn9560k/tjiputra/NorESM/NorESM3_dev_cases/nHISTesm.ne16pg3_tn14.noresm3_0_beta11_20260311
/cluster/work/users/tjiputra/noresm/nHISTesm.ne16pg3_tn14.noresm3_0_beta11_20260311/run/

@maritsandstad @mvdebolskiy @mvertens @rosiealice @JessicaNeedham

[kjetilaas]

Copy of PET logfile:

20260316 074310.274 ERROR PET0846 Unknown error submitted to shr_abort_abort.
20260316 074311.007 INFO PET0846 Finalizing ESMF with endflag==ESMF_END_ABORT
20260316 074311.007 ERROR PET0846 /cluster/work/users/mdeb/eblds/ESMF/8.8.0/iomkl-2022a-ParallelIO-2.6.5/esmf-8.8.0//src/Infrastructure/Trace/src/ESMCI_Trace.C:1816 ESMCI::TraceEventRe
gionExit() Wrong argument specified - Trace regions not properly nested exiting from region: [ESMF] Expected exit from: fates_restart
20260316 074311.007 ERROR PET0846 /cluster/work/users/mdeb/eblds/ESMF/8.8.0/iomkl-2022a-ParallelIO-2.6.5/esmf-8.8.0//src/Infrastructure/Trace/src/ESMCI_Trace.C:1258 ESMCI::TraceClose()
Wrong argument specified - Internal subroutine call returned Error
20260316 074311.007 ERROR PET0846 ESMF_Trace.F90:102 ESMF_TraceClose() Wrong argument specified - Internal subroutine call returned Error

[maritsandstad]

The cesm-logfile is much more instructive than the PET logfile. There are patches that are not managing to fuse with the landuse.

"1228: current landuse state vector: 0.000000000000000E+000 0.465687973498925
1228: 2.937490070904103E-009 0.148476573597735 0.385660828718496
1228: current landuse state vector (not including bare gruond):
1228: 0.000000000000000E+000 0.465769306716242 2.938003108671659E-009
1228: 0.148502505290444 0.385728185055309
1228: driver data landuse state vector: 1.590531138186522E-011
1228: 0.469175567061339 8.572480180729370E-006 0.125211219186274
1228: 0.405604641256302
1228: min_allowed_landuse_fraction: 1.000174651745256E-006
1228: landuse_vector_gt_min: F T T T T
1228: trans matrix from: 1 0.000000000000000E+000 0.000000000000000E+000
1228: 0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
1228: trans matrix from: 2 0.000000000000000E+000 0.000000000000000E+000
1228: 0.000000000000000E+000 0.000000000000000E+000 1.061446970186978E-007
1228: trans matrix from: 3 0.000000000000000E+000 0.000000000000000E+000
1228: 0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
1228: trans matrix from: 4 0.000000000000000E+000 2.126118390244749E-006
1228: 2.938003108671660E-009 0.000000000000000E+000 1.424808643509740E-006
1228: trans matrix from: 5 0.000000000000000E+000 1.307658011800494E-008
1228: 0.000000000000000E+000 0.000000000000000E+000 0.000000000000000E+000
1228: removing all patches with land use type 3
1228: fusing into patch with types, age, and size of: 5 13
1228: 1032.35956488727 3856.60828718496
1228: fusing away patch with types, age, and size of: 3 13
1228: 0.000000000000000E+000 2.937490070904103E-005
1228: resetting currentSite%landuse_vector_gt_min(i) to .false.
1049: GETRDN
1049: 2.98110707821493
1049: 0.766550244353803
1049: 0.549922975950471
1049: -0.301795779677246
1049: 0.881562735577173 0.118437264422827
1049: NaN NaN
1049: 0.000000000000000E+000 0.000000000000000E+000
1049: 1.00000000000000
1049: 0.000000000000000E+000
1049: 0.194100487721157
1049: 1.52431265678027
1049: ENDRUN:
1049: ERROR in TwoStreamMLPEMod.F90 at line 392"

This is happening at multiple coordinates.

Stacktrace:

"
1049: Unknown error submitted to shr_abort_abort.
1049: Image PC Routine Line Source
1049: cesm.exe 00000000037D3087 Unknown Unknown Unknown
1049: cesm.exe 0000000003519A02 shr_abort_mod_mp_ 110 shr_abort_mod.F90
1049: cesm.exe 0000000002DF6DC0 twostreammlpemod_ 258 TwoStreamMLPEMod.F90
1049: cesm.exe 0000000002DFE453 twostreammlpemod_ 1653 TwoStreamMLPEMod.F90
1049: cesm.exe 0000000002A887D5 fatesradiationdri 169 FatesRadiationDriveMod.F90
1049: cesm.exe 00000000025CF2C8 clmfatesinterface 3042 clmfates_interfaceMod.F90
1049: cesm.exe 0000000002D7E4A9 surfacealbedomod_ 1069 SurfaceAlbedoMod.F90
1049: cesm.exe 0000000002590BC8 clm_driver_mp_clm 1297 clm_driver.F90
1049: cesm.exe 000000000252097D lnd_comp_nuopc_mp 915 lnd_comp_nuopc.F90
"

[maritsandstad]

Seems to be happening immediately when the mediater is trying to setup from the restart:

-------->(med_phases_restart_read) mediating for: 1975 1 1 0 0 0 0
1528: FATESWARN: 5 m: small nocomp patch wasnt able to find another patch to fuse wit
1528: h. nocomp pft: 12lu label: 2area: 0.641336E-02
1528: FATESWARN: 5 m: small nocomp patch wasnt able to find another patch to fuse wit
1528: h. nocomp pft: 11lu label: 2area: 0.256978E-02
1528: FATESWARN: 5 m: small nocomp patch wasnt able to find another patch to fuse wit
1528: h. nocomp pft: 2lu label: 2area: 0.251774E-02
1502: FATESWARN: 5 m: small nocomp patch wasnt able to find another patch to fuse wit
1502: h. nocomp pft: 13lu label: 2area: 0.673022E-03
1502: FATESWARN: 5 m: small nocomp patch wasnt able to find another patch to fuse wit
1502: h. nocomp pft: 13lu label: 1area: 0.405279E-03
1448: FATESWARN: 5 m: small nocomp patch wasnt able to find another patch to fuse wit
1448: h. nocomp pft: 13lu label: 2area: 0.998182E-02

(This from within the part of the run that is reading the file).

[maritsandstad]

I may be interpreting incorrectly, but as I see it it is reading in a radiation restart file from CAM, then trying to process the absorbed radiation by the canopy and in there there is some patch fusing is failing (line 1653 of 1653 TwoStreamMLPEMod.F90)

@JessicaNeedham @rosiealice, do you agree? Is this something we should talk to Will Wieder about?

[maritsandstad]

But... This is run with beta11 it seems, so before our restart-fixes that were incorporated in beta12. @tjiputra @kjetilaas can you please test this again with a beta12 checkout? Fingers crossed that just solves the issue...? Or what do you think @mvertens ?

[maritsandstad]

I may be interpreting incorrectly, but as I see it it is reading in a radiation restart file from CAM, then trying to process the absorbed radiation by the canopy and in there there is some patch fusing is failing (line 1653 of 1653 TwoStreamMLPEMod.F90)

@JessicaNeedham @rosiealice, do you agree? Is this something we should talk to Will Wieder about?

Maybe see message above and not do anything for now to see if running with beta12 will just solve it for him

[rosiealice]

I feel like the patch fusion messages here might be a red herring. Given that it then goes straight for the crash in twostream radiation? We always get a lot of output from the patch fusion process... (I guess lots of this land use code is quite new and we haven't optimised its output yet.

I'm not certain Will is the person to talk to - I'd more likely discuss this with Ryan?

[rosiealice]

My first guess here is that something is strange with the incoming radiation https://github.com/NorESMhub/fates/blob/a0a277edd210f9f105cdd9ada58fcd50c53cc384/radiation/TwoStreamMLPEMod.F90#L1653

[mvertens]

@maritsandstad @rosiealice - I would definitely try this with beta12 given that it is happening right after restart.

[tjiputra]

Tried with beta12 now and still getting the warning message (FATESWARN). Note that this warnings come already further back in time… the model still ran for many years before it eventually crashed.

Thank you for looking into this.

[mvdebolskiy]

The fusion error does not do that much. It' mostly due to bad landuse conditions.
The real error comes from the radiation solver, and it seems that some NaNs are propagating into it. I would run with debug and beta12 from the start of 1978, to see if you get NaNs earlier in the call sequence.

[tjiputra]

@mvdebolskiy , i can try that. Can you tell me how run run in debug mode?
Thank you

[mvdebolskiy]

./xmlchange DEBUG=TRUE
./case.build --clean-all
./case.build

[mvdebolskiy]

@mvertens it's not happening right after restart. If you look at the med.log*, the simulation started in 1975 and died in 1978.

[mvertens]

@mvdebolskiy - thanks for pointing this out. I had not looked at the log files - that's what I was understanding from the comments. Your suggestion of running with DEBUG makes sense.