Status test runs from 16.08 #76
Replies: 11 comments 23 replies
-
|
FYI I tested the setup from this morning without the accelerated spinup and that built and ran got to the restart going. Haven't really checked the output yet... We tried this thing, @mvertens : ./create_newcase --case test_noresmfates_config --compset 1850_DATM%GSWP3v1_CLM51%FATES_SICE_SOCN_MOSART_SGLC_SWAV --res ne30pg3_tn14 --run-unsupported --project nn9188k --queue normal --pecount L |
Beta Was this translation helpful? Give feedback.
-
|
But when I turned MOSART off, the manual resubmit restart for BGC, testing the same for the accelerated spinup... But it would be nice if automatic restarts also worked... |
Beta Was this translation helpful? Give feedback.
-
|
With MOSART off for accelerated spinup with FATES the automatic restart dies with the memory issue (like they all do), but the manual restart now chokes on a FATES related issue. Seems it's not happy about not finding GPP in the restart file (on inspection the BGC-run restart file for CLM has GPP as a variable (and a bunch of others) that the FATES restart file doesn't seem to have...) @rosiealice maybe you know why and how this can be solved? |
Beta Was this translation helpful? Give feedback.
-
|
Is it GPP or FATES_GPP? Steanfw either way as fates definitely normally restarts. I will have a look to see if my runs from the other day have this in their restarts. |
Beta Was this translation helpful? Give feedback.
-
|
I think the complaint is about GPP, there is FATES_GPP in the restart file, so is it possible that somewhere the setup doesn't know to read this rather than GPP from the restart file? |
Beta Was this translation helpful? Give feedback.
-
|
@maritsandstad accelerated spinup sets history fields in the namelist. You need to override the history output for FATES. |
Beta Was this translation helpful? Give feedback.
-
|
(For reference, the modelgrid_aliases_nuopc.xml in ccs_config had to be change to allow SOCN for the grid setup) |
Beta Was this translation helpful? Give feedback.
-
You will get mosart spinup after your AD spinup if you just start the model with finidat='clm2.r.spinup_ad.nc`. |
Beta Was this translation helpful? Give feedback.
-
|
Are you doing this with accelerated spin up? I don’t think we -need- to do this for this proof of concept run (except if the concept we are proving is the acceleration, which it’s not really)
…-------------------------
Dr Rosie A. Fisher
Senior Researcher
CICERO Center for International Climate Research
Oslo, Norway
https://cicero.oslo.no<https://cicero.oslo.no/>
n.b. I am on partial medical leave at present. Thanks for your patience!
________________________________
From: mvdebolskiy ***@***.***>
Sent: Friday, August 16, 2024 7:15:25 PM
To: NorESMhub/CTSM ***@***.***>
Cc: Rosie Fisher ***@***.***>; Mention ***@***.***>
Subject: Re: [NorESMhub/CTSM] Status test runs from 16.08 (Discussion #76)
@maritsandstad<https://github.com/maritsandstad> accelerated spinup sets history fields in the namelist. You need to override the history output for FATES.
—
Reply to this email directly, view it on GitHub<#76 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADC2YQ6S3HA5T2IQGU3HWETZRYXS3AVCNFSM6AAAAABMTXFMUCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMZWGE2DEMY>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
So far I don't really think the accelerated spinup is the main stumbling block... I'll try and push a bit further over the Weekend. Hopefully I can get it to get a viable restart... |
Beta Was this translation helpful? Give feedback.
-
|
Without accelerated spinup and MOSART off, the manual restart worked just fine. |
Beta Was this translation helpful? Give feedback.
-
|
Oh great. Thanks Marit! |
Beta Was this translation helpful? Give feedback.
-
|
So the sparse grid case I previously set up -is- restarting. (with the restart happening at the end of the year). I moved the output to here: and the case is here: Getting on a plane to Edinburgh now (yesterday did not in fact go according to plan) but will take a closer look and try some more experiments later... |
Beta Was this translation helpful? Give feedback.
-
|
I have updated ccs_config and config_pes on NorESMhub and made a new tag fates_spinup. I have a spinup running with the following script: #!/bin/bash
git clone https://github.com/NorESMhub/CTSM.git ctsm_nor
cd ctsm_nor
git checkout fates_spinup
./manage_externals/checkout_externals
set -e
# Created 2024-08-17 21:19:51
CASEDIR="/cluster/projects/nn2806k/mdeb/fates2.5_tests/1850_fates_spinup"
./cime/scripts/create_newcase --case "${CASEDIR}" \
--compset 1850_DATM%GSWP3v1_CLM51%FATES_SICE_SOCN_MOSART_SGLC_SWAV --res ne30pg3_tn14 \
--run-unsupported --project nn1004k --queue normal --driver nuopc --pecount L --walltime 26:00:00
cd "${CASEDIR}"
./xmlchange CLM_FORCE_COLDSTART=on,STOP_N=20,STOP_OPTION=nyears,REST_OPTION=nyears,REST_N=10,RESUBMIT=22,MOSART_IGNORE_WARNINGS=TRUE
./case.setup
echo "use_fates_nocomp = .true." >> user_nl_clm
echo "hist_mfilt = 12 \
hist_nhtfrq = 0 \
hist_empty_htapes = .true." >> use_nl_clm
./case.build
./case.submit |
Beta Was this translation helpful? Give feedback.
-
|
Great. That looks good to me in terms of settings. If this works I will find my parameter file and code version and try that as well :) |
Beta Was this translation helpful? Give feedback.
-
|
Not sure if this is the right place to post this update. But xesmf worked like a dream in regridding the spectral element output, and I can make plots just fine or make lat-lon based netcdf-files for analysis if that's a preference. Happy to share the code if you want but might want to clean up a little bit first. otherwise I used this excellent guide to just do monkey-see monkey-do to get it going: https://ncar.github.io/esds/posts/2022/cam-se-regridding/ |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
Plots from your spinuprun @mvdebolskiy using said remapping |
Beta Was this translation helpful? Give feedback.
-
|
@maritsandstad - that is fantastic news. How are you handling the masking? I assume that your destination mesh will have no mesh and you simply regrid from a mask grid to an unmask grid? |
Beta Was this translation helpful? Give feedback.
-
|
I used a mapfile from clm for the model and grid resolution, I think that took care of the masking (or not) for me. This one (/cluster/shared/noresm/inputdata/lnd/clm2/mappingdata/maps/ne30pg3/map_ne30pg3_to_0.5x0.5_nomask_aave_da_c180515.nc ). It might very well be that I didn't do it the right way, but the results looked reasonable to Kjetil... |
Beta Was this translation helpful? Give feedback.
-
|
@mvertens I ran the SP-mode spinup over night. I ran 80 years and that finished in about 12 hours. Output files are here: |
Beta Was this translation helpful? Give feedback.
-
|
Just fwiw, the calibrated parameter file is |
Beta Was this translation helpful? Give feedback.
-
|
@maritsandstad - thanks for this. What should finidat be from your spinup run? |
Beta Was this translation helpful? Give feedback.
-
|
@maritsandstad is traveling today, but as I understand, the finidat should be this file (last restart file from the spinup): |
Beta Was this translation helpful? Give feedback.
-
|
@kjetilaas @maritsandstad @mvdebolskiy - unfortunately I have discovered both a memory leak and restart problems with the non-FATES run I tried to do this weekend. The model crashed at 11 years into the run and when I restarted back a few months - the results were not BFB. However, 11 day restart tests are passing. So right now I have to resolve this before I start the FATES run. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
So, I guess the teams chat is not really visible to most people after, so I thought we could discuss the test runs after today's meeting here
Beta Was this translation helpful? Give feedback.
All reactions