Skip to content

Commit 7e7762e

Browse files
fluidnumericsJoeclaude
fluidnumericsJoe
and
claude
committed
Add runoff experiment, frazil docs, and workflow improvements
- Document frazil ice package and allowFreezing in simulation README - Add repeat-year-runoff experiment with EXF runoff enabled to investigate minimum salinity trends - Support experiment-level override files (data.*) in repeat_year_run.sh - Add --experiment and --n-runs flags to repeat_year_chain.sh - Improve converter and plotter robustness (incomplete flush detection, tile completeness checks) - Consolidate env.sh paths to beegfs - Add postprocess.sh workflow Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 440fea5 commit 7e7762e

10 files changed

Lines changed: 288 additions & 23 deletions

File tree

‎simulations/glorysv12-curvilinear/README.md‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -111,6 +111,40 @@ first launch, then submits the MPI job. The run directory is controlled by
111111
`RUN_DIR` in `run.sh` (default: `demo/`).
112112

113113

114+
## Frazil ice and freezing
115+
116+
This simulation enables the MITgcm frazil ice package (`useFRAZIL=.TRUE.` in
117+
`data.pkg`) together with the `allowFreezing=.TRUE.` flag in `data` PARM01.
118+
119+
**Intention.** In a regional ocean model without a coupled sea-ice model, surface
120+
heat loss during winter can cool seawater below its local freezing point (a
121+
function of salinity and pressure). Without intervention the model would produce
122+
unphysical sub-freezing temperatures, which destabilize the equation of state,
123+
generate spurious convection, and can ultimately blow up the simulation.
124+
125+
**Physical mechanism.** When the frazil package is active, MITgcm checks the
126+
in-situ temperature in every cell at the end of each timestep against the local
127+
freezing point, T_f(S, p). If the temperature falls below T_f, the excess
128+
cooling (T_f - T) is converted into frazil ice formation: the cell temperature
129+
is reset to T_f and the latent heat required to form the implied ice mass is
130+
removed from the ocean heat budget. In effect, the ocean "pays" for the phase
131+
change with latent heat rather than continuing to cool. The `allowFreezing` flag
132+
works in concert by permitting the nonlinear free-surface and vertical mixing
133+
schemes to recognize the freezing-point floor, preventing advection or diffusion
134+
from re-introducing sub-freezing temperatures between frazil corrections.
135+
136+
**Impact on the simulation.**
137+
- Prevents numerical blow-ups in winter, particularly on the Labrador shelf and
138+
in the subpolar gyre where strong surface cooling and fresh meltwater create
139+
conditions favorable for freezing.
140+
- Adds an implicit latent heat sink wherever ice would form, damping the winter
141+
mixed-layer deepening that would otherwise be overestimated.
142+
- Does not simulate ice dynamics, thickness, or transport — it is a
143+
thermodynamic clamp only. Any scientific analysis of ice-affected regions
144+
should note that frazil formation acts as a sub-grid-scale parameterization of
145+
ice-ocean thermodynamics, not a prognostic sea-ice model.
146+
147+
114148
## Data sources
115149

116150
| Dataset | Access | Variables |

‎simulations/glorysv12-curvilinear/code/packages.conf‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,3 +5,4 @@ obcs
55
mnc
66
exf
77
cal
8+
frazil

‎simulations/glorysv12-curvilinear/repeat-year-runoff/data.exf‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
! -----------------------------------------------------------------------------
2+
! data.exf -- example for ERA-Interim 3h, 0.25deg, and a curvilinear model grid
3+
! Edit file names, dates, periods, and the lon/lat metadata to match your files.
4+
! -----------------------------------------------------------------------------
5+
6+
&EXF_NML_01
7+
! General EXF runtime switches
8+
useExfCheckRange = .FALSE., ! disabled — warnings are pre-clamp; windstressmax clamps stress at 2.0 N/m²
9+
useExfYearlyFields = .FALSE., ! we use single (not yearly-suffixed) files
10+
twoDigitYear = .FALSE.,
11+
repeatPeriod = 0.0, ! no global repeat
12+
exf_iprec = 32, ! input precision (32 or 64)
13+
exf_albedo = 0.1,
14+
exf_scal_BulkCdn = 1.0,
15+
! Grid/rotation/wind options for curvilinear grids:
16+
readStressOnAgrid = .FALSE., ! we read winds (uwind/vwind) not stresses by default
17+
rotateStressOnAgrid = .FALSE., ! winds are pre-rotated to model grid by mk_exf_conditions.py
18+
readStressOnCgrid = .FALSE.,
19+
useAtmWind = .TRUE., ! use uwind/vwind to compute stress (and rotate them)
20+
useRelativeWind = .FALSE.,
21+
hu = 10.0, ! height (m) of wind observations (10 m)
22+
ht = 2.0, ! height (m) of temp/humidity (2 m)
23+
zref = 10.0, ! reference height (10 m)
24+
/
25+
26+
&EXF_NML_02
27+
! --- filenames, startdates and periods (seconds) ---
28+
29+
! Wind components (10 m)
30+
uwindfile = 'uwind.bin',
31+
uwindstartdate1 = 20020701,
32+
uwindstartdate2 = 000000,
33+
uwindperiod = 10800.0,
34+
35+
vwindfile = 'vwind.bin',
36+
vwindstartdate1 = 20020701,
37+
vwindstartdate2 = 000000,
38+
vwindperiod = 10800.0,
39+
40+
! 2-m air temperature (K)
41+
atempfile = 'atemp.bin',
42+
atempstartdate1 = 20020701,
43+
atempstartdate2 = 000000,
44+
atempperiod = 10800.0,
45+
46+
! 2-m specific humidity (kg/kg)
47+
aqhfile = 'aqh.bin',
48+
aqhstartdate1 = 20020701,
49+
aqhstartdate2 = 000000,
50+
aqhperiod = 10800.0,
51+
52+
! Shortwave radiation
53+
swdownfile = 'swdown.bin',
54+
swdownstartdate1 = 20020701,
55+
swdownstartdate2 = 000000,
56+
swdownperiod = 10800.0,
57+
58+
! Longwave radiation
59+
lwdownfile = 'lwdown.bin',
60+
lwdownstartdate1 = 20020701,
61+
lwdownstartdate2 = 000000,
62+
lwdownperiod = 10800.0,
63+
64+
! Precipitation
65+
precipfile = 'precip.bin',
66+
precipstartdate1 = 20020701,
67+
precipstartdate2 = 000000,
68+
precipperiod = 10800.0,
69+
70+
! Evaporation
71+
evapfile = 'evap.bin',
72+
evapstartdate1 = 20020701,
73+
evapstartdate2 = 000000,
74+
evapperiod = 10800.0,
75+
76+
! Runoff
77+
runofffile = 'runoff.bin',
78+
runoffstartdate1 = 20020701,
79+
runoffstartdate2 = 000000,
80+
runoffperiod = 10800.0,
81+
/
82+
83+
&EXF_NML_03
84+
! climatological relaxation fields (climsst, climsss, etc.) -- unused here
85+
/
86+
87+
&EXF_NML_04
88+
! All EXF fields are pre-interpolated to the model grid by
89+
! mk_exf_conditions.py. No EXF interpolation metadata needed.
90+
! Wind vectors are pre-rotated to model-grid (i,j) directions.
91+
/
92+
&EXF_NML_OBCS
93+
useOBCSYearlyFields = .FALSE.,
94+
95+
obcsNstartdate1 = 20020701, obcsNstartdate2 = 000000,
96+
obcsNperiod = 86400.0, obcsNrepCycle = 0.0,
97+
98+
obcsSstartdate1 = 20020701, obcsSstartdate2 = 000000,
99+
obcsSperiod = 86400.0, obcsSrepCycle = 0.0,
100+
101+
obcsEstartdate1 = 20020701, obcsEstartdate2 = 000000,
102+
obcsEperiod = 86400.0, obcsErepCycle = 0.0,
103+
104+
obcsWstartdate1 = 20020701, obcsWstartdate2 = 000000,
105+
obcsWperiod = 86400.0, obcsWrepCycle = 0.0,
106+
/

‎simulations/glorysv12-curvilinear/workflows/env.sh‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,6 @@
22

33
# Path where downloaded data is stored
44
export HOST_DATADIR=/mnt/beegfs/spectre-150-ensembles/simulations/glorysv12-curvilinear/downloads
5-
#export SIMULATION_INPUT_DIR=/apps/joe/glorysv12-curvilinear/input ## franklin
6-
export SIMULATION_INPUT_DIR=/mnt/raid/joe/glorysv12-curvilinear/input ## noether
5+
export SIMULATION_INPUT_DIR=/mnt/beegfs/spectre-150-ensembles/simulations/glorysv12-curvilinear/input
76
export SPECTRE_UTILS_IMG="docker://ghcr.io#ocean-spectre/spectre-ensembles/spectre-utils:main"
87
export MITGCM_BASE_IMG="docker://ghcr.io#fluidnumerics/mitgcm-containers/gcc-openmpi:latest"

‎simulations/glorysv12-curvilinear/workflows/postprocess.sh‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
#!/bin/bash
2+
#SBATCH -n1
3+
#SBATCH -c4
4+
#SBATCH --time=3-00:00:00
5+
#SBATCH --job-name=spectre_postproc
6+
#SBATCH --output=%x-%A.out
7+
#SBATCH --error=%x-%A.out
8+
9+
# Post-processor: runs the binary→NetCDF converter and surface field plotter
10+
# as background job steps. Runs until walltime or failure.
11+
12+
if [ -n "${SLURM_JOB_ID:-}" ]; then
13+
SCRIPT_PATH=$(scontrol show job "$SLURM_JOB_ID" --json | jq -r '.jobs[0].command')
14+
SCRIPT_DIR=$(dirname "$(readlink -f "$SCRIPT_PATH")")
15+
SIMULATION_DIR="${SIMULATION_DIR:-$(dirname $SCRIPT_DIR)}"
16+
else
17+
SCRIPT_DIR=$(dirname "$(readlink -f "$0")")
18+
SIMULATION_DIR="${SIMULATION_DIR:-$(dirname $SCRIPT_DIR)}"
19+
fi
20+
21+
source $SCRIPT_DIR/env.sh
22+
23+
echo "======================================="
24+
echo " Post-processor"
25+
echo " Simulation dir: ${SIMULATION_DIR}"
26+
echo " SLURM Job ID: ${SLURM_JOB_ID}"
27+
echo "======================================="
28+
29+
# Step 1: Converter (binary diagnostics → per-tile NetCDF)
30+
srun --ntasks=1 --cpus-per-task=2 --exclusive \
31+
--container-image=$SPECTRE_UTILS_IMG \
32+
--container-mounts=${HOME}:${HOME},${SIMULATION_DIR}:/workspace,${HOST_DATADIR}:/data \
33+
python /opt/spectre_utils/convert_diagnostics_to_netcdf.py \
34+
/workspace \
35+
--poll 60 \
36+
--start-date 2002-07-01 \
37+
--dt 360.0 &
38+
CONVERTER_PID=$!
39+
echo "Converter started: srun PID $CONVERTER_PID"
40+
41+
# Step 2: Plotter (reads NetCDF, writes surface field PNGs)
42+
srun --ntasks=1 --cpus-per-task=2 --exclusive \
43+
--container-image=$SPECTRE_UTILS_IMG \
44+
--container-mounts=${HOME}:${HOME},${SIMULATION_DIR}:/workspace,${HOST_DATADIR}:/data \
45+
python /opt/spectre_utils/plot_surface_fields.py \
46+
/workspace \
47+
--poll 120 \
48+
--start-date 2002-07-01 \
49+
--dt 360.0 &
50+
PLOTTER_PID=$!
51+
echo "Plotter started: srun PID $PLOTTER_PID"
52+
53+
# Wait for either to exit — if one fails, kill the other
54+
wait -n $CONVERTER_PID $PLOTTER_PID
55+
EXIT_CODE=$?
56+
echo "A job step exited with code $EXIT_CODE"
57+
58+
# Clean up the other
59+
kill $CONVERTER_PID $PLOTTER_PID 2>/dev/null
60+
wait
61+
62+
echo "Post-processor finished"
63+
exit $EXIT_CODE

‎simulations/glorysv12-curvilinear/workflows/repeat_year_chain.sh‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,8 @@ for arg in "$@"; do
2828
case "$arg" in
2929
--dry-run) DRY_RUN=true; echo "[DRY RUN] Will print sbatch commands without submitting." ;;
3030
--start=*) START_RUN="${arg#--start=}" ;;
31+
--n-runs=*) N_RUNS="${arg#--n-runs=}" ;;
32+
--experiment=*) EXPERIMENT="${arg#--experiment=}" ;;
3133
esac
3234
done
3335

‎simulations/glorysv12-curvilinear/workflows/repeat_year_run.sh‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,21 @@ srun --ntasks=1 \
5454

5555
echo " > Symlinks created."
5656

57+
###############################################################################
58+
# Step 1b: Copy experiment-level override files (e.g. data.exf) into run dir
59+
###############################################################################
60+
OVERRIDE_DIR="${SIMULATION_DIR}/${EXPERIMENT}"
61+
if compgen -G "${OVERRIDE_DIR}/data*" > /dev/null 2>&1; then
62+
echo "--- Applying overrides from ${EXPERIMENT}/ ---"
63+
for f in "${OVERRIDE_DIR}"/data*; do
64+
fname=$(basename "$f")
65+
dest="${SIMULATION_DIR}/${RUN_DIR}/${fname}"
66+
rm -f "$dest"
67+
cp "$f" "$dest"
68+
echo " > Copied override: ${fname}"
69+
done
70+
fi
71+
5772
###############################################################################
5873
# Step 2: For runs after 001, convert previous pickup to init files
5974
###############################################################################

‎simulations/glorysv12-curvilinear/workflows/run.sh‎

Lines changed: 4 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -34,42 +34,31 @@ echo "======================================="
3434
mkdir -p ${RUN_DIR}
3535
echo ${SLURM_JOB_ID} > ${RUN_DIR}/slurm_job_id
3636

37-
###############################################################################
38-
# Sync namelist files (data*) from beegfs to local input directory
39-
# This ensures the local disk copy always has the latest configuration
40-
###############################################################################
41-
echo "-------------------------------------"
42-
echo " > Syncing namelist files to local input directory..."
43-
cp -v ${SIMULATION_DIR}/input/data* ${SIMULATION_INPUT_DIR}/ 2>/dev/null
44-
cp -v ${SIMULATION_DIR}/input/eedata ${SIMULATION_INPUT_DIR}/ 2>/dev/null
45-
echo " > Done syncing."
46-
echo "-------------------------------------"
47-
4837
###############################################################################
4938
# Set up run directory
5039
###############################################################################
51-
if [[ ! -d "$RUN_DIR" ]]; then
40+
#if [[ ! -d "$RUN_DIR" ]]; then
5241
echo "-------------------------------------"
5342
echo " > Directory $RUN_DIR does not exist. Setting up the run directory now..."
5443
echo ""
5544
srun --ntasks=1 \
5645
--mpi=pmix \
5746
--container-image=$MITGCM_BASE_IMG \
58-
--container-mounts=$SIMULATION_INPUT_DIR:/input,$SIMULATION_DIR:/workspace:rw \
47+
--container-mounts=$SIMULATION_DIR:/workspace:rw \
5948
--container-env=RUN_DIR \
6049
/bin/bash -c /workspace/workflows/run_setup.sh
6150
echo ""
6251
echo " > Done setting up the run directory!"
6352
echo ""
6453
echo "-------------------------------------"
65-
fi
54+
#fi
6655

6756
###############################################################################
6857
# Launch mitgcm under enroot container
6958
###############################################################################
7059
srun --mpi=pmix \
7160
--cpu-bind=cores \
7261
--container-image=$MITGCM_BASE_IMG \
73-
--container-mounts=$SIMULATION_INPUT_DIR:/input,$SIMULATION_DIR:/workspace:rw \
62+
--container-mounts=$SIMULATION_DIR:/workspace:rw \
7463
--container-env=RUN_DIR \
7564
/bin/bash -c /workspace/workflows/run_worker.sh

‎spectre_utils/convert_diagnostics_to_netcdf.py‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,16 @@ def convert_one(data_path, meta, run_dir, Nx, Ny, Nr, nPx, nPy, deltaT, start_da
9595
global_fields[fname] = {"data": raw[offset:offset + fld_size].reshape(shape), "is_3d": fld_3d}
9696
offset += fld_size
9797

98+
# Validate: the first field should have a reasonable number of non-zero values
99+
# (at least 30% — ocean covers ~80% of the domain). If mostly zeros, the
100+
# binary was likely read before MITgcm finished flushing to disk.
101+
first_field = list(global_fields.values())[0]["data"] if global_fields else None
102+
if first_field is not None:
103+
nonzero_frac = np.count_nonzero(first_field) / first_field.size
104+
if nonzero_frac < 0.3:
105+
print(f" SKIP {basename}: only {nonzero_frac:.1%} non-zero — likely incomplete flush")
106+
return False
107+
98108
# Get tile layout from grid files
99109
tile_info = {}
100110
for d in sorted(glob.glob(os.path.join(run_dir, "mnc_*_*/"))):
@@ -152,13 +162,23 @@ def convert_one(data_path, meta, run_dir, Nx, Ny, Nr, nPx, nPy, deltaT, start_da
152162
return written > 0
153163

154164

155-
def find_unconverted(run_dir, prefixes=("state3D", "state2D", "Thermo")):
165+
def find_unconverted(run_dir, prefixes=("state3D", "state2D", "Thermo"), min_age_s=120):
166+
"""Find binary diagnostics files ready for conversion.
167+
168+
Only returns files older than min_age_s seconds to ensure MITgcm
169+
has finished writing and flushing to disk.
170+
"""
171+
import time as _time
172+
now = _time.time()
156173
results = []
157174
for prefix in prefixes:
158175
for meta_path in sorted(glob.glob(os.path.join(run_dir, f"{prefix}.*.meta"))):
159176
data_path = meta_path.replace(".meta", ".data")
160177
if not os.path.exists(data_path):
161178
continue
179+
# Skip files that are too recent (may still be written)
180+
if now - os.path.getmtime(data_path) < min_age_s:
181+
continue
162182
iter_str = os.path.basename(meta_path).split(".")[1]
163183
mnc_dirs = glob.glob(os.path.join(run_dir, "mnc_*_0001/"))
164184
if mnc_dirs:

0 commit comments

Comments
 (0)
⚡