Reorganize, cleanup, and document

Author: fluidnumericsJoe

README.md

@@ -2,47 +2,60 @@
 
 This repository is adapted from https://github.com/quentinjamet/SPECTRE
 
-Main differences include
+Main differences include:
 * MITgcm as a submodule
-* Ocean boundary conditions are from Glorys v12 and the same across all ensemble members
-* All simulation suites are defined by yamls that are used to drive input deck generation and job submission scripts
+* Ocean boundary and initial conditions derived from Glorys v12
+* Atmospheric forcing derived from ERA5, processed through the MITgcm EXF package
+* All simulations are defined by a `config.yaml` that drives input deck generation and job submission
 
 
 ## Get Started
-Clone this repository and recursively grab submodules
 
-```
+Clone this repository and recursively grab submodules:
+
+```bash
 git clone --recurse-submodules https://github.com/ocean-spectre/spectre-ensembles
+cd spectre-ensembles
+```
+
+
+## Repository Layout
+
+```
+spectre-150-ensembles/
+├── MITgcm/                  # MITgcm source (submodule)
+├── env/                     # Environment setup scripts (per host)
+├── opt/                     # MITgcm build option files (per host)
+├── simulations/
+│   └── glorysv12-curvilinear/   # Primary simulation (see its README)
+│       ├── code/            # MITgcm compile-time options
+│       ├── etc/config.yaml  # Simulation configuration
+│       ├── input/           # Static input files and generated forcing
+│       └── workflows/       # Slurm job scripts
+└── spectre_utils/           # Python utilities for pre-processing
 ```
 
 
-## Workflow
-
-Set up template directory (manual work)
-|
-|
-build mitgcm executable
-|
-|
-Define ensemble configurations with simulation sequences
-|
-|
-Create boundary and atm conditions (constant across all members)
-|
-|
-Set up member directories
-|
-|
-Create initial conditions (unique for each member)
-|
-|
-For each member, launch simulation sequences
-
-
-### What's a simulation sequence ?
-
-Defined by
-* sequence stage - a name for this "stage" of the simulation (e.g. spinup, production)
-* map of MITgcm parameters to their values
-* list of required conditions to start
-* list of conditions to verify successful completion 
+## Simulations
+
+### glorysv12-curvilinear
+
+MITgcm re-run of Glorys v12 on the native NEMO curvilinear grid for the North
+Atlantic. See [`simulations/glorysv12-curvilinear/README.md`](simulations/glorysv12-curvilinear/README.md)
+for the full workflow and configuration details.
+
+
+## spectre_utils
+
+Python package containing all pre-processing scripts. All scripts accept a
+`config.yaml` path as their only argument. Key scripts:
+
+| Script | Purpose |
+|--------|---------|
+| `download_glorys12_raw.py` | Download Glorys v12 ocean data from CMEMS |
+| `download_era5.py` | Download ERA5 atmospheric data from CDS |
+| `mk_initial_conditions.py` | Generate MITgcm initial conditions |
+| `mk_ocean_boundary_conditions.py` | Generate open boundary conditions |
+| `mk_exf_conditions.py` | Process ERA5 fields into EXF binary forcing files |
+| `animate_exf_conditions.py` | Animate EXF forcing fields (MP4 per variable) |
+| `review_exf_conditions.py` | QC checks, statistics, and diagnostic figures |

env/derecho.sh

@@ -0,0 +1,120 @@
+# Glorys v12 - MITgcm Curvilinear Grid
+
+MITgcm re-run of the Glorys v12 simulation on the native NEMO curvilinear grid.
+Running on the native grid means no interpolation is needed for GLORYS ocean fields,
+and the Arakawa C-grid velocity fields are numerically divergence-free, which
+avoids the gravity-wave eruptions that typically complicate spinup on remapped grids.
+
+
+## Configuration
+
+| Parameter | Value |
+|-----------|-------|
+| Ocean source | CMEMS Glorys v12 (`cmems_mod_glo_phy_my_0.083deg_P1D-m`) |
+| Atmosphere source | ERA5 single levels (EXF package) |
+| Longitude | -82.0 to -17.5 |
+| Latitude | 26.0 to 50.5 |
+| Vertical levels | 50 |
+| Simulation period | 2002-07-01 – 2017-06-30 |
+| Spinup | 2002-07-01 – 2003-07-01 × 5 repeats |
+| MPI layout | 8 × 8 = 64 ranks |
+
+Open boundary array sizes (5479 daily time steps, 2002–2017):
+
+| Boundary | U/V/T/S shape | Eta shape |
+|----------|--------------|-----------|
+| South | (5479, 50, 768) | (5479, 768) |
+| North | (5479, 50, 768) | (5479, 768) |
+| West | (5479, 50, 424) | — |
+| East | (5479, 50, 424) | — |
+
+
+## Workflow
+
+All steps are run as Slurm jobs from the `workflows/` directory. Each script
+sources `workflows/env.sh` for container image paths and data directories.
+
+### 1. Build MITgcm
+
+```bash
+sbatch workflows/build.sh
+```
+
+Compiles MITgcm inside the base container image using `code/` for compile-time
+options. The executable is written into the simulation directory.
+
+### 2. Download ocean data (GLORYS v12)
+
+```bash
+sbatch workflows/download_glorysv12_raw.sh
+```
+
+Downloads daily GLORYS v12 fields (T, S, U, V, SSH) from CMEMS for the domain
+and years defined in `etc/config.yaml`.
+
+### 3. Download atmospheric data (ERA5)
+
+```bash
+sbatch workflows/download_era5.sh
+```
+
+Downloads ERA5 single-level fields from the Copernicus CDS for each variable
+listed under `atmosphere.variables` in `etc/config.yaml`.
+
+### 4. Generate initial conditions
+
+```bash
+sbatch workflows/make_ocean_initial_conditions.sh
+```
+
+Interpolates GLORYS v12 T, S, U, V, SSH fields onto the MITgcm grid and writes
+binary initial condition files to `input/`.
+
+### 5. Generate ocean boundary conditions
+
+```bash
+sbatch workflows/make_ocean_boundary_conditions.sh
+```
+
+Processes GLORYS v12 daily fields into open boundary condition binary files
+(U, V, T, S, Eta on all four boundaries) in `input/`.
+
+### 6. Generate EXF atmospheric forcing
+
+```bash
+sbatch workflows/make_exf_conditions.sh
+```
+
+Processes ERA5 fields into EXF binary forcing files in `input/`. Applies any
+`scale_factor` values from the config (e.g. for radiation units), and computes
+specific humidity from dewpoint temperature and surface pressure.
+
+To review the produced forcing fields before running the model:
+
+```bash
+sbatch -w franklin -c8 --wrap \
+  "srun --container-image=\$SPECTRE_UTILS_IMG \
+        --container-mounts=\$(pwd)/../:/workspace,\$HOST_DATADIR:/data \
+        python /opt/spectre_utils/review_exf_conditions.py /workspace/etc/config.yaml"
+```
+
+Output (report + figures) is written to `review/atmosphere/`.
+
+### 7. Run the model
+
+```bash
+sbatch workflows/run.sh
+```
+
+Launches MITgcm under the base container image. Sets up the run directory on
+first launch, then submits the MPI job. The run directory is controlled by
+`RUN_DIR` in `run.sh` (default: `demo/`).
+
+
+## Data sources
+
+| Dataset | Access | Variables |
+|---------|--------|-----------|
+| CMEMS Glorys v12 daily | `copernicusmarine` Python package | thetao, so, uo, vo, zos |
+| CMEMS Glorys v12 static | `copernicusmarine` Python package | mask, deptho, deptho_lev |
+| ERA5 single levels | `cdsapi` Python package | winds, temperature, humidity, radiation, precip, pressure |

env/galapagos-franklin-glorysv12-curvilinear.sh

@@ -30,23 +30,26 @@ atmosphere:
       mitgcm_name: "atemp"
     - era_name: "total_precipitation"
       mitgcm_name: "precip"
-    - era_name: "mean_surface_downward_short_wave_radiation_flux"
+    - era_name: "surface_solar_radiation_downwards"
       mitgcm_name: "swdown"
-    - era_name: "mean_surface_net_long_wave_radiation_flux"
+      scale_factor: 2.7778E-04
+    - era_name: "surface_thermal_radiation_downwards"
       mitgcm_name: "lwdown"
+      scale_factor: 2.7778E-04
     - era_name: "mean_sea_level_pressure"
       mitgcm_name: "pressure"
     - era_name: "runoff"
       mitgcm_name: "runoff" 
     - era_name: "evaporation"
       mitgcm_name: "evap"
+    - era_name: "surface_pressure"
+      mitgcm_name: "sp"
   computed_variables:
     - mitgcm_name: "aqh"
       era_name: "q"
       function: "specific_humdity_from_dewpoint"
   years: [2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2015,2016,2017]
   prefix: "era5"
-  cheapaml: True
 
 domain:
   mpi:

env/galapagos-franklin.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 
 # Path where downloaded data is stored
-export HOST_DATADIR=/mnt/beegfs/spectre-150-ensembles/glorysv12-curvilinear-15year
+export HOST_DATADIR=/mnt/beegfs/spectre-150-ensembles/simulations/glorysv12-curvilinear/downloads
 export SPECTRE_UTILS_IMG="docker://ghcr.io#ocean-spectre/spectre-ensembles/spectre-utils:sha-6f41db5"
 export MITGCM_BASE_IMG="docker://ghcr.io#fluidnumerics/mitgcm-containers/gcc-openmpi:latest"

simulations/glorysv12-curvilinear/README.md

@@ -2,9 +2,9 @@
 #SBATCH -n1
 #SBATCH -c32
 #SBATCH --nodelist=franklin
-#SBATCH --job-name=spectre_cheapaml
-#SBATCH --output=./spectre_cheapaml.out
-#SBATCH --error=./spectre_cheapaml.out
+#SBATCH --job-name=spectre_exf
+#SBATCH --output=./spectre_exf.out
+#SBATCH --error=./spectre_exf.out
 
 if [ -n "${SLURM_JOB_ID:-}" ]; then
     SCRIPT_PATH=$(scontrol show job "$SLURM_JOB_ID" --json | jq -r '.jobs[0].command' )
@@ -18,8 +18,8 @@ fi
 source $SCRIPT_DIR/env.sh
 
 ###############################################################################################
-# Run the script to download make the cheapaml boundary conditions
+# Run the script to download make the exf boundary conditions
 ###############################################################################################
 srun --container-image=$SPECTRE_UTILS_IMG \
      --container-mounts=${SCRIPT_DIR}/../:/workspace,${HOST_DATADIR}:/data \
-     python /opt/spectre_utils/mk_cheapaml_conditions.py /workspace/etc/config.yaml
+     python /opt/spectre_utils/mk_exf_conditions.py /workspace/etc/config.yaml