Multi-run dashboard, IC-based breeding, configurable run workflow

Dashboard:
- Takes simulation_dir instead of STDOUT path; discovers all runs
  with STDOUT.0000 and presents a dropdown selector
- Per-run SLURM job ID via slurm_job_id file in each run directory
- All endpoints accept ?run= parameter for run-specific data

Converter & plotter:
- Both take simulation_dir and process all discovered runs
- Converter uses atomic writes (tmp + rename) to prevent partial reads
- Per-run plot/conversion state tracked independently

Bred vectors:
- Rewritten to use IC files (T/S/U/V/Eta.init.bin) instead of pickups
- Each cycle: overwrite member ICs, run from nIter0=0 for 30 days
- breed_vectors.sh: fresh run dir each cycle, copies member ICs,
  generates member-specific data file with nIter0=0 and nTimeSteps
- Production runs start from the pickup at t=30 days

Run workflow:
- RUN_DIR and SIMULATION_DIR configurable via environment variables
- Writes slurm_job_id to run directory for dashboard association
- Syncs data* files to local input dir before each run

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: fluidnumericsJoe

simulations/glorysv12-curvilinear/ensemble/README.md

@@ -23,24 +23,39 @@ far more realistic initial condition perturbations than random noise alone.
 
 ### Breeding cycle
 
+Each member has its own set of initial condition files (`T.init.bin`,
+`S.init.bin`, `U.init.bin`, `V.init.bin`, `Eta.init.bin`). The cycle operates
+on these IC files directly:
+
 ```
-Control:    ─────────────────────────────────────────────►
-                 │                    │
-                 │ perturb            │ rescale & perturb
-                 ▼                    ▼
-Member i:   ─────●════════════════════●════════════════════●───►
-                 cycle 0              cycle 1              cycle 2 ...
-                 (30 days)            (30 days)
+Control ICs:  T.init.bin, S.init.bin, U.init.bin, V.init.bin, Eta.init.bin
+                 │
+                 │  add perturbation (breed_vectors.py init)
+                 ▼
+Member ICs:   T.init.bin, S.init.bin, ... (in member_NNN/)
+                 │
+                 │  run MITgcm from nIter0=0 for 30 days
+                 ▼
+Member pickup at t=30 days (pickup.0000007200.data)
+                 │
+                 │  bred_vector = member_pickup - control_pickup
+                 │  rescale by target_RMS / actual_T_RMS
+                 │  new_IC = control_IC + rescaled_bred_vector
+                 ▼
+Member ICs:   overwritten with new perturbation → next cycle
 ```
 
-Each cycle:
-1. Member starts from `control_state + perturbation`
-2. Runs forward for 30 days (configurable)
-3. At end: `bred_vector = member_state - control_state`
-4. Rescale factor = `target_RMS / actual_RMS` (computed from temperature)
-5. **Same rescale factor applied to ALL variables** (T, S, U, V, SSH) to preserve
-   geostrophic and hydrostatic balance
-6. New perturbation: `control_state + rescale_factor × bred_vector`
+Key points:
+- Every cycle starts from **nIter0=0** — the member's IC files are the
+  perturbation mechanism, not pickup files
+- The **same forcing, grid, and namelist files** are shared across all members
+  (symlinked from the master input directory). Only the IC files differ.
+- Bred vectors are computed from the **pickup at t=30 days**, which captures
+  how the perturbation grew over the cycle
+- The **same rescale factor** (derived from temperature RMS) is applied to
+  all variables to preserve dynamical balance
+- For **production runs** after breeding converges, each member restarts from
+  its pickup at t=30 days (iteration 7200)
 
 ### Design choices
 
@@ -98,42 +113,57 @@ vectors readjust within the first few weeks of the production run regardless.
 
 ### Prerequisites
 
-- Completed control spinup run with at least one permanent pickup file
-- Set `pickup_iter` in `breed_config.yaml` (or leave null to auto-detect latest)
+- Completed control spinup run (1 year)
+- Control run must also be run from nIter0=0 for 30 days (same as members)
+  to produce the reference pickup for bred vector computation
 
 ### Steps
 
 ```bash
 cd simulations/glorysv12-curvilinear
 
-# 1. Initialize 50 perturbed pickups from the control state
+# 1. Initialize 50 perturbed IC files from the control ICs
 uv run python ../../spectre_utils/breed_vectors.py init ensemble/breed_config.yaml
 
-# 2. Run all 50 members for one breeding cycle (SLURM array job)
+# 2. Run all 50 members for one 30-day cycle (SLURM array job)
+#    Each member starts from nIter0=0 with its perturbed ICs
 sbatch --chdir=$(pwd) workflows/breed_vectors.sh
 
-# 3. After all members complete — compute bred vectors and rescale
+# 3. After all members complete — compute bred vectors and overwrite ICs
 uv run python ../../spectre_utils/breed_vectors.py rescale ensemble/breed_config.yaml --cycle 1
 
 # 4. Check convergence (per-variable RMS table)
-uv run python ../../spectre_utils/breed_vectors.py status ensemble/breed_config.yaml --cycle 1
+uv run python ../../spectre_utils/breed_vectors.py status ensemble/breed_config.yaml
 
-# 5. Repeat steps 2–4 for each cycle
-#    Update --cycle 2, 3, ... 8
+# 5. Repeat steps 2–4 for each cycle (2, 3, ... 8)
 ```
 
-### GCP deployment
+### Running the control alongside members
 
-Each member directory (`member_001/` through `member_050/`) is self-contained:
-- Perturbed pickup file (`.data` + `.meta`)
-- `nIter0.txt` with the starting iteration
+The control must also produce a pickup at iteration 7200 (30 days from nIter0=0)
+for the bred vector computation. This can be done:
+- As a separate single run with the same `data` settings as the members
+- On one of the 17 compute nodes alongside 2 members (3 sims per node)
 
-To run on GCP:
-1. Copy the input deck and member pickups to each compute node's local disk
-2. Each member runs standard MITgcm with the member's pickup as the restart file
-3. After all members finish, copy pickups back and run the `rescale` step
+### Transitioning to production
+
+After breeding converges (cycle 5–8):
+1. Each member has a pickup at `member_NNN/run/pickup.0000007200.data`
+2. Copy this pickup to the member's production run directory
+3. Set `nIter0=7200` and full production `endTime`/`nTimeSteps`
+4. Run the production ensemble
 
-## GCP Cost Estimate
+## GCP deployment
+
+Each member directory (`member_001/` through `member_050/`) contains:
+- `T.init.bin`, `S.init.bin`, `U.init.bin`, `V.init.bin`, `Eta.init.bin`
+
+To run on GCP:
+1. Copy the master input deck to each compute node's local disk (one copy per node)
+2. Copy each member's IC files to the node
+3. Set up the member run directory: symlink master input, replace IC symlinks with copies
+4. Run MITgcm with `nIter0=0`, `nTimeSteps=7200`
+5. After all members finish, copy pickups back and run the `rescale` step
 
 ### Cluster configuration
 
@@ -143,28 +173,7 @@ To run on GCP:
 | Login | n1-standard-2 | 1 | SSH access, job submission |
 | Controller | n1-standard-2 | 1 | Slurm controller |
 
-### Compute requirements per cycle
-
-- 50 members ÷ 3 per node = **17 nodes** per cycle
-- 30 sim-days at 12–20 sim-days/wall-hr = **1.5–2.5 wall hours** per cycle
-- 8 cycles × 2.5 hrs = **~20 hours** total wall time (plus ~30 min rescaling between cycles)
-- Total compute: 17 nodes × 20 hrs = **340 node-hours** (conservative)
-
-### Local disk per node
-
-| Data | Size |
-|------|------|
-| EXF forcing (8 variables × 54 GB) | 432 GB |
-| OBC boundary files | 20 GB |
-| Grid, bathymetry, initial conditions | 2 GB |
-| Pickup files (3 members) | 6 GB |
-| Output headroom (diagnostics, pickups) | 40 GB |
-| **Total** | **~500 GB** |
-
-Recommend **1 TB pd-ssd** per compute node, or local NVMe SSD if available
-on the machine type.
-
-### Cost breakdown
+### Cost estimate
 
 | Item | On-demand | Spot (~70% discount) |
 |------|-----------|---------------------|
@@ -173,16 +182,16 @@ on the machine type.
 | n1-standard-2 × 2 × 24 hrs @ $0.095/hr | $5 | $5 |
 | **Total** | **~$3,400** | **~$1,100** |
 
-### Notes
+### Local disk per node
 
-- Spot/preemptible instances are viable since each breeding cycle is only
-  1.5–2.5 hours — short enough to avoid most preemptions
-- The 30-min rescaling step between cycles runs on a single node and is
-  negligible cost
-- Data transfer: ~500 GB input deck upload (one-time) + ~100 MB pickups per
-  cycle (negligible)
-- The control run must also advance 30 days per cycle to provide the reference
-  state — this can run on one of the 17 compute nodes
+| Data | Size |
+|------|------|
+| EXF forcing (8 variables × 54 GB) | 432 GB |
+| OBC boundary files | 20 GB |
+| Grid, bathymetry, other input | 5 GB |
+| Member IC files (3 members × 5 files × 130 MB) | 2 GB |
+| Output headroom (pickups) | 40 GB |
+| **Total** | **~500 GB** |
 
 ## Configuration
 
@@ -213,11 +222,19 @@ consider a shorter `cycle_length_days` to accelerate convergence.
 ```
 ensemble/
 ├── breed_config.yaml       # Breeding parameters
+├── convergence.json        # Per-cycle RMS diagnostics (written by rescale)
 ├── README.md               # This file
 ├── member_001/             # Member 1
-│   ├── pickup.NNNNNNNNNN.data
-│   ├── pickup.NNNNNNNNNN.meta
-│   └── nIter0.txt
+│   ├── T.init.bin          # Perturbed temperature IC
+│   ├── S.init.bin          # Perturbed salinity IC
+│   ├── U.init.bin          # Perturbed zonal velocity IC
+│   ├── V.init.bin          # Perturbed meridional velocity IC
+│   ├── Eta.init.bin        # Perturbed SSH IC
+│   └── run/                # MITgcm run directory (created by breed_vectors.sh)
+│       ├── *.bin → /input/ # Symlinks to master input (forcing, grid, OBC)
+│       ├── T.init.bin      # Copied (not symlinked) from member dir
+│       ├── data            # Member-specific (nIter0=0, nTimeSteps=7200)
+│       └── pickup.0000007200.data  # Output: state at t=30 days
 ├── member_002/
 │   └── ...
 └── member_050/

simulations/glorysv12-curvilinear/ensemble/breed_config.yaml

@@ -12,11 +12,9 @@ grid:
   Ny: 424
   Nr: 50
 
-# Pickup file from the control run to use as the base state
+# Control run directory (relative to ensemble/)
+# Must contain a pickup at iteration 7200 (30 days from nIter0=0)
 control:
-  pickup_prefix: "pickup"
-  # Iteration number of the pickup to use (set after spinup completes)
-  pickup_iter: null
   run_dir: "../test-run-03252026"
 
 # MITgcm run parameters for each breeding member
@@ -28,5 +26,4 @@ member_run:
 
 # Output directories
 paths:
-  ensemble_dir: "."
   member_prefix: "member"

simulations/glorysv12-curvilinear/workflows/breed_vectors.sh

@@ -7,11 +7,16 @@
 #SBATCH --output=breed_%A_%a.out
 #SBATCH --error=breed_%A_%a.out
 
-# Each array task runs one breeding member.
+# Each array task runs one breeding member for one 30-day cycle.
 # SLURM_ARRAY_TASK_ID = member number (1-50)
+#
+# Each member starts from nIter0=0 with its own perturbed IC files
+# (T.init.bin, S.init.bin, U.init.bin, V.init.bin, Eta.init.bin)
+# and runs forward for nTimeSteps (default 7200 = 30 days at dt=360s).
+#
+# The perturbed ICs are created/updated by breed_vectors.py (init or rescale).
 
 MEMBER_ID=$(printf "%03d" $SLURM_ARRAY_TASK_ID)
-MEMBER_DIR="ensemble/member_${MEMBER_ID}"
 
 if [ -n "${SLURM_JOB_ID:-}" ]; then
     SCRIPT_PATH=$(scontrol show job "$SLURM_JOB_ID" --json | jq -r '.jobs[0].command')
@@ -24,50 +29,68 @@ fi
 
 source $SCRIPT_DIR/env.sh
 
+MEMBER_DIR="${SIMULATION_DIR}/ensemble/member_${MEMBER_ID}"
+MEMBER_RUN_DIR="${MEMBER_DIR}/run"
+NTIMESTEPS=${BREED_NTIMESTEPS:-7200}
+IC_FILES="T.init.bin S.init.bin U.init.bin V.init.bin Eta.init.bin"
+
 echo "======================================="
 echo " Breeding member: ${MEMBER_ID}"
-echo " Simulation dir:  ${SIMULATION_DIR}"
-echo " Member dir:      ${MEMBER_DIR}"
+echo " nIter0:          0"
+echo " nTimeSteps:      ${NTIMESTEPS}"
 echo "======================================="
 
-# Read nIter0 for this member
-NITER0=$(cat ${SIMULATION_DIR}/${MEMBER_DIR}/nIter0.txt)
-echo "Starting from iteration: ${NITER0}"
-
 ###############################################################################
-# Set up member run directory if needed
+# Set up member run directory (fresh each cycle)
 ###############################################################################
-if [[ ! -d "${SIMULATION_DIR}/${MEMBER_DIR}/run" ]]; then
-    echo "Setting up member run directory..."
-    mkdir -p ${SIMULATION_DIR}/${MEMBER_DIR}/run
+rm -rf ${MEMBER_RUN_DIR}
+mkdir -p ${MEMBER_RUN_DIR}
 
-    # Symlink input files from the main input directory
-    for f in ${SIMULATION_INPUT_DIR}/*; do
-        ln -sf $f ${SIMULATION_DIR}/${MEMBER_DIR}/run/$(basename $f)
-    done
+# Symlink all files from the master input directory
+for f in ${SIMULATION_INPUT_DIR}/*; do
+    ln -sf $f ${MEMBER_RUN_DIR}/$(basename $f)
+done
 
-    # Symlink namelist files
-    for f in data data.cal data.exf data.kpp data.mnc data.obcs data.pkg data.diagnostics eedata; do
-        ln -sf ${SIMULATION_INPUT_DIR}/$f ${SIMULATION_DIR}/${MEMBER_DIR}/run/$f 2>/dev/null
-    done
+# Remove symlinks for IC files — these will be member-specific copies
+for f in ${IC_FILES}; do
+    rm -f ${MEMBER_RUN_DIR}/$f
+done
 
-    # Override pickup with the member's perturbed pickup
-    ln -sf ${SIMULATION_DIR}/${MEMBER_DIR}/pickup.*.data ${SIMULATION_DIR}/${MEMBER_DIR}/run/
-    ln -sf ${SIMULATION_DIR}/${MEMBER_DIR}/pickup.*.meta ${SIMULATION_DIR}/${MEMBER_DIR}/run/
+# Copy member's perturbed IC files (created by breed_vectors.py)
+for f in ${IC_FILES}; do
+    if [[ -f ${MEMBER_DIR}/$f ]]; then
+        cp ${MEMBER_DIR}/$f ${MEMBER_RUN_DIR}/$f
+    else
+        echo "WARNING: ${MEMBER_DIR}/$f not found"
+    fi
+done
 
-    # Create a member-specific data file with correct nIter0 and nTimeSteps
-    sed "s/nIter0=.*/nIter0=${NITER0},/" ${SIMULATION_INPUT_DIR}/data > ${SIMULATION_DIR}/${MEMBER_DIR}/run/data
+# Copy namelist files from beegfs (latest config, not stale local copy)
+for f in data.cal data.exf data.kpp data.mnc data.obcs data.pkg data.diagnostics eedata; do
+    cp ${SIMULATION_DIR}/input/$f ${MEMBER_RUN_DIR}/$f 2>/dev/null
+done
 
-    echo "Done."
-fi
+# Generate member-specific 'data' file:
+#   nIter0=0, nTimeSteps=NTIMESTEPS, single pickup at end
+CYCLE_SECONDS=$((NTIMESTEPS * 360))
+cat ${SIMULATION_DIR}/input/data | \
+    sed -e "s/^ nIter0=.*/ nIter0=0,/" \
+        -e "s/^ endTime=.*/ nTimeSteps=${NTIMESTEPS},/" \
+        -e "s/^ pChkptFreq=.*/ pChkptFreq=${CYCLE_SECONDS}.0,/" \
+        -e "s/^ chkptFreq=.*/ chkptFreq=0.0,/" \
+        -e "s/^ dumpFreq=.*/ dumpFreq=0.0,/" \
+    > ${MEMBER_RUN_DIR}/data
+
+echo "--- Member data file (key params) ---"
+grep -E '^ nIter0|^ nTimeSteps|^ pChkptFreq|^ chkptFreq|^ deltaT|^ dumpFreq' ${MEMBER_RUN_DIR}/data
+echo "--------------------------------------"
 
 ###############################################################################
-# Run MITgcm for this member
+# Run MITgcm
 ###############################################################################
-cd ${SIMULATION_DIR}/${MEMBER_DIR}/run
+cd ${MEMBER_RUN_DIR}
 
 srun --mpi=pmix \
      --container-image=$MITGCM_BASE_IMG \
-     --container-mounts=${SIMULATION_INPUT_DIR}:/input,${SIMULATION_DIR}:/workspace:rw \
-     --container-env=MEMBER_DIR,NITER0 \
+     --container-mounts=${SIMULATION_INPUT_DIR}:/input:ro,${SIMULATION_DIR}:/workspace:rw \
      /opt/mitgcm/mitgcmuv

simulations/glorysv12-curvilinear/workflows/run.sh

@@ -7,15 +7,16 @@
 #SBATCH --output=%x-%A.out
 #SBATCH --error=%x-%A.out
 
-export RUN_DIR="test-run-03252026/"
+export RUN_DIR="${RUN_DIR:-test-run-03252026/}"
 
 if [ -n "${SLURM_JOB_ID:-}" ]; then
     SCRIPT_PATH=$(scontrol show job "$SLURM_JOB_ID" --json | jq -r '.jobs[0].command' )
     SCRIPT_DIR=$(dirname "$(readlink -f "$SCRIPT_PATH")")
-    SIMULATION_DIR=$(dirname $SCRIPT_DIR)
+    SIMULATION_DIR="${SIMULATION_DIR:-$(dirname $SCRIPT_DIR)}"
 else
     # Fallback for when running the script outside of a Slurm job
     SCRIPT_DIR=$(dirname "$(readlink -f "$0")")
+    SIMULATION_DIR="${SIMULATION_DIR:-$(dirname $SCRIPT_DIR)}"
 fi
 
 source $SCRIPT_DIR/env.sh
@@ -25,9 +26,25 @@ echo ""
 echo " Using simulation directory : ${SIMULATION_DIR}"
 echo " Using run directory        : ${RUN_DIR}"
 echo " Using MITgcm base image    : ${MITGCM_BASE_IMG}"
+echo " SLURM Job ID               : ${SLURM_JOB_ID}"
 echo ""
 echo "======================================="
 
+# Write job ID into the run directory so the dashboard can find it
+mkdir -p ${RUN_DIR}
+echo ${SLURM_JOB_ID} > ${RUN_DIR}/slurm_job_id
+
+###############################################################################
+# Sync namelist files (data*) from beegfs to local input directory
+# This ensures the local disk copy always has the latest configuration
+###############################################################################
+echo "-------------------------------------"
+echo "  > Syncing namelist files to local input directory..."
+cp -v ${SIMULATION_DIR}/input/data* ${SIMULATION_INPUT_DIR}/ 2>/dev/null
+cp -v ${SIMULATION_DIR}/input/eedata ${SIMULATION_INPUT_DIR}/ 2>/dev/null
+echo "  > Done syncing."
+echo "-------------------------------------"
+
 ###############################################################################
 # Set up run directory
 ###############################################################################