fix(metrics): keep pots gauges live at tip (treasury/reserves froze one epoch stale)

dugite_treasury_lovelace / dugite_reserves_lovelace are refreshed from
ls.epochs only on the bulk-sync catch-up branch of apply_fetched_block
and on the forge path. At tip, received blocks take the `at_tip` branch
which calls publish_ledger_view (which does NOT touch the pots atomics),
so once the node reaches tip the gauges freeze at the last catch-up value
and never reflect the reserves->treasury reward-update transfer applied at
an epoch boundary the node did not forge itself.

Observed on preview epoch 1332->1333: the gauges read the exact epoch-1332
pots while the node was correctly in epoch 1333 — nearly triggering a false
reward-divergence investigation. The ledger itself is byte-exact: the
epoch-1333 snapshot has treasury=6820324388335672 reserves=7916699786696095,
matching Koios epoch 1333 to the lovelace. This is purely a metric-staleness
bug, but a dangerous one because it misleads epoch-boundary cross-checks.

Fix: add NodeMetrics::set_pots() (two O(1) atomic stores) and call it from
post_block_apply_updates — the per-block refresh that already reads live
ls.epoch under a fresh read-lock. This runs on every apply path (tip,
catch-up, forge, replay), so the pots gauge is now as live as the epoch
gauge. TDD: metrics::tests::test_set_pots_updates_gauges (red->green).

Also adds a DUGITE_DUMP_LEDGER_PROBE diagnostic to apply_bench that prints
pots + reward-update inputs (bprev_blocks_by_pool, go snapshot, ss_fee,
prev_d) straight from a snapshot — the instrument that distinguished this
metric-lag from a real ledger divergence.

Author: michaeljfazio

crates/dugite-node/src/bin/apply_bench.rs

@@ -381,6 +381,51 @@ fn main() {
         ledger.utxo.utxo_set.len()
     );
 
+    // Diagnostic probe (issue: epoch-boundary reward-pots divergence).
+    // When DUGITE_DUMP_LEDGER_PROBE is set, print the reward-update inputs and
+    // pots straight from the loaded snapshot, then exit — used to confirm
+    // whether `bprev_blocks_by_pool` (eta numerator) is empty at the boundary.
+    if std::env::var("DUGITE_DUMP_LEDGER_PROBE").is_ok() {
+        let e = &ledger.epochs;
+        let bprev_sum: u64 = e.snapshots.bprev_blocks_by_pool.values().sum();
+        eprintln!("[probe] epoch={}", ledger.epoch.0);
+        eprintln!("[probe] treasury={}", e.treasury.0);
+        eprintln!("[probe] reserves={}", e.reserves.0);
+        eprintln!(
+            "[probe] bprev_block_count={} bprev_blocks_by_pool.len={} bprev_sum={}",
+            e.snapshots.bprev_block_count,
+            e.snapshots.bprev_blocks_by_pool.len(),
+            bprev_sum
+        );
+        eprintln!("[probe] ss_fee={}", e.snapshots.ss_fee.0);
+        match e.snapshots.go.as_ref() {
+            Some(g) => eprintln!(
+                "[probe] go: epoch={} pools={} stake_creds={} go_block_count={} go_blocks_by_pool.len={}",
+                g.epoch.0,
+                g.pool_stake.len(),
+                g.stake_distribution.len(),
+                g.epoch_block_count,
+                g.epoch_blocks_by_pool.len()
+            ),
+            None => eprintln!("[probe] go: NONE"),
+        }
+        eprintln!(
+            "[probe] set_present={} mark_present={}",
+            e.snapshots.set.is_some(),
+            e.snapshots.mark.is_some()
+        );
+        eprintln!(
+            "[probe] prev_d={}/{} prev_pv_major={}",
+            e.prev_d.numerator, e.prev_d.denominator, e.prev_protocol_version_major
+        );
+        eprintln!(
+            "[probe] reward_accounts={} rupd_addrs_rew_present={}",
+            ledger.certs.reward_accounts.len(),
+            e.rupd_addrs_rew.is_some()
+        );
+        std::process::exit(0);
+    }
+
     // Note: We deliberately do NOT restore the LSM UTxO store from disk.
     // The in-memory UtxoStore populated from the snapshot is sufficient for
     // measuring the apply-path bottleneck (CBOR decode + validation + UTxO ops).

crates/dugite-node/src/metrics.rs

@@ -1452,6 +1452,22 @@ impl NodeMetrics {
             .store(u64::from(peer_sharing), Ordering::Relaxed);
     }
 
+    /// Refresh just the pots gauges (`dugite_treasury_lovelace` /
+    /// `dugite_reserves_lovelace`) from the live ledger state.
+    ///
+    /// Two O(1) atomic stores — cheap enough to run on the per-block apply
+    /// path (`post_block_apply_updates`) so the gauges stay live at tip, not
+    /// only during bulk-sync catch-up.  Without this, the at-tip apply path
+    /// (which calls `publish_ledger_view`, not `set_governance_snapshot`)
+    /// left the pots frozen at the last catch-up value, reading one epoch
+    /// stale across every boundary the node did not forge itself.
+    pub fn set_pots(&self, treasury_lovelace: u64, reserves_lovelace: u64) {
+        self.treasury_lovelace
+            .store(treasury_lovelace, Ordering::Relaxed);
+        self.reserves_lovelace
+            .store(reserves_lovelace, Ordering::Relaxed);
+    }
+
     /// Update every governance-related gauge from a flattened snapshot.
     ///
     /// Called from the node's startup init path (`run`) and the sync loop's
@@ -3375,6 +3391,39 @@ mod tests {
         assert!(output.contains("dugite_drep_active 7500\n"));
     }
 
+    // Regression: the per-block metric refresh (`post_block_apply_updates`)
+    // must keep the pots gauges live at tip, not only during bulk-sync
+    // catch-up.  Before the fix, the at-tip apply path called
+    // `publish_ledger_view` (which does NOT touch the pots atomics) so
+    // `dugite_treasury_lovelace` / `dugite_reserves_lovelace` froze at the
+    // last catch-up value and never reflected an epoch-boundary reserves→
+    // treasury transfer — making the gauges read one epoch stale at tip and
+    // nearly triggering a false reward-divergence investigation on preview
+    // epoch 1332→1333.  `set_pots` is the O(1) seam the per-block path uses.
+    #[test]
+    fn test_set_pots_updates_gauges() {
+        let metrics = NodeMetrics::new();
+        assert_eq!(metrics.treasury_lovelace.load(Ordering::Relaxed), 0);
+        assert_eq!(metrics.reserves_lovelace.load(Ordering::Relaxed), 0);
+
+        // Preview epoch-1333 pots (byte-exact vs Koios) — the values the
+        // gauge must show once the boundary block is applied at tip.
+        metrics.set_pots(6_820_324_388_335_672, 7_916_699_786_696_095);
+
+        assert_eq!(
+            metrics.treasury_lovelace.load(Ordering::Relaxed),
+            6_820_324_388_335_672
+        );
+        assert_eq!(
+            metrics.reserves_lovelace.load(Ordering::Relaxed),
+            7_916_699_786_696_095
+        );
+
+        let output = metrics.to_prometheus();
+        assert!(output.contains("dugite_treasury_lovelace 6820324388335672\n"));
+        assert!(output.contains("dugite_reserves_lovelace 7916699786696095\n"));
+    }
+
     // D5: dugite_drep_count HELP text must document the active+inactive
     // semantics and point to Koios for DReps-with-delegated-stake.
     #[test]

crates/dugite-node/src/node/mod.rs

@@ -6998,16 +6998,24 @@ impl Node {
                 .slot_to_wallclock_ms(block_slot.0, &view.slot_config)
                 .await;
             self.metrics.set_tip_slot_time_ms(slot_time_ms);
-            let (live_epoch, pv_major, pv_minor) = {
+            let (live_epoch, pv_major, pv_minor, treasury, reserves) = {
                 let ls = self.ledger_state.read().await;
                 (
                     ls.epoch.0,
                     ls.epochs.protocol_params.protocol_version_major,
                     ls.epochs.protocol_params.protocol_version_minor,
+                    ls.epochs.treasury.0,
+                    ls.epochs.reserves.0,
                 )
             };
             self.metrics.set_epoch(live_epoch);
             self.metrics.set_protocol_version(pv_major, pv_minor);
+            // Keep the pots gauges as live as the epoch gauge.  This per-block
+            // path runs at tip too (unlike the catch-up-only inline store in
+            // `apply_fetched_block`), so the reserves→treasury transfer applied
+            // at an epoch boundary is reflected immediately even when the node
+            // did not forge the boundary block.  Cheap (two atomic stores).
+            self.metrics.set_pots(treasury, reserves);
         }
         // Authoritative era from the applied block's HFC era tag — NOT from the
         // ledger protocol-version major (which is Shelley-shaped and reads 2