Show & Tell: Online Fisher estimation via EMA — math, trade-offs, and surrogate gradients (Phase 6.4)

[web3guru888]

What Is Online Fisher Estimation?

Standard EWC computes the Fisher information matrix once per task, using a full pass over the episode buffer at SLEEP_PHASE time. This works, but:

• It creates staleness between SLEEP cycles — parameters can shift significantly before importance weights are refreshed.
• For agents with long episode buffers, the full-pass computation is O(B · P) where B = buffer size and P = parameter count.

Phase 6.4 introduces incremental Fisher estimation via exponential moving average (EMA):

F_i(t) = α · g_i(t)² + (1 - α) · F_i(t-1)

Symbol	Meaning
F_i(t)	Fisher importance of parameter i at step t
g_i(t)	Gradient (or surrogate) at step t
α	Decay rate — freshness vs. stability
1 - α	Memory factor — how much we trust history

At α = 1.0 this degenerates to a last-step estimate. At α → 0 it approaches a uniform average over all history. In practice α = 0.01 to 0.05 gives good results.

---

EMA vs. Full-Batch: Comparison

Property	Full-batch (Phase 6.1)	EMA Online (Phase 6.4)
Computation cost	O(B · P) per SLEEP	O(P) per step
Staleness	Up to one full SLEEP cycle	Decays within ~1/α steps
Memory	Must retain episode buffer	Only current EMA state
Accuracy	High (over large buffer)	Approximate, recency-biased
Use case	Infrequent task boundaries	Continuous adaptation

Both are complementary: the EMA accumulator provides fast, approximate importance tracking during a task; the SLEEP_PHASE full-batch pass provides accurate, stable snapshots at task boundaries.

---

Surrogate Gradients in Spiking Networks

True log-likelihood gradients require differentiating the network output with respect to parameters — this is straightforward in gradient-descent networks but undefined for discrete spikes.

For STDPOnlineLearner we use the weight delta Δw as a surrogate gradient:

g_i(t) ≈ Δw_i(t)  =  w_i(t) − w_i(t−1)

• Large |Δw| → parameter is actively changing → high Fisher importance.
• Small |Δw| → parameter is stable → lower importance.

This is closely related to the Synaptic Intelligence (SI) approach (Zenke et al. 2017), which computes per-synapse importance as the running sum of Δw · g. For STDP rules, Δw² is a recognised proxy for Fisher.

---

FisherAccumulator State Diagram

                 ┌──────────────────────────────┐
                 │         FisherAccumulator    │
                 │                              │
STDP update ──►──┤ update(gradients)            │
                 │   EMA step per param         │
                 │   steps += 1                 │
                 │                              │
                 │   if steps % N == 0:         ├──►── store.save(snapshot)
                 │     _write_snapshot()        │
                 │                              │
SLEEP_PHASE ──►──┤ force_snapshot()             ├──►── store.save(snapshot)
                 │   bypass step counter        │
                 │   reset step counter         │
                 └──────────────────────────────┘

---

Alpha Sensitivity Analysis

α	Half-life (steps)	Tracks fast changes	Stable under noise
0.001	~693	No	Very stable
0.01	~69	Moderate	Stable
0.05	~14	Yes	Moderate
0.1	~7	Yes	Noisy
1.0	1	Yes	Very noisy

Half-life formula: t_{1/2} = ln(2) / α

For a 10ms timestep agent tick, α = 0.01 gives a ~690-step (~7s) half-life, a reasonable working memory window for STDP learning.

---

Expected Prometheus Metrics

Metric	Type	Labels	Description
ewc_fisher_ema_steps_total	Counter	task_id	Total EMA update steps
ewc_fisher_snapshot_age_steps	Gauge	task_id	Steps since last snapshot
ewc_fisher_max_delta	Gauge	task_id	Max
ewc_fisher_staleness_warnings_total	Counter	task_id	Staleness threshold breaches

---

Open Questions

1. Gradient accumulation granularity: should the EMA step happen once per episode or once per spike event? Per episode is safer but coarser.
2. Mixed-precision: Fisher EMA states are float32; should we also support bfloat16 for large weight tensors?
3. Cross-agent Fisher sharing: if two agents share a CachedFisherStore, do their EMA steps interfere? Should task_id include an agent UUID?

See issue #252 for the full specification. Thoughts welcome below! 👇