Q&A: Phase 33.1 ElasticWeightConsolidator — Design Questions

[web3guru888]

Q&A: Phase 33.1 — ElasticWeightConsolidator Design Questions

This thread is for technical questions about the ElasticWeightConsolidator component.

Open design questions

1. Diagonal vs. K-FAC Fisher: Diagonal Fisher is O(p) memory but loses parameter correlations. K-FAC captures second-order structure at O(p·k) cost. When is the accuracy improvement worth the overhead?

2. Online vs. offline Fisher: Online EWC (exponential moving average) is memory-efficient but may over-weight recent tasks. What decay rate γ best balances recency and retention?

3. Lambda scheduling: Should λ_ewc be constant across tasks, or should it increase as more tasks are consolidated? What schedule prevents under-regularization early and over-regularization late?

4. Interaction with replay: When combined with experience replay (Phase 33.3), does EWC provide additive benefit, or is there redundancy? Should λ be reduced when replay is active?

5. Scalability to Transformers: Fisher computation requires per-sample gradients. For models >1B parameters, what approximations (e.g., layer-wise Fisher, Fisher subspace) maintain effectiveness?

References

• Issue: Phase 33.1 — ElasticWeightConsolidator: Fisher Information-based parameter protection for continual learning #691
• Planning: Phase 33 Planning — Continual Learning & Catastrophic Forgetting Prevention #690