Q&A: Phase 33.2 ProgressiveNetExpander — Design Questions

[web3guru888]

Q&A: Phase 33.2 — ProgressiveNetExpander Design Questions

This thread is for technical questions about the ProgressiveNetExpander component.

Open design questions

1. Scalability ceiling: With O(T²) lateral connections for T tasks, how do we prevent quadratic growth? Should we limit lateral connections to the K most relevant columns?

2. Column architecture selection: Should new columns match the base architecture, or should we use NAS to find task-specific optimal architectures? What is the search cost budget?

3. Similarity threshold tuning: When using similarity-based reuse, how do we set the threshold? Too high → unnecessary expansion; too low → negative transfer from reusing dissimilar columns.

4. Pruning timing: Should columns be pruned immediately after training, or after consolidation once importance is clearer? How does pruning interact with lateral connections?

5. Hybrid with EWC: Can we combine progressive expansion (for structurally different tasks) with EWC (for similar tasks) in the same pipeline? What triggers the switch?

References

• Issue: Phase 33.2 — ProgressiveNetExpander: Dynamic capacity allocation via lateral connections #692
• Planning: Phase 33 Planning — Continual Learning & Catastrophic Forgetting Prevention #690