Show & Tell 48.2: ProgressiveNetworkExpander — Column Architecture & Dynamic Growth

[web3guru888]

ProgressiveNetworkExpander — Implementation Deep-Dive

Progressive Neural Networks Architecture

Progressive nets solve forgetting by never modifying learned parameters. Each new task gets a fresh column with lateral connections to all previous columns:

Task 1:  [Col1] ──────────────────────> Output1
Task 2:  [Col1]frozen ──lateral──> [Col2] ──> Output2
Task 3:  [Col1]frozen ──lateral──> [Col3] ──> Output3
         [Col2]frozen ──lateral──┘

Lateral Connection Design

Lateral adapters enable forward transfer without backward interference:

h_new = σ(W_new · x + Σ_k U_k · h_k_prev)

where U_k are learnable lateral weights from column k to the new column.

PackNet: Single Network, Multiple Tasks

PackNet takes a different approach — one network, but different weight subsets per task:

1. Train on task T, then prune smallest 75% of weights
2. Assign surviving weights to task T (freeze them)
3. Freed weights become available for task T+1

This gives guaranteed zero forgetting with constant model size.

Dynamic Expandable Networks

DEN adaptively decides between:

• Selective retraining: Reuse existing neurons (L1 sparse selection)
• Network expansion: Add new neurons when reuse is insufficient
• Neuron splitting: Split neurons with high drift into task-specific copies

Capacity Growth Analysis

Method	Memory Growth	Forgetting	Forward Transfer
Progressive Nets	O(T) columns	Zero	Via laterals
PackNet	Constant	Zero	Via shared features
DEN	Sublinear	Near-zero	Via selective reuse

Share your implementations and design choices below!

Related: Issue #939 | Planning: #937