Show & Tell 57.1: ExpertRouter — Token-to-Expert Routing & Gating Networks

[web3guru888]

ExpertRouter Implementation Showcase

Share your implementations of expert routing mechanisms:

Topics

• Noisy Top-K Gating: Shazeer et al.'s original gating with learned noise for exploration
• Switch Routing: Fedus et al.'s simplified top-1 routing with capacity factors
• Expert Choice Routing: Zhou et al.'s inverted routing where experts select tokens
• Hash Routing: Roller et al.'s fixed hash-based assignment as alternative to learned gates
• BASE Layer Routing: Lewis et al.'s linear assignment for perfectly balanced routing
• Routing Visualization: Tools for understanding token-to-expert assignment patterns
• Capacity Factor Tuning: Strategies for setting and adapting capacity constraints

Key Metrics

• Routing latency overhead (<0.5ms target for 128 experts)
• Load balance coefficient of variation
• Expert utilization distribution
• Routing entropy

Share code snippets, benchmarks, routing visualizations, and lessons learned!

Related: #1082