Show & Tell 57.3: LoadBalancer — Auxiliary Losses & Expert Capacity Management

[web3guru888]

LoadBalancer Implementation Showcase

Share your implementations of MoE load balancing:

Topics

• Auxiliary Balancing Loss: Switch balancing loss (Fedus et al.) for even expert utilization
• Router Z-Loss: ST-MoE stability loss (Zoph et al.) to prevent large router logits
• Importance Loss: Shazeer et al.'s CV-based importance balancing
• Sinkhorn-Knopp: BASE Layers doubly-stochastic assignment for perfect balance
• Dead Expert Revival: Detecting and recovering underutilized experts
• Routing Collapse Prevention: Strategies to avoid mode collapse in routing
• Adaptive Capacity: Dynamic capacity factor adjustment during training

Key Metrics

• Expert utilization coefficient of variation
• Token drop rate
• Dead expert count
• Balance score

Share loss implementations, monitoring dashboards, and balancing strategies!

Related: #1084