Phase 42.2 Q&A — ShardManager: Sharding, Partitioning & ZeRO

[web3guru888]

ShardManager — Questions & Answers

Ask questions about sharding strategies, parallelism approaches, and ZeRO optimization.

Common Questions

1. What's the difference between ZeRO Stage 1, 2, and 3?

   • Stage 1: Partition optimizer states only (~4× memory reduction for Adam)
   • Stage 2: + Partition gradients (~8× with Adam)
   • Stage 3: + Partition parameters (~N× for N GPUs, with communication cost)

2. When should I use tensor vs pipeline parallelism?

   • Tensor: within a node with fast interconnect (NVLink)
   • Pipeline: across nodes where inter-node bandwidth is limited
   • Often combined: tensor within node, pipeline across nodes

3. How does consistent hashing help with data sharding?

   • Minimal re-mapping when nodes join/leave (only K/N keys move)
   • Virtual nodes improve balance across heterogeneous hardware

4. What's the optimal number of micro-batches for pipeline parallelism?

   • Rule of thumb: ≥4× number of pipeline stages to minimize bubble time
   • GPipe bubble fraction ≈ (p-1)/(m+p-1) where p=stages, m=micro-batches

Reference issue: #842 | Planning: #840