Phase 42.4 Q&A — CheckpointManager: Snapshots, WAL & Recovery

[web3guru888]

CheckpointManager — Questions & Answers

Ask questions about distributed checkpointing, recovery, and the Chandy-Lamport algorithm.

Common Questions

1. How does the Chandy-Lamport algorithm work?

   • Initiator saves local state and sends markers on all outgoing channels
   • When a node receives a marker on channel C: if first marker, save state and send markers on all channels; record messages on C between state save and marker as channel state
   • Result: consistent global snapshot (no orphan messages)

2. What's the optimal checkpoint interval?

   • Too frequent: high overhead from I/O and synchronization
   • Too infrequent: more work lost on failure
   • Optimal: balance checkpoint cost vs expected recovery cost
   • Plank et al. formula: T_opt ≈ √(2 × C × M) where C=checkpoint time, M=MTBF

3. How does incremental checkpointing work?

   • Hash each parameter tensor; only save tensors that changed
   • Maintain chain: full → delta → delta → ...
   • Periodically create new full checkpoint to limit chain length
   • Typically 60-80% size reduction for fine-tuning workloads

4. When should I use activation checkpointing?

   • When GPU memory is the bottleneck for batch size or model size
   • Trades ~33% more compute for ~50% less activation memory
   • Selective: only checkpoint expensive layers (attention, FFN)

Reference issue: #844 | Planning: #840