Question: Multi-stage RL worth trying?

[fangyuan-ksgk]

DeepScaler demonstrates that incrementally increasing context length (8K -> 16K -> 24K) during GPT training allows even small 1.5B parameter LLMs to outperform O1 models on mathematical tasks while significantly reduce the FLOPs compared to full-context RL. Their multi-stage reinforcement learning training approach might be worth experimenting with for our project?

https://github.com/agentica-project/deepscaler

[ftgreat]

Current Progress on RL for Math Tasks
We are currently experimenting with reinforcement learning (RL) on mathematical tasks. The baseline model's math capabilities have been improved through an annealing phase. Subsequent stages included supervised fine-tuning (SFT) and RL fine-tuning.

Next Steps for Multi-Stage RL
Planning is underway for further RL phases involving:

• Sequence length adaptation
• Progressive problem difficulty scaling
• Rollout strategy optimization