跳到正文
SemiAnalysis· @SemiAnalysis_ · X·· 21 天前AI 评分48
AI 导读

被训练器与推理不匹配坑过的 RL 从业者都对此避之不及。 但如果你拥抱这种不匹配、与火共舞会怎样? 看看在 RL 训练中为 rollout 使用不同硬件和精度会发生什么: https://x.com/modal/status/2110074623337677046

AI 生成摘要 · 以原文为准

正文

RL practitioners who got burnt by trainer inference mismatch avoid it like a plague
What if you embrace the mismatch and dance with the fire?
Check out what happens when you use different hardware and precision for rollouts in RL training:
https://x.com/modal/status/2110074623337677046

引用Modal@modal
Training large models with RL typically runs on a fixed homogeneous cluster of the latest accelerators for numerical stability and faster throughput. This throws away inference optimization work done for other vendors and generations, limits the capacity you can utilize, and is unnecessarily expensive. So we trained a model on agentic coding tasks using fully asynchronous RL and mixed-precision, mixed-accelerator samplers. With off-policy correction, training stayed stable through convergence, on par with homogeneous samplers, at a lower cost.
在 X 查看被引用的帖子

来源:SemiAnalysis · x.com