TinyZero: Reproducing DeepSeek R1-Zero's Reasoning with RL for Under $30

Fri, 01 May 2026 00:00:00 +0000

DeepSeek R1-Zero was widely regarded as a breakthrough when it was released in January 2025. The model demonstrated that pure reinforcement learning — without any supervised fine-tuning on human reasoning examples — could produce advanced chain-of-thought reasoning, self-correction, and even surprising “aha moments” where the model independently discovered better reasoning strategies mid-conversation. The catch? The training infrastructure was assumed to require massive compute clusters and budgets in the tens of millions of dollars.

Jiayi Pan’s TinyZero shatters that assumption entirely.

TinyZero is an open-source, minimal reproduction of the DeepSeek R1-Zero methodology that runs on a single GPU for under $30 in cloud compute costs. Using the veRL framework — a versatile reinforcement learning library for language models — TinyZero applies PPO (Proximal Policy Optimization) to small base models like Qwen-2.5-1.5B-Instruct and Qwen-2.5-7B. The training task is deceptively simple: given four numbers, the model must combine them using arithmetic operations (+, -, *, /) to reach a target value. Yet from this humble starting point, the same emergent reasoning behaviors that made DeepSeek R1-Zero famous begin to appear.

VeRL: ByteDance's Reinforcement Learning Framework for LLMs

Fri, 01 May 2026 00:00:00 +0000

The most exciting frontier in large language model research in 2025-2026 has not been about making models bigger. It has been about making them smarter through reinforcement learning. DeepSeek-R1 demonstrated that RL training – specifically GRPO (Group Relative Policy Optimization) – can dramatically improve a model’s reasoning capabilities, enabling chain-of-thought reasoning, self-correction, and structured problem solving that rivals much larger models. ByteDance, one of the world’s largest technology companies and the creator of TikTok and Douyin, has been applying these same techniques at scale to train its own models. VeRL is the framework behind that effort.

VeRL (Voltron Reinforcement Learning) is ByteDance’s open-source reinforcement learning framework designed specifically for LLM training. It implements state-of-the-art RL algorithms including PPO (Proximal Policy Optimization) and GRPO, integrates tightly with vLLM for efficient inference during training, and supports distributed training across hundreds of GPUs. VeRL is the production framework that powers ByteDance’s internal LLM development, including the Doubao (豆包) AI assistant.

VeRL on SoloSoft

TinyZero: Reproducing DeepSeek R1-Zero's Reasoning with RL for Under $30

VeRL: ByteDance's Reinforcement Learning Framework for LLMs