TinyZero: Reproducing DeepSeek R1-Zero's Reasoning with RL for Under $30

Fri, 01 May 2026 00:00:00 +0000

DeepSeek R1-Zero was widely regarded as a breakthrough when it was released in January 2025. The model demonstrated that pure reinforcement learning — without any supervised fine-tuning on human reasoning examples — could produce advanced chain-of-thought reasoning, self-correction, and even surprising “aha moments” where the model independently discovered better reasoning strategies mid-conversation. The catch? The training infrastructure was assumed to require massive compute clusters and budgets in the tens of millions of dollars.

Jiayi Pan’s TinyZero shatters that assumption entirely.

TinyZero is an open-source, minimal reproduction of the DeepSeek R1-Zero methodology that runs on a single GPU for under $30 in cloud compute costs. Using the veRL framework — a versatile reinforcement learning library for language models — TinyZero applies PPO (Proximal Policy Optimization) to small base models like Qwen-2.5-1.5B-Instruct and Qwen-2.5-7B. The training task is deceptively simple: given four numbers, the model must combine them using arithmetic operations (+, -, *, /) to reach a target value. Yet from this humble starting point, the same emergent reasoning behaviors that made DeepSeek R1-Zero famous begin to appear.

X-R1: Open-Source Reasoning Model Exploration

Fri, 01 May 2026 00:00:00 +0000

The revelation that language models could develop sophisticated reasoning capabilities through reinforcement learning – without human demonstrations – was one of the most surprising results in AI research of 2024 and 2025. DeepSeek R1 showed that models trained with RL could learn to think step by step, producing chain-of-thought reasoning that dramatically improved performance on mathematical, logical, and coding tasks. X-R1 is an open-source project that explores these techniques, aiming to reproduce, understand, and extend the reasoning-through-RL paradigm.

Developed by researcher dhcode-cpp, X-R1 implements the key techniques from the DeepSeek R1 and related papers, making them accessible for experimentation with open-source models. The project provides training scripts, reward function implementations, and evaluation pipelines that researchers can use to investigate how RL shapes reasoning behavior in language models.

DeepSeek R1 on SoloSoft

TinyZero: Reproducing DeepSeek R1-Zero's Reasoning with RL for Under $30

X-R1: Open-Source Reasoning Model Exploration