MiniMind Audited: 63,912,192 Parameters, a 52,762-Row 'LoRA Sample' That Is 99.25% of the CEval Test Split, and Four Definitions of 'Two Hours'

I recounted MiniMind's parameters (63,912,192) and fingerprinted its LoRA file: the 64M claim holds, but '2 hours' has four definitions and the tests are gone.

MiniMind has 67,682 stars and a one-line pitch that fits in a screenshot: train a 64M-parameter language model from scratch, on one RTX 3090, in two hours, for about three yuan. The repo is Apache-2.0, the data is public on Hugging Face, and the training scripts are 4,695 lines of plain PyTorch. It is the rare project where almost every claim can be checked from a filesystem — so I checked them.

Three claims survived contact with the files. Several did not, and the interesting part is that the failures are not the usual ones. The parameter count is exact to the digit. The compute budget, once you convert it to FLOPs, is plausible — it implies a 3090 running at 44% of its bf16 tensor peak, which is a real number a real person could hit. What doesn’t hold together is the bookkeeping around that number, and the description of the dataset used to produce the project’s most flattering benchmark row.

Everything below was measured on 2026-11-21 UTC against commit cc9629e (2026-11-18, [update] fix decoding, drop tests), the tokenizer shipped in model/tokenizer.json, and the dataset files served by jingyaogong/minimind_dataset on Hugging Face. Where I could not verify something, I say so instead of guessing.

The one number that is exactly right

Config is short: hidden_size=768, num_hidden_layers=8, num_attention_heads=8, num_key_value_heads=4, vocab_size=6400, tie_word_embeddings=True, and an intermediate size computed as ceil(hidden * π / 64) * 64 = 2,432. From that you can write down every tensor:

ComponentShapeParameters
q_proj768 × 768589,824
k_proj768 × 384294,912
v_proj768 × 384294,912
o_proj768 × 768589,824
q_norm + k_norm (per head, 96 each)96 + 96192
gate_proj, up_proj, down_proj3 × (768 × 2,432)5,603,328
2 × RMSNorm768 each1,536
Per block7,374,528
8 blocks58,996,224
Token embedding (tied to lm_head)6,400 × 7684,915,200
Final RMSNorm768768
Total63,912,192

63.91 million. The README says “about 64M”, the model card says “64M”, and both are correct. The “1/2700 the size of GPT-3” line is also correct: 175,000 / 63.91 = 2,738.

The MoE variant checks out too, which matters more than it sounds. Four experts of 2,432-wide SwiGLU per layer plus a 768×4 router gives 193,500,672 parameters in the trunk plus the same 4,915,968 in embeddings = 198.4M total, 63.9M active. The README claims “198M-A64M”. That is the correct notation for this architecture, and it is the notation most MoE marketing gets wrong.

So the model is what it says it is. Hold that thought, because every later discrepancy is about scope — which hours, which tokens, which data — not about the model secretly being smaller than advertised.

Four different definitions of “two hours”

This is where the README stops agreeing with itself. There are four places that define the headline number, and they do not describe the same thing:

LocationWhat “2 hours” is said to coverImplied cost
Project intro“train MiniMind … entirely from scratch with only about RMB 3 in cost and 2 hours of training time”the whole pipeline
English footnote, line 42“‘2 hours’ refers to the measured time for running 1 epoch of the SFT stage on a single NVIDIA 3090”SFT only, 1 epoch
Training Cost tablepretrain_t2t_mini ≈1.21h + sft_t2t_mini ≈1.10h, “1 epoch … approximately 2.31 hours, cost approximately 3.0 CNY”pretrain + SFT, 1 epoch each
Shipped script defaultstrain_pretrain.py --epochs 2, train_full_sft.py --epochs 24.62 hours for the same mini pipeline

Read the footnote against the table directly: the footnote says two hours is one SFT epoch; the table says one SFT epoch is 1.10 hours, and that 1.21 + 1.10 = 2.31 hours is pretrain plus SFT. Both sentences are in the same file. Neither is a typo — the footnote appears to have been written to defuse the “from scratch” objection (SFT is not from scratch) and in doing so assigned the SFT stage a duration the project’s own measurements contradict.

The yuan figure is the one that actually reconciles. At the stated rental price of 1.3 CNY/hour, 2.31 hours costs 3.003 CNY — exactly “about RMB 3”. So the money line was computed from the 1-epoch, two-stage sum, and then the hours were rounded down to “roughly 2” one paragraph later and re-scoped to “SFT only” in the footnote.

Then the scripts apply 2 epochs to both stages by default, which doubles it to 4.62 hours and about 6 CNY. That is still cheap. It is also 2.3× the headline, and no part of the README says the headline assumes a single epoch only if you pass --epochs 1 yourself.

There is a fifth, historical definition too: the 2025-02-09 changelog entry “Streamlined pretraining dataset … reproducible in 2 hours on a single 3090!” belongs to the minimind2 release, whose models were 26M and 104M. The minimind-3 rewrite (2026-04-01) changed the architecture, the tokenizer, and the default datasets, and kept the sentence.

One more wording detail worth noticing: the intro footnote calls the number “measured”, while the Training Cost section calls the same numbers “empirical estimates” and adds “the above values are estimates and are meant only as a rough reference”. A measured number and an estimated number are different objects; a reader deciding whether to rent a GPU is entitled to know which one they are being quoted.

Measuring the tokens instead of trusting the table

To find out whether 1.21 hours for pretraining is physically reasonable, you need the token count. The README gives file sizes, not tokens. So I pulled the first 12 MiB of each mini file over an HTTP range request, encoded it with the project’s own tokenizer, dropped the trailing partial line, and extrapolated by bytes:

FileSize on diskBytes/token (measured)Est. tokens per epoch
pretrain_t2t_mini.jsonl1,241,043,656 (1.24 GB)3.49~355.9M
sft_t2t_mini.jsonl1,739,201,170 (1.74 GB)3.73~466.6M
lora_exam.jsonl24,650,716 (23.5 MiB)—52,762 rows

Two things fall out immediately. First, the SFT file is larger than the pretraining file — 1.4× the bytes, 1.31× the tokens. The “Zero” model is not pretrained on one corpus and lightly tuned on another; it spends more tokens on instruction data than on pretraining. Second, the ratio between compute and parameters is tiny: 355.9M tokens against 63.9M parameters is 5.6 tokens per parameter for one pretraining epoch, and 11.1 at the shipped default of two.

Chinchilla’s compute-optimal point is 20 tokens per parameter. At the default settings MiniMind trains to 56% of that budget. This is a defensible choice for a teaching project — you are not trying to win a benchmark, you are trying to finish before lunch — but it is the single most important fact for interpreting the benchmark table later in this article, and the README never states it as a number.

Now the FLOPs. Training compute is approximately 6ND:

6 × 63,912,192 × 355,901,392  = 1.365e17 FLOPs   (pretrain, 1 epoch, 1.21 h)
6 × 63,912,192 × 466,627,176  = 1.789e17 FLOPs   (SFT, 1 epoch, 1.10 h)

1.365e17 / (1.21 × 3600) = 31.3 TFLOPS  →  44.1% of a 3090's 71 TFLOPS bf16 peak
1.789e17 / (1.10 × 3600) = 45.2 TFLOPS  →  63.6% of peak

A 63.6% model-FLOPs utilisation on a 3090 for 768-token sequences is aggressive but not absurd; 44.1% for 340-token sequences is comfortably in the range people report for small transformer training with SDPA and bf16 autocast. The compute claim is credible. If the project had claimed twenty minutes, the arithmetic would have exposed it. It didn’t, and that deserves to be said before the criticisms: the numbers here are the product of someone who actually ran the job.

The problem is that the same table is only reachable at a context length the docs advise against.

The sequence-length contradiction inside the same repo

The dataset guide lists recommended settings, and for pretrain_t2t_mini it says max_seq_len≈768. The string is right there in the README’s own dataset table, next to the file’s size.

train_pretrain.py ships with --max_seq_len default 340.

Those two numbers do not coexist peacefully. If you follow the documented recommendation instead of the undocumented default, each sample contributes 768 tokens instead of 340 — 2.26× the tokens per epoch, so 2.26× the FLOPs, so about 2.73 hours for one pretraining epoch rather than 1.21. At the documented recommendation, the two-stage, one-epoch pipeline lands near 3.8 hours, and the default two-epoch run near 7.7.

Which one produced the table? The table doesn’t say, and neither does the changelog. But the arithmetic points one way: 1.21 hours is consistent with 340-token sequences and not with 768. So the headline cost was measured — or estimated — at a shorter context than the same README recommends for that exact file, which means the fastest reproduction path also throws away the tails of the longest documents. Same table, two rows apart.

The tokenizer conversion is off by up to 74% on literary Chinese

The docs offer a rule of thumb: “Chinese 1 token ≈ 1.5–1.7 characters” (the script help text repeats it). I ran four passages through the shipped tokenizer:

TextCharactersTokensChars/token
Modern news/announcement prose69561.23
Classical-flavoured narrative (Journey to the West style)57660.86
Technical/ML exposition67411.63
English139353.97
Whole 12 MiB pretrain sample (73% CJK)4,796,3863,608,4741.33

The 6,400-token vocabulary is the cause: with only 6,400 BPE merges, uncommon characters and literary compounds fall through to byte-level fallbacks, and you get more tokens than characters (0.86). A 57-character line becomes 66 tokens.

This matters because the same conversion rule is what a reader would use to estimate whether their own corpus fits in an epoch. Using 1.5 chars/token on that literary passage predicts 38 tokens where the tokenizer produces 66 — a 74% underestimate of compute. The rule is roughly right for modern technical Chinese (1.63) and badly wrong for exactly the style of text the README uses in its own RoPE perplexity demo. It is a small thing, but it is the difference between a one-hour and a two-hour run.

The benchmarks are at chance, and the project says so

The objective-evaluation table reports seven benchmarks. Five are four-option multiple choice (25% chance), PIQA is two-option (50%), Social-IQa is three-option (33.3%). Here is the same table with the chance line drawn in:

Modelceval (25%)cmmlu (25%)arc (25%)obqa (25%)hellaswag (25%)piqa (50%)siqa (33.3%)
minimind-3, 64M24.8925.3828.4923.6028.2850.6534.19
minimind-3-moe, 198M25.4824.3227.7426.2027.4350.7134.03
minimind-3-exam, 64M30.9826.1235.6124.2028.4056.2634.19

OpenBookQA at 23.60 is below four-way guessing, which happens: with 500 questions and a 1.9-point standard error, one point under chance is noise. That is precisely why the README’s resolution table — which lists sample counts and ± stderr per dataset, and warns that a gap smaller than ~5 points on obqa or ~3 on ceval “does not separate two models” — is the most respectable thing in the document. It pre-empts the criticism a careful reader would otherwise make.

It also means the honest way to read the table is: this model learned format and fluency, not knowledge, and 5.6 tokens per parameter is why. Nothing in the README disputes that; the sentence “the results are only for fun” is doing real work.

Which brings us to the one row where the numbers move a lot, and to the dataset behind it.

52,762 alignment rows: what is actually inside lora_exam.jsonl

The minimind-3-exam variant gains 6 points on C-Eval and 7 on ARC-Easy over the base model. The README explains why, in a disclosure paragraph that is more candid than most:

“It is simply minimind-3 after a lightweight LoRA alignment on lora_exam.jsonl … This alignment data is sampled from the test subsets of ceval and English mmlu, with additional prefix/suffix augmentation. Its purpose is to align the context and option format commonly seen in multiple-choice evaluation, rather than to teach the answers.”

And then, one paragraph later:

“The 7 benchmarks used in this section have no sample overlap with the alignment data above, so this result can be regarded as free of data contamination.”

Two claims: sampled from the test subsets, and no sample overlap with the evaluated benchmarks. The first is checkable by opening the file. The second is checkable by comparing it against the actual benchmark splits. Both are 24.6 MB away, so I downloaded the file and all seven benchmark splits, and compared.

Method, because the method is the finding. I built two fingerprints per benchmark item: (a) the question stem alone, and (b) the question stem concatenated with all four option texts, whitespace and full-width spaces normalised, option labels stripped. Then I asked whether each fingerprint appears verbatim anywhere in the 52,762 rows.

Benchmark splitItemsPresent by question stemPresent by question + options
CEval test12,34212,322 (99.84%)12,249 → 99.25% (after exact option matching)
MMLU test14,04214,026 (99.89%)13,649 → 97.20%
CEval validation (the evaluated split)1,34623 (1.71%)0 (0.00%)
ARC-Easy test2,37600
OpenBookQA test50000

Read those rows carefully, because they say two different things at once.

The contamination claim holds for what is actually evaluated. MiniMind’s harness command uses --tasks ceval-valid, which is the 1,346-item validation split, and zero of those items appear in the alignment file once you match on options. The 23 stem-only “hits” are all generic stems — 下列说法中,正确的是____, 下列各项中,应征收资源税的是____ — which recur across subjects with different options. This is the trap: a stem-only audit would have “found” a 1.71% contamination rate and published it. Adding the options collapses it to zero, because the options are where the information is. Anyone auditing a multiple-choice alignment set by question text alone will manufacture findings.

But “sampled” describes the complete test split. 99.25% of the CEval test set and 97.20% of the MMLU test set are in the file — 12,342 + 14,042 = 26,384 distinct items, rendered twice each through different prompt prefixes (52,768 expected rows; the file has 52,762, six short). The prefix counter shows the augmentation directly: 8,561 rows start with the bare question, 7,082 start with “The following is a multiple choice quest…”, 6,062 with 以下是一道单项选择题,请直接给出正确选项的字母。, and the remainder with assorted hand-written openers (考考你:, 判断以下哪个选项正确:, 来做道题吧:). Sampled 250 CEval items, each appeared in 1.98 rows on average.

So the honest summary is: the file is not a sample of the CEval and MMLU test sets, it is those test sets, with prompt augmentation. The word “lightweight” and the phrase “a small amount of format alignment” both describe a 24.6 MB corpus that is 1.85× the combined size of the two benchmarks’ test splits. And the reported average gain — “+2.9 percentage points across the 7 tasks” — comes from a model that has seen every item of one of those benchmarks’ other half.

The practical consequence is concrete and worth pinning to the wall: never run lm_eval --tasks ceval (the test split) on minimind-3-exam. You would be scoring a model on 12,342 questions it was fine-tuned on, and the README’s own note says what that looks like — it mentions the base model reaching “about 97% accuracy on contaminated ceval / cmmlu subsets … but such numbers are not meaningful.” The project knows the failure mode. It just doesn’t say which of its own published weights can no longer be honestly scored on which split.

What the +2.9 actually is, item by item:

Benchmarkminimind-3minimind-3-examΔ
C-Eval (val)24.8930.98+6.09
ARC-Easy28.4935.61+7.12
PIQA50.6556.26+5.61
CMMLU25.3826.12+0.74
OpenBookQA23.6024.20+0.60
HellaSwag28.2828.40+0.12
Social-IQa34.1934.190.00

Mean: 2.897, which is the “about 2.9” in the README — arithmetic checks out. The pattern is exactly what format alignment predicts: the two 4-option tasks where a plain-text completion was being scored worst (C-Eval, ARC) gain 6–7 points; the two- and three-option tasks (PIQA, Social-IQa) barely move or don’t move; HellaSwag, which has long contexts and four options, barely moves. That the gains land on held-out validation data makes the mechanism claim credible — this is what “the bottleneck was format, not knowledge” looks like when it’s true.

Repository health: what 67,682 stars do not tell you

MetricValueReading
Stars / watchers67,682 / 2990.44% — the classic bookmark ratio
Forks8,858 (13.1% of stars)high, consistent with “I want to run this myself”
Contributors27, of which maintainer has 231 commitsone-person project with a PR queue
Releases / tags2, both old: v2 (2025-10-21), minimind-v1 (2024-09-02)latest release is 396 days before this audit
minimind-3 (released 2026-04-01 per changelog)no tag, no releaseyou cannot pin the version the README describes
Open items88 (55 PRs, 33 issues, 50 with zero comments)a backlog, not a conversation
Tests / CInonesee below

The missing release is the one that would bite a team: the README describes minimind-3 and its mini datasets, but the last tag predates that model by five months. If you need “the code that produced these weights”, the only reference you have is a commit hash in a moving master.

The tests existed for less than a day

On 2026-11-18, five pull requests from contributor Afloat16 were merged, each pairing a fix with a regression test: synchronous inference off the API event loop, boolean padding masks in the manual attention path, temperature ignored under greedy decoding, LoRA deltas accumulated before fp32 rounding, empty-supervision distillation batches kept finite. Five test files landed in tests/.

The same day, the maintainer’s own commit cc9629e is titled:

[update] fix decoding, drop tests

It is +6/−163 across seven files. Three lines changed in model/model_minimind.py, three in scripts/serve_openai_api.py, and five files removed:

removed tests/test_api_inference_event_loop.py        -41
removed tests/test_attention_bool_mask.py             -29
removed tests/test_distillation_empty_supervision.py  -21
removed tests/test_generation_greedy_temperature.py   -34
removed tests/test_lora_merge_fp32.py                 -31

There is no .github/ directory, so there was never a CI gate for those tests to run behind. Today the repository has 22 Python files and zero tests.

This is not fraud; it’s a maintainer choosing, visibly and honestly, not to carry a test suite contributed as a side effect of five bug fixes. But it changes what the project is. A 67k-star repo that people use to learn how training pipelines are built is, at this commit, a codebase where none of the five behaviours those tests covered is protected against regression, and where the newest change to the core model file was made in the same commit that deleted its coverage.

The install step is broken, and the resolver cannot see why

requirements.txt pins numpy==1.26.4 and datasets==3.6.0, and does not mention pyarrow. On a fresh 2026-11 machine, datasets pulls the newest pyarrow, and the current newest is 26.0.1.

I ran the resolver rather than trusting the issue tracker:

pip install --dry-run --python-version 3.12 --only-binary=:all: "numpy==1.26.4" "datasets==3.6.0"
# → Would install … numpy-1.26.4 … pandas-2.3.3 … pyarrow-26.0.1 …

No error. The conflict is invisible to pip for a specific reason: pyarrow 26.0.1’s published metadata declares no numpy requirement at all (checked against the PyPI JSON API — requires_dist contains no numpy entry), while the incompatibility is enforced at import time. So resolution succeeds and import datasets fails afterwards with pyarrow requires NumPy 2.0 or newer, found 1.26.4.

Three separate people hit this within two and a half hours on 2026-11-20 — issues #947 (07:12 UTC), #948 (08:33), #949 (09:33) — all proposing the same one-line pin. As of this audit all three are open with zero comments.

The cost of that is not aesthetic. The pitch is “two hours to reproduce”; step one is a dependency install that fails on a clean environment, and the fix is sitting unmerged in the queue next to 54 other open PRs.

“All core algorithms implemented in native PyTorch, without high-level abstractions”

This line in the project intro is half exactly right and half worth narrowing.

Right: the training algorithms are genuinely hand-written. DPO, PPO, GRPO, CISPO, the rollout engine, distillation — the trainers are plain PyTorch loops with explicit loss arithmetic, get_lr schedules, and gradient clipping. For a teaching repo that is the part that matters, and it is real.

Narrowed: 11 of the 22 Python files import from transformers, and the core model file is built on them:

from transformers.activations import ACT2FN
from transformers import PreTrainedModel, GenerationMixin, PretrainedConfig
from transformers.modeling_outputs import MoeCausalLMOutputWithPast

MiniMindForCausalLM subclasses PreTrainedModel and mixes in GenerationMixin, so generate(), KV-cache plumbing, from_pretrained loading and the config serialisation all come from the library the intro says it avoids. The tokenizer is tokenizers-format tokenizer.json. That’s a reasonable engineering decision — nobody wants to reimplement generate() to teach attention — but it means the sentence “does not rely on high-level abstractions from third-party libraries” should be read as “the algorithms”, not “the model”.

What holds up, in detail

For fairness, the parts I tried to break and couldn’t:

  • The architecture is textbook-correct for Qwen3-style models. GQA with 8 query and 4 KV heads, per-head RMSNorm on q and k, RoPE with θ=1e6 and optional YaRN (factor 16, original window 2,048), tied embeddings, SwiGLU at 2,432. The manual attention path is only used when SDPA can’t be; the causal mask is added with triu(1) rather than triu(0), which is correct.
  • The MoE router has both the pieces people forget. Aux loss at 5e-4 with a load-balancing term, and for top-1 routing a straight-through weight (top1 - top1.detach() + 1.0) so the router gets gradient even though the gate value is exactly 1.0. The MoE block also has the y[0,0] += 0 * sum(params) trick to keep unused experts in the autograd graph.
  • Checkpointing is real. Resumable state includes optimizer and scaler, and SkipBatchSampler adjusts the step count when you resume on a different GPU count.
  • The serving path is more complete than the pitch requires. OpenAI-compatible API with reasoning_content, tool_calls, and an open_thinking switch; 52,762 rows of evaluation data would have been easier to fake than this.
  • The disclosure habit. The stderr table, the “only for fun” sentence, the exam-model paragraph admitting the alignment data comes from test subsets, and the honest admission that the agent weights degrade on factual questions (“more willing to fabricate on open-ended Q&A”) are all things most projects hide. Every criticism above exists in a document that told you where to look.

How to cost this yourself in five commands

If you are about to rent a 3090 for this, do these instead of trusting any table, mine included:

  1. Count parameters from config, not from the badge.
    per_block = 768*768 + 2*(768*384) + 768*768 + 2*96 + 3*(768*ceil(768*pi/64)*64) + 2*768
    total     = 8*per_block + 6400*768 + 768
    
  2. Measure bytes/token on your own file. Two lines with the project’s tokenizer on a 10 MB head sample beats any chars-per-token rule of thumb:
    tok = Tokenizer.from_file("model/tokenizer.json")
    ids = tok.encode(open("head.jsonl", encoding="utf-8").read(), add_special_tokens=False).ids
    tokens_est = file_bytes / (sample_bytes / len(ids))
    
  3. Convert to FLOPs and to a utilisation number. 6 * params * tokens / seconds, divided by 71 TFLOPS for a 3090 at bf16. Under 40% and the estimate is optimistic; over 65% and someone is measuring something else.
  4. Decide max_seq_len deliberately. The script default (340) and the dataset recommendation (768) differ by 2.26× in compute. Whichever you choose, write it down next to your cost estimate.
  5. Audit any alignment set with option-level fingerprints. Question stems collide across subjects; question + options does not. If you skip this step you will either miss real contamination or invent fake contamination — this audit did the second one first and had to throw it away.

FAQ

Is the “two hours” claim false? No — it is under-specified in four mutually inconsistent ways. One epoch of pretrain plus one epoch of SFT on the mini datasets is 2.31 hours at the implied utilisation, which is genuinely close to two hours. The shipped scripts run two epochs by default (4.62 hours), and the footnote’s claim that two hours is “1 epoch of SFT” contradicts the project’s own 1.10-hour figure for that stage.

Can I really train a 64M model from scratch on one 3090 in an afternoon? Yes, and that is the achievement worth crediting. At default settings the mini pipeline is about 4.6 hours of GPU time plus a ~3 GB download. Add the dependency pin from issue #947 first, or you will spend the first hour on ImportError.

Are the published benchmarks contaminated? Not the ones that are evaluated. The alignment file contains zero of the 1,346 CEval validation items and zero ARC-Easy or OpenBookQA items, matched on question plus options. It does contain 99.25% of the CEval test split and 97.20% of MMLU test — so the -exam weights must never be scored on those splits.

Why is OpenBookQA below random chance? 500 questions, four options, and a model with 5.6 tokens per parameter. The standard error is ±1.9 points; 23.60 vs 25.00 is noise. The README says as much in its own resolution table.

Do I need the -exam weights? Only if your evaluation uses ceval-valid-style held-out validation splits. If your harness defaults to ceval (test), the -exam weights are unusable and the base weights are the honest comparison.

Is this a good project to learn LLM training from? For the pipeline, yes — 4,695 lines, hand-written DPO/PPO/GRPO, working DDP and resumption, and a README that documents its own error bars. For production, no: no tests, no CI, no pinned release for the current model, and an install step that currently fails on a clean environment.

Conclusion

MiniMind is not a project that inflated its model to look bigger than it is. 63,912,192 parameters is exactly what the config says, the MoE variant’s “198M-A64M” notation is technically correct, and the compute budget, once converted into FLOPs, implies a utilisation figure a real 3090 can reach. That is a stronger result than most audits of a 67,000-star repository produce.

What fails is the accounting layer around it. “Two hours” is four different measurements; the tokenizer rule of thumb is wrong by 74% on the text style the project demonstrates with; the recommended context length and the script default differ by a factor of 2.26 in compute; the last release predates the model the README documents; the tests it had for one day are gone; three separate fixes for a broken install sit unmerged; and the “lightweight, sampled” alignment corpus is, verified item by item, 99.25% of the CEval test split and 97.20% of MMLU test — while correctly excluding, to the item, the validation split it is actually scored on.

The last one is the most instructive. The project’s contamination claim is true, and it took an option-level fingerprint match across 26,384 items to establish that, because the naive check finds 23 false positives and the true answer is zero. Every number in this article came from files that anyone can download and run the same scripts against. The gap between a project’s headline and its filesystem is usually not a lie — it is the accumulated distance between what was measured, what was estimated, and what was remembered when the sentence was written down.

Measured on 2026-11-21 UTC: repo at commit cc9629e (2026-11-18); GitHub API for stars, watchers, contributors, releases and issues; jingyaogong/minimind_dataset for pretrain_t2t_mini.jsonl, sft_t2t_mini.jsonl and lora_exam.jsonl; ceval/ceval-exam (val and test splits) and cais/mmlu (test) for the contamination comparison; token counts from 12 MiB head samples encoded with the repository’s own model/tokenizer.json. Token totals and all FLOPs/utilisation figures are extrapolations and are labelled as such. Related: TinyZero and the economics of reproducing a reasoning model on one GPU.