Headroom at 77,253 Stars: 300 Log Lines Compressed to Zero, and Four Different Savings Numbers for the Same Tool

Headroom compresses what your agent reads before the model sees it. The GitHub description states the pitch in one line: “20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers.” The PyPI page says “Cut costs by 50-90%”. The README proof table says 21%, 57%, 42% and 30%.

Those are four different claims on four surfaces the same team controls. This article measures what the library actually does to seven kinds of payload, and then compares every published number against every other published number, because the compression tool’s own repository contains the evidence that its front-page figures were not reproducible — “by anyone, including us”, in the words of a script it ships.

Measurements below were taken on 22 October 2026 against headroom-ai 0.41.0 installed from PyPI into a clean Python 3.14 virtualenv (pip install headroom-ai, core extras only). Token counts use tiktoken cl100k_base. No API keys were used and no model was called: only compress() was invoked. Repository facts come from the GitHub REST API and a fresh clone of main (3,324 commits). The vendor’s numbers are theirs; the probe payloads and the comparison are mine.

The result that matters: 300 log lines became zero

I generated 300 lines of ordinary application log — ISO timestamp, level, worker, request id, path, status, duration — and passed them to compress() as a tool result, with a user message asking “Anything wrong in the logs?”

'[300 lines omitted: 300 INFO]\n[300 lines compressed to 0. Retrieve more: hash=23769fb19ad8a5495f533075]'

10,199 tokens to 35. A 99.7% saving. Every line is gone.

The escape hatch is real: the hash= is a CCR pointer, and an agent that knows to call headroom_retrieve can pull the original back out of the local store. But the compressed payload contains no signal that anything specific was dropped — it reports a count and a hash. An agent that reads “300 INFO, compressed to 0” and answers from it is answering from nothing.

I then changed exactly one line, replacing line 211 with the same shape but carrying business facts instead of noise:

2026-10-21T03:14:07 INFO worker-3 request id=447120 path=/v1/orders status=200 dur=88ms tenant=acme-eu plan_downgrade_scheduled=2026-11-01 seats=14

Result: 10,220 tokens to 38. The needle did not survive. acme-eu is nowhere in the output. It is a INFO line, it is not an error, and the compressor treated it as one of 300 interchangeable log lines.

Now add a FATAL line instead, at position 67:

2026-10-21T02:11:09 FATAL worker-3 out of memory while loading shard 7 (heap 3.9GB/4.0GB)

Result: 10,203 tokens to 282, nine lines retained, and the FATAL line comes through byte for byte, as the README’s animated demo promises. The README is not lying. It is showing you the case where the safety net catches the important line.

The thing to notice is what separates the two outcomes: a keyword. The retention rule is not “keep what matters to the question” — compression is not query-dependent, which I verified by running the same payload against three different user questions and getting byte-identical output. The rule is “keep lines that match an error pattern, plus a window of context around them”. Change one word in one line and the same 300 lines go from 9 retained to 0 retained.

That is not a bug in the sense of a crash. The docs describe it openly: error items “containing ’error’, ’exception’, ‘failed’, ‘critical’” are kept, and this is listed under Safety guarantees (additive, never dropped). The guarantee is about a keyword list, and no compressor can promise more than its retention rule. What the guarantee does not tell you is the complement: everything not matching the list, in a log with no matches at all, is removed, and the payload says so in a line a model will read as housekeeping.

What the seven payloads actually did

Two of these rows are the best evidence in this article that the project is not a wrapper around wishful thinking.

The JSON path is conservative in the right way. All 300 rows survived, all 500 numeric rows survived, and every value I checked came through unchanged. What changed was the encoding: JSON objects became a header line plus CSV rows.

[300]{amount:float,created:string,id:string,note:string,status:string}
613.15,2026-10-06T05:00:00Z,ORD-00000,manual,paid
42.46,2026-10-15T18:00:00Z,ORD-00001,retry,paid

That is a 42.2% saving with zero information loss — braces, repeated keys and quote characters are the redundancy. The row an operator would ask about, ORD-00042, is present. This is the honest core of the product, and 42.2% in my test sits right on the limitations page’s own “~52%” for arrays of dicts and well inside the README’s 21–57% band.

The protective gates work. Code and stack traces come back byte-identical, labelled protected:recent_code and protected:error_output. The limitations page says this is intentional and explains why: “Code is almost always fetched because the user wants to work with it. Compressing function bodies would remove exactly what they need.”

Which raises the question of what “20% fewer tokens for coding agents” is counted on. In this test, on code, the answer is 0%. The README’s own proof table answers it more precisely than the tagline: “Code search (100 results) 21%”, “Codebase exploration 42%” — those are JSON search results and file listings, not source files. The savings in a coding agent come from the tool output envelope, not from the code.

Four savings numbers for one tool

Every row below is the same project describing itself.

The PyPI tagline is not a measurement of anything; it is a marketing string that reads as a measurement. The 60–95% on GitHub is the arrays-of-strings/numbers case, which is the easy case — deduplicated paths, tags, metric series. The README’s own proof table, which is the most defensible of the five, never exceeds 57% and averages 41%.

The limitations page goes further than any marketing page usually does. It says, in a callout box:

“The compression figures below are not from a published, repeatable benchmark suite – we could not trace a prior version of this table to one. Instead of an unsourced number, each row is a directional single-sample measurement taken with compress() against a synthetic payload of that shape.”

Read that twice. The project’s own limitations page states that the numbers on its own front page could not be traced to a benchmark. That is unusually candid, and it deserves saying plainly: this is a repository that audits its own claims and publishes the correction. Most projects at 77,000 stars do not.

But the correction went in the docs while the older, larger numbers stayed on PyPI and in the GitHub description, which are the two surfaces a reader meets first.

The provenance problem is documented in code as well. benchmarks/index_proof_table.py opens with this:

“docs/content/docs/index.mdx published four precise before/after token counts with no reproducible source. The nearest harness, real_world_agent_benchmark.py, seeded nothing, so its corpus differed on every run and the published figures could not be reproduced by anyone, including us. A number on the front page of the docs that nobody can regenerate is a liability, not evidence.”

So they built a seeded harness. Which brings me to the smallest finding in this article, and the one I like most.

The harness written to separate “out of the box” from “best case” produces one number

The script is written to print two configurations: default (protect_recent=4), described as “what a coding agent actually gets out of the box”, and full corpus (protect_recent=0), described as “the honest number for ‘how far can this payload compress’”. The comment explains why the distinction matters: “Under the default, protect_recent=4 shields almost every tool result and the measurement says more about the guard than about the compressor.”

The committed results file, benchmarks/results/index_proof_table.txt, contains both runs:

=== default (protect_recent=4) ===
Code search (100 results)          17,199     13,597       21%
SRE incident debugging             55,957     24,340       57%
Codebase exploration               58,801     33,895       42%
GitHub issue triage                46,067     32,429       30%
TOTAL                             178,024    104,261       41%

=== full corpus (protect_recent=0) ===
Code search (100 results)          17,199     13,597       21%
...
TOTAL                             178,024    104,261       41%

Identical, row for row. The parameter the harness was written to test has no effect on these scenarios, because protect_recent is a code-protection window and the scenarios are JSON. So the distinction the script exists to make does not exist in its own output, and the README quotes the single number that both configurations produce, unlabelled as either.

That is not a scandal. It is what happens when a guard is documented as general and implemented as narrow, and it is a good illustration of why “read the harness’s own comment” beats “read the table”.

The default mode does not compress much, and the changelog says so

Headroom’s most important mode is the one its name does not advertise: --mode cache. From the changelog, an entry fixing the savings tracker:

“SavingsTracker.record_request() only appended a history point when tokens_saved > 0 (headroom’s own lossy compression). In --mode cache, tokens_saved is near-always 0 by design, since the frozen prefix is byte-replayed rather than compressed to keep the provider’s prompt cache warm.”

Byte-replayed. In the default cache mode, the frozen prefix of the conversation is not compressed at all — it is re-sent verbatim so the provider’s KV cache still hits, and the savings in that mode are cache savings, not compression savings. The README says the same thing from the other side: “Live-zone compression — only new bytes are compressed (fresh tool output, the latest turn). The frozen prefix stays byte-identical, so the provider cache survives, and history is never dropped.”

This is a design bet, and it is the correct one: an input token read from Anthropic’s cache costs a tenth of a fresh one, so recompressing the prefix on every turn to save 40% of the tokens would lose money. The cost of the bet is that compression applies only to the newest tool output, once, before it freezes.

It also explains a discrepancy that would otherwise look like a bug: the tool compresses “everything your agent reads”, but a long session is mostly frozen prefix, so most bytes are never compressed at all. The bytes you see in headroom savings are the live zone.

I tested the invariant this design depends on, because it is the hardest thing to get right: does the compressor produce stable output for the same message across turns? Two runs on identical input were byte-identical. A three-message conversation and the same conversation extended with a new tool result and a new user question produced a byte-identical compression of the first tool message. Same for three different user questions against the same payload. That determinism is not accidental — it is the property that makes prefix caching survivable — and on this evidence it holds.

The accuracy evidence does not cover the failure mode above

The README reports four accuracy rows. python -m headroom.evals suite --tier 1 in the repository defines nine tier-1 benchmarks, on gpt-4o-mini by default, and the differences matter more than the numbers.

GSM8K and TruthfulQA run through the standard lm-eval harness against the proxy. Their prompts are short word problems and factual questions. There is no 300-line log, no 14,000-token JSON array, and nothing for the JSON or log paths to do — which is consistent with GSM8K coming out at exactly the same score to three decimals (0.870 → 0.870) and with the README’s own note that a ±0.03 delta is inside the confidence interval at N=100. Those two rows demonstrate that a request with nothing to compress is unchanged. They cannot demonstrate that a compressed tool output still supports a correct answer.

The benchmark that measures the thing that can go wrong is the one called Tool Outputs, it samples eight cases, and it does not appear in the README table. The docs do not state what any of these runs compress, which model serves them, or what the SQuAD/BFCL baselines are — their Baseline column is an em dash. If you want to know whether your agent still answers correctly on a compressed log, the project’s published evidence does not answer it, and its own harness, as configured, cannot.

The text-compression model card is the one place where an actual fidelity metric exists, and it is also the most self-critical document in the repository:

“An earlier version of this card reported metrics on the v2.0 split, where 12.2% of test rows were byte-identical to training rows; those numbers were optimistic by roughly 0.02 F1 and have been replaced.”

and

“The rest of the split is text where almost every word should be kept, which inflates F1 for any model.”

At the default threshold (0.5) the model removes 13% of subwords with must-keep recall 0.989. That is a modest, well-documented text compressor. It is also the same product whose PyPI line says 50–90%.

The repository, by the numbers

Two things stand out. First, 226 watchers against 77,253 stars is 0.29% — the lowest ratio I have measured on any project in this series, where comparable repositories sit between 0.55% and 0.64%. Stars are a bookmark; watchers are people who asked to be notified. Second, the project ships 174 releases in nine and a half months — a new version every 1.7 days — inside a 0.x version number, with a changelog of 2,048 entries and a Rust core (12.2% of the language mix) growing next to a Python core. There is no 1.0, no LTS, and no upgrade path document; the changelog is the upgrade path.

Adoption is not in question: 292,636 PyPI downloads in 30 days and 168,866 npm downloads is a tool people are actually installing, and the coverage is wider than most projects this young, with 18 agent integrations listed and a documented unwrap for each.

What to check before you install it

Five checks, in the order that would have caught everything above. All are local and cost nothing.

1. Compress your own worst payload, not the demo. Run compress() on a real tool output from your agent — not the README’s synthetic one — and read the output before you put it in a prompt. The failure mode is a payload that looks complete.
2. Count what survived, not what was saved. For each compressed tool result, assert that the identifiers you actually needed are in the output. In my log test, acme-eu was absent and the marker said “300 INFO” — a model reading that would report a clean run.
3. Treat a compressed to 0 marker as an alarm, not a summary. The default config injects a retrieval tool (inject_tool: True), so the recovery path exists; it only helps an agent that decides to use it.
4. Measure the mode you will actually run. headroom savings reports the live zone. If you are in the default cache mode, most of your context is byte-replayed and will never appear in that number.
5. Run the accuracy check your traffic deserves. The published suite is gpt-4o-mini on academic datasets with N=8 on the one tool-output row. Build a 200-case regression set from your own sessions, run it both ways, and diff the answers — that is the only number that applies to you, and the docs say so themselves: “Your own traffic will vary; measure it with client.chat.completions.simulate() rather than treating these as guarantees.”

FAQ

Is Headroom’s JSON compression lossy? In my test of a 300-row array of objects, all 300 rows survived with values intact; the saving came from converting JSON to a header-plus-CSV encoding (42.2%). That is a re-encoding, not a summary. The lossy path in my tests was the plain-text log path, and the arrays-of-strings/number cases the docs describe as “statistical summary” are explicitly lossy.

Why did 300 log lines become zero? The log compressor keeps lines matching an error pattern plus a context window around them, and keeps head/tail context. With no matching line, there is nothing to anchor retention, and the payload collapses to a count plus a CCR hash. Adding one FATAL line changed the outcome from 0 retained lines to 9.

Does compression break prompt caching? Not in my tests. Output was byte-identical for identical input and stable when the conversation grew, which is what prefix caching requires. Note that the default --mode cache does not compress the frozen prefix at all — that is a deliberate trade to keep cache reads warm.

Is 60–95% real? It is real for arrays of strings and numbers, which is the case the docs measure at ~95% and ~97%. For arrays of dictionaries — “primary use case” on the limitations page — it is ~52% there and 42.2% in mine. The README’s four scenarios average 41%.

Which install do I need for each path? JSON needs core only. Code compression needs headroom-ai[code] (tree-sitter) and protect_analysis_context=False to fire at all. Text/RAG needs [ml], which loads a 274 MB int8 ONNX model. Proxy needs [proxy], MCP needs [mcp]. [all] covers the core stack but not framework adapters or the [vector] backend.

Is it safe to let it rewrite tool outputs invisibly? The protective gates are real — code and stack traces passed through unchanged in every test — and the fail-open behaviour is documented. The risk is concentrated in text and log payloads that compress a lot because most of their content is redundant, where the one line that was not redundant is removed with the rest.

The short version

Headroom is a serious tool with an honest core and four different numbers on its cover. The JSON path does what the README’s proof table claims — I reproduced 42.2% where they claim 21–57% — and it does it by re-encoding rather than deleting. The code path is gated off, so the “20% for coding agents” is really “20% from the JSON around the code”. The log path is where the compression ratio becomes meaningless as a metric: 99.7% saved and 100% of the content removed, with the difference between “nine lines kept” and “nothing kept” decided by whether one line contains the word FATAL.

The project knows most of this. Its limitations page says its own table could not be traced to a benchmark; its harness says the published numbers could not be reproduced “by anyone, including us”; its model card corrects its own contaminated evaluation. That is a rare amount of self-correction, and it is precisely why the numbers that were never corrected — 50–90% on PyPI, 60–95% on GitHub — are the ones to discount hardest.

Measure your own payload. Count what survived. The percentage is the least informative number the tool prints.