Ponytail at 169,651 Stars: a 2,721-Byte Ruleset, and the 26% Saving That Shrinks to 1.7% Once the Code Is Checked

Ponytail is one markdown file. The project says so itself, in a section called The prompt: “Ponytail is one prompt: skills/ponytail/SKILL.md. The compact version, for agents that read a rules file, is AGENTS.md. Everything else in this repo loads that prompt into different agents.”

That is 2,721 bytes for AGENTS.md. Around it: 214 files, 12,066,966 bytes, and 169,651 GitHub stars. The ratio is 4,435 to 1.

This article is about the number that gets quoted — 53% less code, 41% faster, 26% cheaper, 45% fewer tokens — and what happens when you open the per-task table behind it and sort it by which tasks have a correctness check. Two statistics of mine reproduce the vendor’s headline to the decimal, so the table is the one they used. Sorted differently, it says something the headline does not.

Measurements in this article were taken on 21 October 2026 from the GitHub REST API, raw.githubusercontent.com, an npm test run on a fresh clone, and Playwright renders of ponytail.dev and the JetBrains AI blog. The vendor’s numbers are theirs; my decomposition of their table is mine.

What the artifact actually is

Start with the shape of the repository, because “169,651 stars” and “one markdown file” are both true and they describe very different things.

Two details worth stopping on. The largest text file in the repository is not the product; it is the benchmark’s task corpus, benchmarks/agentic/tasks.py at 1,544 lines. And the single biggest non-image file is hooks/band/logo.ts: three lines and 49,871 bytes, one SVG path string for the statusline logo, rounded to whole units with the comment “invisible at board sizes”. That one file is 18.3 times the size of the rule that does the work.

The rule is also copied. Five files are byte-identical (md5 a7bdf895…), each 2,623 bytes: .agents/rules/ponytail.md, .clinerules/ponytail.md, .qoder/rules/ponytail.md, .windsurf/rules/ponytail.md, .github/copilot-instructions.md. Two more carry the same body with different front matter (.cursor/rules/ponytail.mdc at 2,748 bytes, .kiro/steering/ponytail.md at 2,688). That is deliberate and guarded: node scripts/check-rule-copies.js exits 0 with “Rule copies match AGENTS.md; 10 rule invariants present in SKILL.md and AGENTS.md”, and node scripts/check-versions.js exits 0 with “All 7 version files pinned at 5.3.2”. Both checks pass on a fresh clone, and npm test runs 143 of 143 plus 30 of 30 in the pi extension.

One caveat on that test run, because it matters for anyone reproducing this: without pandas installed, one test fails — csv: correct pandas one-liner passes — with a bare false !== true assertion. The dependency is documented in benchmarks/README.md and in CONTRIBUTING.md, but not in package.json, so npm test on a Node-only machine reports a failure that reads like a defect and is not one. With pandas on PATH it passes.

The engineering is real. It is just not the rule. The band — the statusline HUD added in 5.3 — is 1,676 lines across six files (board.ts 560, register.tsx 487, metrics.ts 317, ponytail-band-import.js 274), 14.2% of all code in the repository, for a feature the README itself limits to “Claude Code 2.1.290 or later, in the terminal or the desktop app. It does not show in the VS Code chat panel, cloud sessions or claude -p.”

How the headline is computed

The claims come from benchmarks/results/2026-10-07-agentic.md: 39 tasks, three arms (no skill, Ponytail v4.13, Ponytail 5), 5 runs each, 585 headless Claude Code sessions on Opus 5.5, CLI 2.1.288, Bash disabled in every arm so that only what the agent writes is measured. Isolation is described honestly, including a disclosed bug: “An earlier version of the harness ran some tasks inside the Ponytail repo. Claude Code then showed the model the repo’s recent commit messages. That is fixed.”

The statistic is stated precisely: “For each task I take the median of the 5 runs, divide arm by baseline, and report the geometric mean of those ratios over the tasks, with a 95% bootstrap interval.” So −53% is not a mean, not a median, and not a total. It is a geometric mean of 39 per-task ratios.

I rebuilt that from the per-task table at the bottom of the same document:

• geometric mean of line ratios: 47.4% of baseline, i.e. −52.6% — their −53%
• geometric mean of cost ratios: 74.1% of baseline, i.e. −25.9% — their −26%

That agreement is the point. The published table and the published headline are the same data, and the table carries one more variable the headline does not: which task has a hidden check that actually runs the delivered code.

Sorting the same table by tier

Eighteen of the 39 tasks are “surgical tasks with hidden checks” — bug fixes, refunds, SQL with untrusted input, path traversal, forged tokens, rate limiting, a privacy case, renames. The other 21 are 12 feature tickets in a fixed FastAPI + React template, 3 open-ended examples, and 6 “build me a small app” requests. The document says this about those 21: “Their ‘pass’ only means the code was delivered or compiles, so I do not count them as a quality result.”

Sum the cost column of the 18 checked tasks, and of the 21 unchecked ones:

The advertised −26% is concentrated almost entirely in the tier that has no execution-based correctness signal. On the tier that does, the saving is one and a half cents per task basket, and it is not uniform: 8 of those 18 tasks got more expensive.

Every one of the eight is in the checked tier. Not one of the 21 unchecked tasks got more expensive. That is not a random split; it is the shape of the effect. On a three-to-twenty-line fix, there is nothing to over-build, so there is nothing to save, and the injected ruleset plus the required “one small test or assert self-check” is pure addition. On tmpl-fe-datepicker — the hero example, 335 lines of hand-built calendar replaced by 10 lines reusing the repo’s own Input — the saving is $0.427 → $0.099.

The other statistics behave the same way when you look per task rather than per geomean:

None of this is hidden. The document says plainly: “Less code did not cost correctness or safety. It did not add any either: the three arms are level.” Hidden checks pass 86/90, 86/90 and 87/90 — 95.6%, 95.6%, 96.7%. What the document does not do is price the two tiers separately, and the gap between them is the difference between adopting this for cost reasons and adopting it for style reasons.

An independent 80-task run, and the vendor table that omits it

On 28 July, JetBrains published the third part of a series that runs public “token saver” skills against the same paired harness. It is the most rigorous outside test of Ponytail I could find, and the numbers are smaller than anything in the repository.

Setup: 80 paired SkillsBench tasks, Harbor 0.18 Docker sandboxes with task verifiers, claude-sonnet-5 at medium effort, Claude Code 2.1.201 headless, Ponytail v4.8.4, 251 billed trials for USD 246.09. They generated the injected text by calling Ponytail’s own hooks/ponytail-instructions.js rather than paraphrasing it, and audited every trial to confirm the ruleset reached the model — 100% of treatment trials, 0% of baselines.

10,205 lines became 8,756. The cost result is the firm one — cheaper on 46 tasks, dearer on 34 — with the authors’ own qualifier that “a bootstrap interval on the median just touches zero”, so “roughly 10% cheaper on this workload” is defensible and “ponytail saves you 10%” is not.

Two findings from that piece matter more than the percentages.

First, the skill does not activate itself. JetBrains installed SKILL.md the obvious way and let Claude Code decide: “Across all ten sessions it self-activated zero times. Not rarely. Never.” The install method decides whether you get anything at all — the plugin’s SessionStart hook is what injects the ruleset, and a plain skill copy very likely measures as nothing.

Second, their bucketing reproduces the tier split I found, from a different direction. Grouped by how much code the baseline wrote: tasks needing almost no code (26 tasks) moved 0.0%; small jobs of 20–99 lines (21 tasks) −11.9%; medium 100–299 lines (24 tasks) −20.9%; builds of 300+ lines (9 tasks) −31.2%. They flag their own method: “we chose these thresholds after seeing the data, and grouping by the baseline’s own output can stretch a gradient like this on its own.”

The vendor’s benchmarks/README.md keeps an “Independent benchmarks” table with two entries, both from June: one at ~44% less code over 24 tasks, one reporting leaner output but higher process cost. I grepped the whole repository for JetBrains: three hits in the markdown, all of them describing the Junie host, plus two JetBrains Mono font strings in board.ts. The July test is not cited anywhere. The counterweight is that this project has an unusually strong habit of self-correction, so the likelier reading is maintenance lag — the table’s newest entry predates that test by five weeks and three major versions — than selection. But a table whose stated purpose is transparency should not be the only place the fourth data point is missing.

The tests claim, and the row underneath it

The second headline is that “98% of risky logic ships with a test” against 68% without. True, and it comes with the row that completes it:

Read those last two rows together and you get a quantity-for-depth trade: the skill writes tests far more often, and each individual test is weaker at catching an injected bug than the test an unassisted model writes when it bothers. The document states this without softening: “No skill writes the most thorough single tests, but often writes none.”

The last row is the one I would put on the landing page if I wanted the project to be taken seriously. On tasks that needed no test, the skill wrote one anyway 45% of the time, up from 36%. In a project whose entire premise is the best code is the code you never wrote, that is the rule failing on its own terms, in its own table, and the author leaves it there.

The reply-judge result is quoted more selectively in the README. The repository says: “in a blind comparison, Ponytail 5’s replies beat the previous Ponytail’s 110 to 67.” The source table has a second row the README does not mention: against no skill at all, Ponytail 5 lost 82 to 106 (p=0.09), and on identical diffs 99 to 118 (p=0.22). “No significant difference” is the document’s own summary; “slight lean towards the longer no-skill replies” is its own gloss.

The number’s own history

Ponytail has now shipped five headline reductions, and the sequence is more informative than any single figure.

The 80–94% figure is still in the repository, in a table, with a warning above it: “Read this number honestly (updated 2026-06-18). The gap above is single-shot, against a bare model that answers with several options plus commentary, so it counts prose, not just code, and overstates the win. #126 was right about that.” The June results file is blunter: “The original 80–94% single-shot numbers were inflated by a chatty baseline, Colin was right. The honest number on real tickets is ‘huge where there’s bloat to cut, nothing where there isn’t, and not at the cost of safety.’”

I want to be fair to that. Most projects in this genre never retract anything, and this one retracts in public, credits the critic by name, discloses a contamination bug in its own harness, and publishes the uncomfortable rows. The 60–94% line in that June conclusion — “huge where there’s bloat to cut, nothing where there isn’t” — is exactly the tier split my recomputation produces and exactly what JetBrains’ buckets found. The information was in the repository the whole time. What travels is the four-number banner.

Distribution: where 169,651 stars actually land

Stars are a bookmark. Downloads are a decision. Ponytail distributes through npm and through GitHub plugin marketplaces, not through release assets:

The release page is a changelog, not a delivery channel. About 102,000 npm installs a month against 169,651 stars is a ratio near 0.6 downloads per star per month, and npm counts include CI and repeat installs, so the real number of humans is lower. That is not a scandal — a 2.7 KB prompt is a curl away and the plugin marketplaces bypass npm entirely — but it does mean the star count and the adoption count are not measuring the same event.

The contributor distribution is the pattern of a project that gets starred far more than it gets worked on:

Commit volume tells the same story with a hole in the middle. By month: June 155, July 51, August 4, September 14, October 139. The npm timeline has a matching gap: 4.8.4 on 29 June, then nothing until 4.9.0 on 7 August — thirty-nine days — then 13 releases between 2 and 16 October, ending at 5.3.2 four days before I measured. v5.0.0 shipped on 8 October, one day after the benchmark file that justifies it.

This is not a criticism of a solo maintainer taking a summer off. It is the answer to a question buyers ask: a project with a four-commit August and a 139-commit October is a project whose roadmap is one person’s calendar. There are no 0.x to 1.x semantics to read either — v1.0.0 and v4.0.0 were both published on the repository’s first day.

The part that is a business

The repository is MIT and the rule is free. Around it, four commercial surfaces:

• A sponsor. GreenPT gets the logo slot in the README.
• Merch. ponytail.dev/merch.html lists fourteen items from a €3.49 sticker to a €59.99 hoodie, with the pitch “No slogans, no name. People who get it will ask.”
• A waitlist. ponytail.dev/soon says only “He’s building something. Won’t say what. Won’t say when. Leave your email and you’ll get exactly one message the day it ships.” Double opt-in.
• A partner hunt. ponytail.dev/stories asks readers to “Connect us with a team that builds AI coding agents or developer tools, someone who picks the tools for a large engineering team, or a possible strategic partner”, contactable at chris@ponytail.dev. The GitHub owner and npm maintainer is dietrichgebert; the stories address is a different name.

That is the standard sequence for a viral developer tool: free artifact, audience, merch, email list, then something paid behind the list. Nothing about it conflicts with MIT — the prompt stays forkable. The thing to note is only that the free/paid boundary has not been drawn yet, so any plan that depends on this ruleset staying exactly as it is today is a plan made against an unannounced roadmap.

One detail in the README deserves quoting for a different reason. “Only install ponytail from DietrichGebert/ponytail on GitHub or @dietrichgebert/ponytail on npm. It never ships .exe or .dll files; a copy that does is not mine.” A project that already has to warn about impostor copies, distributed through fourteen host directories, four root manifests and the twenty marketplaces its badge claims, has a supply-chain surface that no amount of markdown can audit. If you install skills for a team, that sentence is your threat model, not the benchmark table.

What HN said when it mattered

169,651 stars and the best Hacker News showing is 98 points and 17 comments — on 14 June, four days after the repo was created. A second thread in September reached 33 points and 20 comments. For a repository in this star bracket that ratio is unusual, and the comments are the useful part:

• “I like how this repo has 159 files with 11635 LoC, and the load-bearing (!!!) part of it is a couple lines of natural language instructions” — the same observation as my file counts, made in June, when the repo was 55 files smaller.
• “isn’t all this a little excessive for a few line long markdown file!? Why not just display the ‘skill’ itself with a few lines commenting on what it does/how it helps?”
• A two-month user: “it is hit or miss. It’ll make actual high quality suggestions at times. But the thing it’s missing is the actual experience that makes its namesake persona. There’s no sense of nuance of context applied.”
• “Concision begets perplexity” — the objection that optimising for fewer lines makes the trade-offs harder to see, which is the honest counter-argument to the whole ladder.
• And one accusation I will report as a claim and not repeat as a finding: a commenter asserted the stars are “clearly manipulated with bots and fake accounts”. I found no evidence for that and the growth numbers do not require it — a repo on GitHub Trending with four Trendshift badges in its README can reach six figures without anyone buying anything. But claims like that circulate, and the only durable answer to them is the download count, not the star count.

There is also a small ecosystem already: “Show HN: Fleecevest – The Product Counterpart to Ponytail” (September), “Show HN: Piggy – lazy senior dev mode for AI agents (80–94% less code)” (July), and “My benchmark, it beat Ponytail on code, tokens, cost, and time” (August). When your idea generates clones and counter-benchmarks within weeks, the format is the contribution, not the implementation — and the format is four kilobytes of English.

What I would actually do

1. Price your own task mix before believing any of these numbers. Count last quarter’s merged PRs by diff size. If most are 3–50 lines, the checked tier is your reality: expect roughly 2% on cost, occasional 10–30% regressions on individual tasks, and buy it for the review discipline rather than the invoice.
2. Verify the ruleset is actually in the context. Not installed — in. JetBrains got zero self-activations out of ten sessions with SKILL.md dropped in; the plugin’s SessionStart hook is the only thing that reliably injects it. Open one finished session’s transcript and search for a distinctive rule string, such as the shortcut: comment format. If it is not there, you have measured a placebo and will conclude the tool does nothing.
3. Mutation-test the tests it writes. The 98% figure counts presence. The 69%-versus-77% row counts what a test catches. On your own domain, flip a comparison and a boundary in a PR the agent produced and see whether its test goes red.
4. Run the project’s own checks before you trust its CI. node scripts/check-rule-copies.js and node scripts/check-versions.js both pass today, and npm test is 143/143 — with pandas installed. Without it you get one failure that looks like a bug.
5. Read the tier split, not the banner. If a skill vendor publishes a per-task table, sum the columns by whether the task has a checker. That one operation tells you more about where the effect lives than any headline percentage, and almost nobody does it before adopting.

Conclusion

Ponytail is the clearest case I have looked at of a real, working, modest effect wrapped in a large number. The rule is good: six rungs, one paragraph of scope discipline, an explicit list of things never to cut, and a habit of ending every reply with what it skipped. The engineering around it is careful — drift checks that pass, version pinning that passes, 143 tests, a benchmark harness that discloses its own contamination bug, and an author who publicly conceded his first number was inflated.

And the number that travels is the one that mixes 18 checked tasks with 21 unchecked ones and reports a single geometric mean. Split them and the 26% becomes 1.7%, with eight individual regressions, all in the tier where somebody actually ran the code. An independent 80-task test landed at −15.4% code and −10.3% cost, agreed with the shape of the effect, and found that the skill activates itself never.

None of that makes it a bad tool. It makes it a tool you adopt for code review discipline and test presence, at a cost that is roughly flat where correctness is measured and meaningfully lower where an agent was going to over-build anyway. The repository already says that, in one line, in a results file from June: huge where there’s bloat to cut, nothing where there isn’t. The banner just never got the update.

FAQ

Is Ponytail just a prompt? Yes. skills/ponytail/SKILL.md is 3,496 bytes and AGENTS.md is 2,721 bytes; the rest of the 12 MB repository is fourteen host directories, four root-level manifests, a pi extension, a benchmark harness, an i18n set and a statusline HUD.

Does it work in Cursor, Copilot or Windsurf? It installs, but with limits the README states: instruction-only adapters load the always-on ruleset without the slash commands, and Cursor with hooks gets /ponytail level switching “only, typed as a plain message”. Every published number was measured in Claude Code.

Why does the cost saving disappear on small tasks? Because there is nothing to cut and something to add. The ruleset is injected every session and the rule requires a test where logic exists; on a 3–20 line fix, those additions can exceed what the ladder saves. Eight of the 18 checked tasks cost 3–30% more.

What does the independent test actually say? −15.4% code (p=0.088, not significant at 0.05), −10.3% cost (p=0.004, solid), −11% time, −7.5% tokens (within noise), and no detectable quality difference: 6 better, 9 worse, 65 identical.

Is my data sent anywhere? The ruleset does not call anything. The band writes to ~/.config/ponytail/band/ and the board is described as a local page with “no server and no network: inline CSS, inline SVG and a few lines of script”. The site’s privacy page states the opposite of the usual default: “This website sets no cookies and uses no analytics, tracking or advertising services”, with fonts served locally; the waitlist is handled by Brevo as an EU processor under double opt-in.

Should I install it for my team? If you want smaller diffs and more tests, yes — it is MIT, 2.7 KB, and reversible. Do not budget the 26%. Budget 0–10%, verify the injection, and mutation-test two generated tests before you let it near a release branch.