TradingAgents at 116,096 Stars: the Paper Says It Eliminated Look-Ahead Bias, the Changelog Lists Nine Corrections Since

116,096 stars. 866 watchers.

That ratio — 0.75% — is the first thing worth knowing about Tauric Research’s TradingAgents, a multi-agent LLM framework that assigns fundamental, sentiment, news and technical analysts to a stock, lets a bull and a bear debate them, then hands the result to a trader, a three-way risk committee and a portfolio manager. The second thing worth knowing is that the paper behind it, arXiv:2412.20138v7, reports a 26.62% cumulative return and a Sharpe ratio of 8.21 on Apple over 88 days, and states in its experimental setup:

Agents make decisions based solely on data available up to each trading day, ensuring no future data is used (eliminating look-ahead bias).

The framework’s own CHANGELOG, meanwhile, records nine separate look-ahead or point-in-time corrections across six releases in 2026 — the most recent eleven days before this article, in v0.7.0.

That gap is the article. Not “the star count is inflated” (it may be perfectly real), and not “LLM trading is a scam” (the repository is unusually diligent in places). The interesting question is narrower and answerable: if you install this today and run it on the window the paper scored, what do you actually get?

I installed it, ran its test suite, instrumented a full decision, and probed every analyst data path on a date inside the paper’s backtest. Everything below is reproducible in an afternoon.

The claim sheet

Before any criticism, here is what is claimed and where.

Note the shape of that table. The two rows a reader most wants — the returns and the Sharpe — are the two that cannot be checked, and the repository says so in the README in almost those words:

Backtest results are not guaranteed to match any published figure. Returns depend on the model, the temperature, the date range, data quality, and the sampling above. Treat the framework as a research scaffold for studying multi-agent analysis, not as a strategy with a fixed, replicable return.

That is an honest sentence, and it is also a disclaimer that resolves the tension by giving up the claim. What follows is what happens when you take the framework at its word and try.

First: the paper’s own arithmetic does not close

Appendix S1.2 defines annualized return as

AR = ((V_end / V_start) ^ (1/N) - 1) x 100%     (S2)

where N is the number of years in the simulation. The simulation is stated as January 1st to March 29th, 2024 — 88 calendar days, or N = 0.2411.

Table 1 prints these numbers:

Compound 26.62% over a quarter and you get an annualized figure near 166%, not 30.5%. Reproducing the printed 30.50% requires a window of roughly 324 days — and the other two rows require 327 and 343, so it is not one wrong constant applied uniformly. There is no note in the paper explaining which convention produced the column.

The pattern holds across the whole table, which is the part I did not expect. Divide the printed AR by the printed CR for every row:

Every baseline row is annualized by a factor near 1.0 — that is, not annualized at all — while the three rows belonging to the proposed system are multiplied by 1.07 to 1.15. Whatever the column is, no row in Table 1 satisfies the paper’s own formula S2 for an 88-day window, and the only rows that received a boost are the authors’ own. Two of the baseline cells are visibly broken in other ways: ZMR on GOOGL prints CR −0.58 with AR +0.58 (sign flip), and MACD on AMZN is four dashes with no explanation.

Two more inconsistencies inside a single section:

• Five stocks, three reported. §5.1 says the simulation runs “across major technology stocks including Apple, Nvidia, Microsoft, Meta, and Google.” §6.1.1 refers to “the three sampled stocks.” Table 1 has columns for AAPL, GOOGL and AMZN — Amazon is never named in the setup, and Nvidia, Microsoft and Meta never appear in any result.
• Drawdown “not exceeding 2”. §6.1.3: “ensured that the maximum drawdown remained within a manageable limit, not exceeding 2.” Table 1 prints AMZN MDD = 2.11.

None of this proves fraud, and I am not alleging it. It does establish something more useful: the numbers in the paper cannot be recomputed from the numbers in the paper. A reader who wants to know whether the framework does anything must run it, and that is the experiment the rest of this article performs.

The footnote that deserves more attention than the table

Under the Sharpe ratio section sits a footnote that is, in my view, the most honest paragraph in the paper:

We benchmarked TradingAgents over 3 months due to intensive LLM and tool use (11 LLM calls & 20+ tool calls/prediction). The highest Sharpe Ratio exceeds our expected empirical range (SR above 2 — very good, above 3 — excellent). We exported TradingAgents’s decision sequences and examined them to ensure calculation correctness. We believe the exceptionally high SR resulted from the phenomenon that there were few pullbacks in TradingAgents during that period. We report results as they are in our experiments faithfully.

Read the second sentence twice. The authors are telling you that a number they published is outside the range they themselves consider credible, and that they chose not to adjust it. The explanation offered — “few pullbacks in that period” — is not a mechanism; it is a restatement of the outcome. Sixty-one trading sessions is enough for an annualized Sharpe estimate to have a standard error of roughly 2 (using the usual approximation SE ≈ √((1 + SR²/2)/T) × √252 at T = 61), so “8.21” is a point estimate whose sampling distribution spans something like 4 to 12. A quarter with no drawdown produces a high ratio mechanically. It does not produce evidence of skill.

The footnote’s first sentence is also the only place the paper states its inference budget: 11 LLM calls and 20+ tool calls per prediction. Keep that number; I will measure it.

Second: the code no longer agrees with the paper’s look-ahead claim

Here is what the CHANGELOG actually records. Every row is the project’s own words, with its own issue number.

Nine corrections, six releases. The paper is dated (v7) June 2025 and asserts look-ahead was eliminated before any of them shipped. Some of these alter decisions and therefore returns — the social feed serving 2026 chatter to a 2024 run, the macro vintage, the memory leak, the JSON fundamentals bypass. Others alter the measurement of returns — #1518 in particular: until eleven days ago, a decision made with information from after the close on day d was scored from the close of day d, a one-session head start baked into the scoring itself.

I want to be precise about what this does and does not imply. It does not imply the paper’s numbers were computed with the later bugs (the paper predates most of the code). It implies something narrower and more damaging to the reader’s use case: the artifact you can download today is not the artifact the paper measured, and the direction of travel has consistently been “we found another leak.” Nine times.

There is a second-order problem that no changelog entry can fix, and the README states it plainly:

Live data moves. News, StockTwits, and Reddit return different content as time passes, so a run today sees different inputs than a run last week even for the same historical trade date. Pin the analysis date to hold the price and indicator window fixed, but the social and news sources still reflect “now”.

That sentence is now wrong — in the safe direction — and the code proves it.

Third: what the shipped stack actually returns for a date inside the paper’s window

I cloned the repository at v0.7.0, installed it in a Python 3.13 virtual environment (pip install -e ".[dev]"), and called the same data functions the agents call, with a window of 2024-02-25 to 2024-03-01 — inside the paper’s simulation. Default vendor configuration, live network.

The same calls on a recent window return real content: 4,166 characters of articles for 2026-10-20..2026-10-26, 4,596 characters of StockTwits messages, 907 characters of Reddit posts.

So the honest reading is this. Four of the six inputs the analyst team depends on are unavailable for the exact dates the paper reported 23–27% returns on. The news analyst, the macro part of its job, and both halves of the sentiment analyst receive a paragraph explaining why they have no data — and the two remaining inputs are prices (fine) and SEC filings as-filed (fine, and genuinely well built: the README’s own example is that Apple’s 2008 total assets were filed as $39.6B and restated to $36.2B in 2010, so a run dated in between reads $39.6B).

I want to credit the engineering here, because the design choice is the right one. The feeds return “unavailable … this is not an absence of X” rather than an empty string, which is the difference between an analyst that knows it is blind and one that hallucinates from silence. The CHANGELOG entry that introduced it (v0.5.0) says exactly that: “A feed that never observed a window reports it as unavailable rather than as an absence, across news, Reddit and StockTwits.”

But the consequence is unavoidable: the paper’s experiment cannot be reproduced on the shipped stack at all. The paper’s dataset section lists Bloomberg, Yahoo, EODHD, FinnHub and Reddit as its news sources. Of those, only Yahoo is in the vendor registry, and Yahoo cannot serve the past. A backtest you run today is a prices-plus-SEC-filings system wearing the paper’s name.

The README’s Reproducibility section is the one place the documentation is actively behind the code: it warns that social and news “still reflect ’now’”, which was true before v0.4.0 and is now the opposite of what happens. If you read only the README, you will expect contaminated inputs; you will actually get absent inputs. Neither is the 2024 dataset the paper used.

Fourth: what the backtest measures, in the code’s own words

The scoring module opens with a docstring that should be on the README’s front page:

Scope: this evaluates decision quality. It is not a portfolio simulator, and must not grow one. Turning a rating into a filled order needs a quantity, a fill price and a cash ledger, none of which the system has; inventing them here would put an execution model behind an evaluation tool. Cells are therefore independent, and a portfolio, when given, is the same standing book for every cell rather than a position carried forward.

That is an unusually disciplined position, and it also means the output is not a strategy return. Concretely:

That last line is from summarize()’s own output footer, not from me. The framework, when it reports a number, appends a sentence saying the number is not repeatable.

Two consequences follow directly.

The horizon is a week. A pipeline that spends 30 model calls and 48 tool calls to reach a decision is then evaluated on five trading sessions of price movement, overlapping across consecutive cells. With a daily grid, cell n and cell n+1 share four of their five outcome sessions, so the aggregate has far less independent information than the cell count suggests.

A 5-day alpha with zero costs is the most forgiving possible test. No model needs skill to look good when position sizing, financing, spread and impact are all free.

Fifth: the grid quietly doubles your sample

iter_grid() steps calendar days. I called it on the paper’s own window:

>>> iter_grid("2024-01-01", "2024-03-29", every_n_days=1)
89 cells          # 24 of them fall on a Saturday or Sunday
>>> len(get_closes("AAPL", "2024-01-01", "2024-03-29"))
61                # actual trading sessions in that window

89 cells against 61 real sessions. Then I checked what the weekend cells actually resolve to, using the framework’s own settlement function:

cell 2024-03-01 (Fri): first close after -> 2024-03-04, exit -> 2024-03-11
cell 2024-03-02 (Sat): first close after -> 2024-03-04, exit -> 2024-03-11
cell 2024-03-03 (Sun): first close after -> 2024-03-04, exit -> 2024-03-11

Friday, Saturday and Sunday all enter at Monday’s close and exit at the same later close. If all three cells carry the same rating — which they usually will, since the model sees nearly identical inputs — they are three copies of one outcome, each counted separately in the mean alpha and in n. A daily grid over Q1 2024 therefore reports n = 89 for what is at most 61 observations, a 46% inflation before any overlap correction.

This is not a look-ahead bug and it does not favour one direction; it simply makes the confidence you might place in a printed n= lower than it looks. every_n_days=7 (13 cells) avoids it by accident.

Sixth: what one decision costs, measured

The paper’s footnote says 11 LLM calls per prediction. I instrumented the repository’s own offline end-to-end test — a scripted chat model, all four analysts, default config (max_debate_rounds=1, max_risk_discuss_rounds=1, max_tool_rounds=20) — and counted every model invocation and every tool call.

=== ONE FULL RUN (default config, 4 analysts) ===
model_calls: 30
tool_calls: 48
max_tools_in_one_turn: 5

Thirty, not eleven. And this is a floor: the scripted model answers after calling each bound tool exactly once, whereas a real analyst may loop up to 20 tool rounds. Multiply by a grid:

Each of those cells also settles the previous cells for its ticker, which adds reflection calls on top. The paper’s own footnote explains that budget pressure is why the backtest is three months long: “Future work will optimize LLM reasoning & tool use to enable longer backtesting under limited budgets.” Three months was a budget constraint, and the current build costs roughly 2.7× more per decision than the build that produced the published table.

You can price that yourself at your provider’s rates — I am deliberately not inventing a dollar figure, because the two default models (gpt-6-sol and gpt-6-luna) are whatever your endpoint charges, and the framework’s model catalog lists 19 provider entries (openai, anthropic, google, xai, deepseek, qwen, qwen-cn, glm, glm-cn, kimi, minimax, minimax-cn, ollama, openai_compatible, mistral, groq, nvidia, bedrock, chatgpt).

Seventh: the repository’s health, on the numbers

Stars are a popularity signal. These are the signals that say whether anyone is maintaining the thing.

The suite is the strongest single signal in the whole audit. Dedicated look-ahead tests — code whose entire purpose is asserting that historical runs cannot see the future — is not what a project writes when it is indifferent to the problem. Somebody cared enough to write them nine times over.

The one test that fails, and why it is a real bug

The failure is tests/test_api_key_env.py::test_ensure_api_key_prompts_and_writes_to_env. It asserts that after prompting for a missing API key, a .env exists in the test’s temporary directory. It does not. The cause is in cli/prompts.py:

env_path = find_dotenv(usecwd=True) or str(Path.cwd() / ".env")

find_dotenv(usecwd=True) walks up the directory tree until it finds an existing .env. In a nested checkout — a monorepo, a container, a CI workspace under a home directory that already has a .env — it finds the ancestor’s file and writes the key there. During my run it wrote DEEPSEEK_API_KEY into a .env three directories above the test’s temp dir. On your machine that file might be a project’s real secrets file.

I removed the line afterwards, and I am reporting it rather than quietly patching it because it is a clean example of the audit’s general point: the test suite defends against look-ahead, and still has a path-writing bug that only shows up when the environment is not the author’s. The suite’s own conftest goes to some trouble to keep tests off the developer’s .env (“The test suite is independent of the developer’s .env, time zone and network” — v0.5.2), and this one path still escapes it.

What is genuinely good here

An audit that only lists defects is as misleading as a README. On the evidence:

• SEC EDGAR as-filed fundamentals, keyless, with a worked example in the README of the same figure filed at $39.6B and later restated to $36.2B. That is point-in-time data done properly, and it is rare.
• Fail-loud vendors. A VendorUnavailableError rather than an empty result; a chain that says so when every vendor is down; coverage_gap() producing “this is not an absence of X” instead of silence.
• A REVIEW sentinel instead of a silent Hold. An unparseable portfolio-manager rating used to be coerced to a tradeable Hold; it now surfaces for review, because “a decision nobody can read is not a Hold” (#1170).
• The execution honesty in backtest.py: refusing to grow a portfolio simulator, and printing “these figures are indicative rather than repeatable” under its own output.
• Regional benchmarks by exchange suffix, so .TW is scored against TAIEX rather than against SPY — a detail most retail backtests get wrong.
• 1,360 tests in 17.6 seconds, ruff clean.

If you want a research scaffold for studying multi-agent argumentation over a stock, this is one of the better-engineered ones. If you want a strategy, nothing in the repository supports that use, and the repository says so.

Before you use it for anything that matters

Six things to run yourself. All of them take minutes and none of them require an API key except the last.

1. Probe a historical date. Call get_news("AAPL", "2024-02-25", "2024-03-01") and read what comes back. If it is a placeholder rather than articles, you are not testing the system you think you are testing.
2. Count your cells against sessions. len(iter_grid(start, end, 1)) versus the number of rows get_closes() returns for the same window. If the first is larger, your n is inflated; use every_n_days=7 or filter weekends.
3. Read one memory log entry. ~/.tradingagents/memory/trading_memory.md. It is the actual results table — rating, raw return, alpha, benchmark, holding window, reflection. If the alphas are clustered inside ±1% after a 5-day window, the system is not distinguishing itself from noise.
4. Run the suite before you trust the build. If pip install -e ".[dev]" && pytest -q does not give you 1,360 passes on your machine, you are on a dependency stack nobody tested — there is no lockfile to save you.
5. Check n per rating bucket, not the total. A Sell: n=3, called the direction 67% line is not a finding. summarize() will print it without complaint.
6. Then, and only then, price a live decision. With a measured 30+ model calls per decision floor, a daily sweep of ten tickers is a five-figure call count per quarter.

FAQ

Does TradingAgents make money? Nothing published lets you answer that. The paper’s returns cannot be recomputed from the paper’s table (see the AR column), the paper’s look-ahead claim has been contradicted nine times by the project’s own changelog, and the shipped data layer cannot serve the paper’s window. The README’s own sentence — “not as a strategy with a fixed, replicable return” — is the accurate answer.

Is the 8.21 Sharpe ratio fake? It is arithmetic on a short sample. Over 61 sessions the standard error of an annualized Sharpe is roughly 2, so 8.21 is an estimate whose distribution is wide; the authors themselves write that it “exceeds our expected empirical range.” A quarter without a drawdown plus zero transaction costs plus a favourable entry convention will produce it. I would not call it fake. I would call it uninformative.

Did the paper use look-ahead? The paper asserts it did not, and I cannot test a 2024 pipeline I cannot run. What I can test is the code that exists now: nine look-ahead or point-in-time corrections shipped between March and October 2026, including one scoring fix on 2026-10-16. Whether the 2024 numbers benefited from the older code’s behaviour is unanswerable from outside, and the paper provides no audit trail (no seeds, no decision logs, no archived inputs).

Why does get_news return a placeholder instead of 2024 news? Because Yahoo’s feed only carries recent articles, and the framework now refuses to pass a window it never observed off as an empty result. That is the correct behaviour and it is what makes the paper’s experiment unreproducible: the paper’s dataset section lists Bloomberg, EODHD and FinnHub as sources, none of which are in the vendor registry.

Can I add a historical news vendor and get real results? Yes in principle — data_vendors is a chain, and Alpha Vantage’s news endpoint accepts historical windows if you have a key. But then you are benchmarking your own dataset, not the paper’s, and you should say so in your notes. The framework’s point-in-time plumbing will not corrupt you; your data vendor might.

Is the 116,096 stars suspicious? Not on its own. The commit pattern (430 of 493 from one author, 24 contributors) is normal for a research project with a strong lead, and 910 closed pull requests is real maintenance. The suspicious-looking number is not the stars — it is 866 watchers, which is the count of people who asked to be told when something changes.

The short version

TradingAgents is not a scam and not a strategy. It is a well-tested research scaffold whose published numbers cannot be reproduced, whose own changelog documents nine look-ahead corrections after the paper claimed the problem solved, and whose default data stack refuses to serve the historical window those numbers came from.

That last part is, paradoxically, the most trustworthy behaviour in the repository: when the current code cannot see the past, it says so, in a sentence designed to stop you from mistaking silence for data. The 116,096 stars were earned by the architecture diagram. The 866 watchers are who you should ask before you point it at a brokerage account.