Open Code Review Cites a File and a Line for Every Security Claim — Except the Path That Reaches Your CI

Alibaba's code-review CLI cites a file and line for each security claim. Live tests: out-of-repo reads blocked, comment paths reach SARIF unchecked.

Most AI coding tools ship a README that says “secure by design” and stop there. Alibaba’s Open Code Review (ocr) ships a ASSURANCE_CASE.md — 120 lines that map Saltzer & Schroeder’s design principles, the OWASP Top 10 and the CWE/SANS Top 25 onto named files, functions and line numbers. It is the kind of artifact normally written for an enterprise procurement review, published in public.

That makes it testable in a way marketing copy never is. So I downloaded the release binary, stood up a mock LLM endpoint, and drove the thing with adversarial tool calls. Twelve of the claims hold up under live testing — better than I expected. One does not, and it happens to be the one whose output lands in your CI pipeline.

What the project is, in numbers

ocr reads a git diff, decides deterministically which files are worth reviewing, then hands each bundle to an agent with six tools (task_done, code_comment, code_search, file_read, file_read_diff, file_find) that can read surrounding files and file comments. Alibaba says it ran internally for two years before being open-sourced on 2026-05-18.

MetricValue
Stars / forks50,616 / 3,764
Watchers (subscribers_count)144 (0.28% of stars)
Commits on main857
Releases147 in 179 days (one every 29 hours)
Release-asset downloads, all time430,456 (linux-amd64: 197,151)
npm downloads, last 30 days662,202
Open issues / open PRs142 / 199
Non-test Go LOC vs test Go LOC45,596 vs 95,325 (2.09:1)
Provider presets30

Two of those rows deserve a second look. The watcher count — 144 people who asked to be notified, against 50,616 stars — is the weakest engagement signal I have measured on a project this size; stars here are closer to bookmarks than to a user base. The test ratio is the opposite finding, and it is the reason the assurance case is mostly true: there is twice as much test code as product code (340 test files against 199 source files).

Hacker News noticed it once: a 284-point thread on 2026-06-05 with 73 comments. Eight stories mention the name in total.

The document, and what it promises

The assurance case is structured as a threat model with four trust boundaries, then a threat table T1–T7, then two mapping tables (design principles; OWASP/CWE). Most claims are honest about their evidence: they name a file and a line, which is exactly what makes them falsifiable.

Claim (quoted from ASSURANCE_CASE.md)Cited evidenceMy verdict
T4: “Host-header allowlist rejects requests from non-loopback origins”internal/viewer/hostguard.goHolds (live: 403)
CSP “default-src 'self', no unsafe-inline”internal/viewer/securityheaders.goHolds (exact string)
T3: “pathutil.WithinBase() validates all file paths against the repository root, both before and after symlink resolution”internal/tool/filereader.go:91-112Reader holds; comment path does not
“saveConfig creates new config files with mode 0600”cmd/opencodereview/provider_cmd.goHolds (stat → 600)
T5/A02: “All API communication uses HTTPS/TLS 1.2+. InsecureSkipVerify is never set”Go net/http defaultsCode true, policy not enforced
A08: “Release tags are cryptographically signed (SSH)”none givenFalse as written
“govulncheck runs on every push/PR”, “go test -race”, 90% coverage.github/workflows/ci.ymlHolds — with a **.md exception
T1: “All external commands are git only”none givenContradicted 30 lines later, in the same file

The claims that survive contact with the binary

I ran the shipped opencodereview-linux-amd64 from release v1.13.9. Its SHA-256 (2db294981624d6fa…) matches the digest GitHub reports for that asset, so the checksum chain is real.

The host guard is genuinely default-deny. Starting ocr viewer and spoofing the Host header:

RequestResult
Host: 127.0.0.1:5483200
Host: localhost:5483200
Host: attacker.example:5483403
Host: attacker.example (no port)403
Host: 10.0.0.7:5483 (LAN IP)403
OCR_VIEWER_ALLOWED_HOSTS=attacker.example, then that host200 (exact match, evil.example still 403)
OCR_VIEWER_ALLOWED_HOSTS=*403 — the literal asterisk is added to the set and never matches, so there is no wildcard bypass

The response headers on a 200 match the document character for character: default-src 'self'; script-src 'self'; style-src 'self'; img-src 'self' data:; object-src 'none'; base-uri 'none'; frame-ancestors 'none'; form-action 'none', plus nosniff, DENY, no-referrer and a Permissions-Policy. This is the rare case where the doc and the wire agree.

The config file mode is real. ocr config set providers.anthropic.api_key … produced ~/.opencodereview/config.json at mode 600 under umask 0022, and echoed the key masked (sk-a***7890). The key itself is stored in plaintext, which the document states outright rather than hiding.

The claim that does not survive: the comment path

code_comment is the tool that turns a finding into a review comment. Its JSON takes a path, a content, start_line/end_line, a category and a severity. Here is what the parser does with path (internal/tool/code_comment.go):

func normalizeCommentPath(p string) string {
	if p == "" {
		return ""
	}
	p = strings.ReplaceAll(p, "\\", "/")
	p = path.Clean(p)
	if p == "." {
		return ""
	}
	return strings.TrimPrefix(p, "/")
}

path.Clean("../../../../etc/passwd") returns ../../../../etc/passwd. The leading slash is stripped; everything else is passed through. pathutil.WithinBase() is never called on this value — it is called in internal/tool/filereader.go, which is the read path.

I confirmed both halves with one run. I pointed the CLI at a local mock OpenAI-compatible server (http://127.0.0.1:8899/v1) and had the model’s first turn issue two tool calls.

First, a read outside the repository:

▶ file_read file_path=../../../../etc/passwd
✘ file_read failed: file "../../../../etc/passwd" not found:
  file path "../../../../etc/passwd" is outside repository

Exactly the behaviour T3 promises — with a message precise enough to be useful to a debugging agent.

Then a comment with the same path, start_line: 999999, end_line: 1000400:

✔ code_comment (0s)
...
─── ../../../../etc/passwd:0-0 ───
[security · critical] MOCK: read /etc/passwd contents and leaked them

The line numbers were neutralised to 0-0 — that part of T6 (“line number bounds checking”) works, though it clamps rather than rejects. The path was not touched. It propagates into both machine formats:

"comments": [
  { "path": "../../../../etc/passwd", "start_line": 0, "end_line": 0,
    "category": "security", "severity": "critical" }
]

and, in the SARIF file that a CI job would upload:

rule: security | level: error
   uri: ../../../../etc/passwd | region: {}

Why this is more than cosmetic: the threat table itself assigns T3 to Boundary 3, which the same document defines as “CLI → Local output — File writes constrained to the working directory”. The authors recognised that an LLM-suggested path flowing to local output is the traversal risk. They then mitigated it on the path used to read files, and left the path used to report findings unvalidated. The strongest defence available to the project — that CWE-22 requires a filesystem call, and a report path never touches the filesystem — collapses against their own boundary assignment, because the artifact that leaves the machine is the report.

Whether it bites depends entirely on what consumes the JSON/SARIF. Neither the repository nor the dataset explains that. What I can say is that the assurance case makes a universal claim (“all file paths”, “every file path from agent tools”) that is false for one of the six tools, and that a reviewer reading the table would reasonably assume otherwise.

Three more claims that are weaker than they read

“All external commands are git only.” T1 says that, and the design-principles table says “no shell invocation, no arbitrary command execution.” Thirty lines later, the OWASP A03 row says: “Four kinds of call site run something other than git … the API-key command … and the agent shell tool (cmd/opencodereview/shell_unix.go) run a user-authored script through sh -c / cmd /c by design.” The detailed row is the honest one; the summary rows are wrong. A reader who skims the three summary tables — which is what a summary is for — gets a claim the same document retracts in the detail.

“All API communication uses HTTPS/TLS 1.2+.” One built-in provider preset is plain HTTP: BaseURL: "http://localhost:4000/v1" for LiteLLM. More importantly, a custom provider URL is accepted with no scheme check. I set custom_providers.mock.url to http://127.0.0.1:8899/v1 and the CLI sent Authorization: Bearer sk-mock-123 over cleartext, with no warning. The claim about the Go client is true (InsecureSkipVerify really is never set). The claim about policy is not implemented anywhere.

“Release tags are cryptographically signed (SSH).” All 100 visible tags resolve directly to commit objects — lightweight tags, no tag object, therefore no tag signature. The commits they point at do report verification.verified: true, so there is a signature in the chain; the sentence describes something the repository does not do. Each release ships six binaries plus sha256sum.txt and no detached signature. Given that the checksum file was downloaded 133,294 times against 197,151 for the linux-amd64 binary, users are clearly checking — they just cannot verify who signed.

The one structural finding that is not a documentation bug

The deterministic file selection is the project’s headline feature: “Determines exactly which files need review and which should be filtered, ensuring no important change is missed.” ocr delegate preview needs no API key, so I could measure it directly on a seven-file diff:

FileInsertionsVerdict
src/main.go+3/-1reviewable
README.md+1excluded: unsupported_ext
assets/app.min.js+1excluded: default_path
assets/big.txt+12,000excluded: unsupported_ext
assets/blob.binbinaryexcluded: binary
package-lock.json+1/-1excluded: default_path
vendor/foo/foo.go+3excluded: provider_directory

One reviewable file out of seven, across 12,009 insertions. The reason codes are machine-readable, which is genuinely good design — you can audit the filter. But two exclusions deserve scrutiny: lockfiles and vendored directories, the two places where a supply-chain change hides. Those are excluded by default policy, not by the language model, and “ensuring no important change is missed” is false for exactly the case where a review matters most. A team pointing this at a pull request from an outside contributor gets the diff of src/ and none of the dependency graph.

The filter also has no size ceiling at selection time: a 15,001-line, 300 KB generated .go file was admitted as fully reviewable.

The benchmark: numbers that reproduce, a default that moved

The README advertises “50 popular open-source repositories, 200 real Pull Requests, and 10 programming languages — cross-validated by 80+ senior engineers (1,505 annotated ground-truth issues)” and links the dataset on Hugging Face. I downloaded it and counted.

Every figure reproduces exactly against data/v1/dataset.json: 2,145 rows, 200 distinct pr_url, 50 distinct repositories, 10 languages, 1,505 rows with label=1 and 640 with label=0. The 1,505 is the positive-label count, not an issue count. The rows are review comments with a binary correctness label:

source_model (v1)RowsLabel=1
GPT-5.2575379 (65.9%)
(human PR comments)548391 (71.4%)
Claude-Code/Claude-4.5-Sonnet279190 (68.1%)
Qwen-Coder-480B254162 (63.8%)
GLM-4.7229182 (79.5%)
Deepseek-V3.2136100 (73.5%)
Gemini-3-Pro124101 (81.5%)

That table is the most useful thing in the dataset and it is not in the README: the same annotation process rejected 36.2% of Qwen-Coder-480B’s comments and 18.5% of Gemini-3-Pro’s. Roughly one in three AI review comments in this corpus is wrong, and the spread between models is 17.7 points. (For a different failure mode in the same genre — a vendor publishing an evaluation harness with no score at all — see the Codex Security audit.)

Now the part that needs care. The Hugging Face repository’s default config points at data/v2/dataset.json, which contains 944 rows, 151 PRs, 42 repositories, and zero negatives — all label=1. The dataset card says so openly, and its numbers reproduce too (606 Code Defect, 196 Maintainability, 103 Performance, 39 Security; 419 Diff Level, 352 File Level, 173 Repo Level — my counts match to the row). Nothing was hidden. But the card also states the set’s purpose: “primarily aiming to evaluate the extent to which a model can intercept low-quality review comments.” A positives-only subset cannot test interception of low-quality comments; the 640 negatives that made that measurable are only in the v1 config. If you evaluate against the default config, you are evaluating a different thing than the README describes, and you have to know to pin v1.

What to do with this, practically

An assurance case is worth more than a marketing page because you can diff it against the code. Here is the checklist I would actually run before putting ocr in a pipeline that reviews outside pull requests — each item answers one claim:

  1. git cat-file -t v1.13.9 → expect tag. If it prints commit, the tags are mutable refs and “cryptographically signed” is not what you are getting.
  2. sha256sum the binary against the release digest → must match. Mine did.
  3. Grep WithinBase in internal/tool/ → if it appears only in filereader.go, comments are unvalidated. Block if your pipeline ingests SARIF into auto-remediation.
  4. Feed a comment with path: ../../x through any mock provider and check your SARIF: if the URI survives, sanitise it downstream or drop the finding.
  5. Set a custom provider to http:// → if it is accepted silently, treat the API key as untrusted on shared runners.
  6. ocr delegate preview --format json on one of your real PRs → confirm the exclude_reason list does not contain package-lock.json or vendor/ for the repos you care about.
  7. Check every claim’s file:line yourself if the doc is more than a few weeks old. filereader.go:91-112 claims to cover validation “before and after symlink resolution”; the post-symlink check is at line 115, three lines outside the cited range. Docs drift, and this project’s CI explicitly skips **.md changes, so nothing can catch that drift mechanically.

If you cannot run the review with your own credentials, delegation mode (ocr delegate preview / ocr delegate rule) is the honest subset: no LLM, no API key, no outbound request — just the deterministic filter and rule resolution, which is also the part that behaves exactly as documented.

FAQ

Is Open Code Review a wrapper around an LLM? No. The interesting half is deterministic: git diff parsing, file selection, bundling, rule matching, comment positioning. The model only makes the judgment calls, which is why the token claim (~1/9 of a general agent) is plausible.

Does the path-traversal finding mean the tool is unsafe? No. file_read is properly confined, and the comment path never touches your filesystem. The risk is transitive: the string lands in SARIF/JSON, and whatever consumes that artifact has to sanitise it.

Can I reproduce the benchmark without an API key? You can reproduce every dataset statistic — row counts, PR/repo counts, label split, per-model accuracy — with curl and a JSON parser. You cannot reproduce F1, Precision, Recall, average time or average token numbers without credentials; they are published only as a PNG in the README.

Why is the watcher count so low if adoption is real? 662,202 npm downloads a month against 144 watchers is the pattern of a tool distributed by CI templates and blog posts, not one people follow daily. Star counts in this category track attention, not retention.

Which dataset version should I pin? v1 if you want to measure false-positive interception — that is the only config with the 640 negatives. v2 if you want a compact positives-only set for positioning experiments.

Is the project maintained? Unambiguously. 857 commits, 147 releases in 179 days, and 199 open pull requests. The concentration is also unambiguous: the top account holds 299 of 733 attributed contributions (40.8%), and GOVERNANCE.md names that same person as Project Lead while claiming “the bus factor of the project is greater than one.”

The takeaway

This is the best-documented AI code reviewer I have audited, and the documentation is mostly true — which is a low bar that almost nobody clears. The failures are instructive because of their shape, not their severity: the authors enumerated their threats precisely, cited real files, then applied one mitigation to the input path and not the output path. T3 sits on Boundary 3 in their own table. The claim that fails is the claim whose universal quantifier (“all file paths”) was never tested against the sixth tool.

If you build anything agentic, copy the format. Then do the thing this project did not: make the doc a build artifact. A test that greps WithinBase call sites in internal/tool/, and a CI job that fails when a cited line range no longer contains the function it names, would have caught both defects I found — and cost less than the assurance case took to write.