Codex Security Ships a 2,737-Case Accuracy Benchmark and Publishes No Score: 22,234 Semgrep Alerts Bucketed Into 2,737 Labels, 29.4% of the Evidence Is Vendored Code, and a Cost-Stopped Scan Is Sealed as Completed

OpenAI’s Codex Security is the most adopted AI security scanner on npm: 785,567 downloads in the last 30 days, 27,360 of them on 11 November alone. The repository has 12,416 stars, 978 forks, 87 contributors, 1,989 commits and 55 releases since July.

It also ships the most carefully specified accuracy harness I have seen in this category. Two thousand seven hundred and thirty-seven labelled cases, pinned to an exact upstream commit and verified by SHA-256. Twenty-one derived metrics with formulas written out one by one. A representative sampling scheme with inverse-probability weights. A documented label boundary that keeps ground truth out of the model’s working directory.

And in 17,785 lines of markdown, plus every one of the 55 release notes, there is not a single reported precision, recall, F1, accuracy or MCC value.

That is not a rhetorical flourish. I grepped for it. The repository defines strict_precision, strict_recall, strict_f2, strict_mcc, decided_coverage, unsafe_closure_rate, remaining_analyst_workload and fourteen more, with exact denominators, and then never prints a number for any of them. The only percentage in the documentation that describes scan behaviour at all is a UI threshold: “at 80% of a single interactive scan’s limit, the dashboard offers a higher total.”

So I did the only thing an auditor can do with a benchmark harness and no results: I fetched the dataset it pins, verified the hash, and analysed the labels myself. Then I installed the CLI and ran what can be run without an OpenAI account. The composition of the benchmark turns out to be more informative than any score would have been.

The tool, before the critique

There is a version of this article that is unfair, so here is the ground truth first. Codex Security is a real product with a real surface, not a wrapper around a prompt.

The product does six things: scan a repository or a diff, validate candidate findings against source evidence, patch, verify an existing fix, draft SECURITY.md and threat models, and ingest other vendors’ scanner output into a local workbench with deduplication. It exports SARIF, JSON and CSV, and posts to Linear, Jira and GitHub. The TypeScript SDK exposes the same engine.

The GitHub description is “finding, validating, and fixing security vulnerabilities”. The evaluation harness measures one of those three words.

If you are new to the Codex family, the Codex complete guide covers the surrounding toolchain; this article is only about the security product’s measurement claims.

The harness nobody quotes

The benchmark is SastBench, from Rival Security, and OpenAI does not just cite it — it vendors a full runner in evals/triage-finding/sastbench/. The setup is unusually disciplined:

• Dataset pinned to commit b8d95b77 of RivalSecurity/sastbench, with the file’s SHA-256 (57c76563…c9487) asserted at install time, along with record counts and class counts.
• 275 repository revisions hydrated at the exact commits the labels refer to.
• Ground truth stored in test metadata, never in a prompt variable; the model-readable checkout is a throwaway directory that is explicitly denied the label-bearing dataset.
• A documented verdict mapping that decides what a needs_review answer costs you under each metric.
• A representative 100-case sample with a fixed seed and stratum weights, so iteration runs can be compared against a baseline on the same case IDs.

If you wanted to design a credible eval for a security agent, this is close to how you would do it. The metrics table is the part worth reading closely, because it shows the authors thought about the failure modes that flatter a scanner:

Two of those definitions — decided_coverage and unsafe_closure_rate — are the two metrics that most vendor benchmarks quietly omit, because they are the two that punish an agent for going quiet or for closing real bugs. Their presence is evidence that whoever wrote this harness was not trying to produce a flattering number.

And then: nothing. The README says to “run the full benchmark after reviewing sample results and approving a cost and runtime budget”. There is no reference result, no baseline, no target, no regression threshold. The only score assertion anywhere in the repository is the opposite of one, in the sample application’s QA notes: “no detection score is asserted until actual scanner output has been reviewed.”

One more date is worth noting. SastBench was submitted to arXiv on 6 January 2026. The Codex Security repository was created on 13 July 2026. The benchmark existed for six months before the product did, which means no published SastBench result can possibly contain a Codex Security score.

What is actually in those 2,737 cases

I fetched the pinned commit, confirmed the dataset hash matched the value the installer asserts, and parsed all 2,737 records. Here is the composition.

Read the first five rows together. A benchmark with 2,737 cases sounds like breadth. In practice it is 151 repositories, and one PHP application supplies almost a third of the entire test set. If a model has a systematic blind spot in that one codebase, it moves the headline number by 29 points. If it has a blind spot in Rust, the benchmark has almost nothing to say — the file extensions in the labelled locations are dominated by JavaScript (5,580), HTML (5,438), PHP (3,011), Python (2,069) and Ruby (1,695).

Now read the vendored row. Nearly thirty per cent of all evidence locations point into bower_components/, ckeditor, jquery, bootstrap, mocha, or node_modules. In the WeGIA commit I examined, one negative case aggregates 88 alerts spread across help pages, and another aggregates 55 alerts across Bootstrap’s build scripts. Those are not application findings.

And the empties: 219 cases, 8% of the benchmark, carry vulnerability_name set to an empty string. CWE-353 appears in 83 cases with no name at all, because the CWE catalogue entry was never resolved. An agent receiving one of those cases gets a code location, a CWE number, and the generic MITRE description — not a description of the alert.

The label is a proxy, not a judgement

This is the part that changes what a score would mean.

Every one of the 299 positives carries metadata.source = cvefixes. Every one of the 2,438 negatives carries metadata.source = semgrep. All 2,737 records use aggregation_type = cwe_per_commit.

The pipeline, as documented by SastBench itself, is: build a CVE database from NVD plus PyDriller, check out the vulnerable commit, run Semgrep, then aggregate findings by CWE per commit. A case is a true positive when its CWE matches a CVE recorded for that commit. A case is a false positive when Semgrep fired at that commit with a CWE the CVE table does not cover.

The paper says so in its abstract, without hedging: SastBench “combines real CVEs as true positives with filtered SAST tool findings as approximate false positives.” Codex Security’s own evaluation design says the same thing at more length: “SastBench’s false-positive class is produced by its benchmark methodology and is not a fully human-adjudicated corpus. Results therefore measure agreement with SastBench labels, not absolute vulnerability truth.”

So the negative class is not a set of verified-benign alerts. It is a set of alerts at commits where a CVE happened to be filed for a different CWE. Two consequences follow, and both are measurable in the data.

First, the classes are interleaved in the same code. 91.1% of the 269 true-positive commits also carry at least one negative case (245 of 269). At the file level, 64 negative cases point at a file that a positive case also flags at the same commit, and 39 of those have overlapping line ranges. Zero of them share file and CWE, which is exactly what the cwe_per_commit construction predicts.

Here is a concrete pair from DumbWareio/DumbDrop at commit 854f69a0c3b8. server.js lines 393-405, the sendNotification function that builds an apprise command string from a filename, is labelled true_positive CWE-78. server.js lines 377-392, calculateDirectorySize, which calls path.join while walking the upload directory, is labelled false_positive CWE-22. The two records are adjacent in the same file, and they carry opposite labels — not because one function was adjudicated safe and the other dangerous, but because a CVE with CWE-78 existed at that commit and a CVE with CWE-22 did not.

Second, precision here rewards the wrong instinct. 95.7% of all raw alerts live in the negative class. 17.6% of negative cases bundle 10 or more alerts, one bundles 232. A model that answers “not_actionable” whenever the path contains bower_components or mocha will clear an enormous share of the negative class without reading a line of code, and its strict_precision will look excellent.

Put those two facts together and the honest reading of a hypothetical high score is narrow: it would demonstrate that the agent can suppress vendor-directory noise and agree with a CVE-table proxy. It would not demonstrate that the agent can tell a real vulnerability from an adjacent benign one, and it would not transfer to your codebase, where the default assumption — that most alerts are false — is usually wrong in the other direction.

There is a corollary that matters more than the benchmark. If your agent correctly confirms a genuine vulnerability that no CVE covers — which is the normal case for the bug you actually shipped — this benchmark scores it as a false positive.

The sample app: 10 vulnerabilities in 789 lines

The repository does contain something you can run end to end without an account or a model: examples/invoice-desk.

It is 789 lines of dependency-free Node across seven modules — server.mjs, invoices.mjs, imports.mjs, integrations.mjs, accounts.mjs, store.mjs, views.mjs — with ten planted vulnerabilities and an answer key in qa/expected-findings.json. The ten are deliberately varied: secret exposure through a support report, broken object authorization on invoice lookup, cross-tenant cache key collision, stale approval after edit, fail-open validation on import, credential reuse at a different connector destination, shared mutable nested policy, an import check that validates one metadata field and persists another, mass assignment on profile update, and stored HTML injection in notes.

I ran the harness. node --test examples/invoice-desk/qa/app.test.mjs gives 21 tests, 21 pass, including one per planted scenario and a guard that the manifest accounts for all ten.

Two things to notice. One vulnerability every 78.9 lines of application code is a density you will never see, so a scanner scoring 10/10 here has demonstrated that it can find bugs in a target built to be found — the same way a 100% score on a spelling test proves nothing about prose. And the QA directory’s own README is careful about exactly this: “Treat unexpected findings as needing review rather than automatically declaring them false positives”, and “no detection score is asserted until actual scanner output has been reviewed.”

That second sentence is the whole article in one line: the vendor shipped an answer key and declined to grade against it.

What runs without an account (I tried)

I installed @openai/codex-security 0.10.0 globally and worked through the surface with no credentials.

cs scan ./app --dry-run exits 0 and prints “Preflight complete”, followed by a resolved configuration that includes authentication.verified: false, model: gpt-5.6-sol and reasoningEffort: xhigh. That is a dry run that passes while certifying nothing about your ability to authenticate.

A real cs scan ./app on this host then failed, before any model call, with:

Codex could not run a command in its sandbox: bwrap: No permissions to create a new namespace…

That specific error is a property of this machine — an unprivileged container without user namespaces — not a defect in the product. The docs do cover the container case with a Compose override and an AppArmor profile. But it illustrates a real operational point, and open issue #2575 makes it explicitly: --dry-run “check[s] local inputs without starting Codex or loading credentials”, so it cannot demonstrate that the sandbox or the credential path will work in your CI runner. The person who filed that issue asked for a supported, model-free way to qualify the actual worker sandbox and got a documentation request instead.

What did work with no account was the import path — and it worked cleanly. Following the repository’s own example, with a throwaway git repo whose remote matches the report’s repository field:

cs publish findings trivy-report.json --provider trivy --to local \
  --dry-run --source-key example/project/main

returned exit 0, "read": 1, "ready": 1, "excluded": [] and a normalised finding with a stable ID. My two earlier attempts failed with precise, machine-readable validation errors (Missing required option --source-key, then Imported repository "https://github.com/example/project" does not match a configured remote), both exit 2, both with code and fieldErrors. That is good CLI hygiene and worth saying: the free surface of this tool is plumbing for sixteen other vendors’ reports, and the plumbing is better built than the scanning preflight.

The failure mode that costs you a release

None of the above would matter much if a clean scan meant a clean repository. Open issue #2415 — still open, reported against 0.8.3 and unfixed on main at 9b3ce7a — says it does not.

When a Deep Scan crosses --max-cost after discovery has finished, the remaining candidates are never validated, and then:

the bundle is sealed with scan.status: "completed". The scan contract defines completed as “The requested scan reached normal completion” and says “Only completed supports a completed-scan conclusion”. The SARIF export reports executionSuccessful: true, the CLI emits scan.completed and exits 0.

The same condition during discovery produces the opposite outcome: scan.failed classification="cost_limit_exceeded" and exit 2. The reporter traced the path precisely — budget_exhausted_draft records the unvalidated candidates as needs_follow_up and adds one deferred entry with id scan-cost-limit, complete_scan_locked then seals the bundle with the default status completed, and finalize_scan_contract.py sets execution_successful purely from that status. The reason is known to the code — readDeepScanStop produces “Reached the … cost limit. More issues may remain.” — but it reaches the human summary only. The scan.completed event has no stop-reason field.

The reporter’s own test names are the cleanest summary: test_budget_exhaustion_with_no_candidates_is_honestly_partial seals a cost-stopped scan with zero findings as complete, and the only marker is an undocumented deferred ID. And then the sentence that should end every argument about this class of bug:

Candidates that were discovered but not validated are not SARIF results. Suppose a SARIF consumer closes alerts that are missing from a newer successful analysis. It can then close alerts this run never re-checked.

A second open report, #2446, is the same shape from a different cause: Deep Scan workers hit a usage limit, the run finalized rather than paused, and the report was produced after 17 of 2,100 files had been fully reviewed — 0.81% coverage. To its credit the agent’s own summary disclosed that in prose. The report exists; the status was still finalized.

This is the same shape as OpenMausBot’s sandbox gap: the capability exists, the default path does not enable it. The difference is that here the symptom is not a missing dependency but a status field that says completed.

The documentation does not leave room to call these surprises. --max-time is described as advisory: “When selected, it is advisory: spend the available time on useful investigation… This is not a hard timeout.” --max-cost “stops the scan and workers after estimated cost exceeds the limit; in-flight requests can finish above it”, and “Post-scan prompts run after cost tracking and are outside this limit.” The CI example ships a $5 limit and reports findings without failing the job. And reaching a limit “with a valid finalized report returns 0 by default”.

The bug-bounty policy closes the loop. SECURITY.md states that “Missed findings, false positives, incorrect scan results, and performance issues are ordinary bugs” unless they also bypass an enforced restriction. For a security scanner, missed findings are the product’s primary failure mode, and they are explicitly out of scope.

What to verify before you trust a green scan

This is a good tool with a measurement gap and one dangerous status bug. Both are addressable from the outside in an afternoon.

1. Assert on the deferred ledger, not the status. Parse the saved report or manifest JSON and fail the job if deferred contains an entry with id: "scan-cost-limit", or if any candidate carries a needs_follow_up reason beginning “Validation was deferred because the scan reached its cost limit”. That is currently the only machine-readable evidence that a cost stop occurred.
2. Compare coverage to inventory. Take the reviewed-file count out of the report and compare it to the repository’s file count. The 0.81% case above passed every status check available.
3. Prove the sandbox in the real runner. Run one actual scan in the exact CI image before you wire it into a required check, and assert authentication.verified: true from a preflight whose output you actually parse. --dry-run alone will not tell you.
4. Get your own baseline from examples/invoice-desk. Record scanner version, model, mode and reasoning effort alongside the result — 0.10.0’s default is xhigh, and the README’s cost estimates do not survive a model change. Compare against the ten expected findings, and count unexpected findings separately rather than as false positives.
5. Build a 20-case labelled set from your own incident history. Run one vulnerable commit and one fixed commit per case, as the calibration lane in the repo does. That number is the only detection rate that means anything for your codebase. Twenty of your own cases beats 2,737 someone else’s, and you can produce it this week.

FAQ

Does Codex Security need an OpenAI account? Yes for scanning. cs scan needs ChatGPT login or an API key in the environment. The import and export paths do not: publish findings --to local --dry-run runs offline with no login and no model call.

Is it free? No. Scans consume model usage, the CI example sets a $5 estimated limit per scan, and the default reasoning effort is xhigh. Imports, exports and deduplication are local.

Does it replace Semgrep or Snyk? Not as a substitute, and it does not try to. It scans and validates with a model, and separately ingests sixteen other scanners’ report formats into one local workbench so you can deduplicate and triage across them.

What does the shipped benchmark actually measure? Triage of supplied findings: you hand it an alert with a CWE, a location and a description, and it says confirmed or not. Its own design document says: “It does not measure whether an LLM or deterministic scanner discovers vulnerabilities that were never provided as input.”

Why publish no score? No source says. I looked for an explanation in the README, the eval docs, the changelog and all 55 release notes. There is none. The absence is notable only because they built the apparatus; I am not going to invent a motive for it.

Is the cost-stop status bug fixed? Not as of 0.10.0 (2026-11-12). Issue #2415 is open, was reproduced on 0.8.3 and checked on main at 9b3ce7a, and the reporter states the code path is unchanged since 0.8.2.

The number you should want

A security scanner is the one category of software where “no news” is the most expensive possible outcome. A false positive costs a reviewer twenty minutes. A false negative costs you a disclosure, and an unlabeled truncation costs you the ability to tell the difference — you get a green check, a SARIF file with executionSuccessful: true, and no way to know that the run stopped at 0.81% coverage.

Codex Security has done the hard part. It built a 21-metric harness that measures abstention, unsafe closure and analyst workload, pinned it to a hash, and kept the labels out of the model’s reach. That is more rigour than most vendors in this category will ever ship, and it is available to you today: pnpm --dir evals/triage-finding run eval:sastbench with your own budget.

What it has not done is tell anyone the result — and the composition of the test set suggests why that might be uncomfortable. A benchmark whose negative class is a CVE-table proxy, 29.4% of whose evidence sits in vendored libraries, and 29.3% of whose cases come from one PHP application will produce a precision figure that is easy to read as a product claim and impossible to read as a prediction about your repository.

So the useful posture is the one the repository itself recommends: run it, review the output, and do not accept a number you did not measure. Including mine.