Codex Security Ships a 2,737-Case Accuracy Benchmark and Publishes No Score: 22,234 Semgrep Alerts Bucketed Into 2,737 Labels, 29.4% of the Evidence Is Vendored Code, and a Cost-Stopped Scan Is Sealed as Completed

Fri, 13 Nov 2026 00:00:00 +0000

OpenAI’s Codex Security is the most adopted AI security scanner on npm: 785,567 downloads in the last 30 days, 27,360 of them on 11 November alone. The repository has 12,416 stars, 978 forks, 87 contributors, 1,989 commits and 55 releases since July.

It also ships the most carefully specified accuracy harness I have seen in this category. Two thousand seven hundred and thirty-seven labelled cases, pinned to an exact upstream commit and verified by SHA-256. Twenty-one derived metrics with formulas written out one by one. A representative sampling scheme with inverse-probability weights. A documented label boundary that keeps ground truth out of the model’s working directory.

Codex-Security on SoloSoft

Codex Security Ships a 2,737-Case Accuracy Benchmark and Publishes No Score: 22,234 Semgrep Alerts Bucketed Into 2,737 Labels, 29.4% of the Evidence Is Vendored Code, and a Cost-Stopped Scan Is Sealed as Completed