AWS Emulator Measured: 56.46% of Its Own Fidelity Tests Pass, 8,261 of 40,130 Operations Answer 501, and 306 Return Success Where AWS Returns an Error

AWS shipped a local AWS emulator in September 2026. It is a Rust binary you install from a Debian or RPM package, point AWS_ENDPOINT_URL at, and run your integration tests against without an AWS account. Anyone who has run that category of tool before knows the failure mode: the tests go green locally and the stack falls over on real AWS.

What makes this one unusual is that AWS publishes the evidence. Every release is supposed to carry a scores.json.gz — the raw pass/fail ledger of a public conformance suite — and the repository tracks individual gaps as markdown files in two directories, issues/things-that-do-not-work/ and issues/things-that-now-work/.

So this is not a review based on a demo. I downloaded the ledger attached to v1.0.322, downloaded the evaluation-suite manifest behind it, sparse-cloned the repository to count the operation files, ran the actual Debian package on a 2-core Debian box, and pointed boto3 at it. Everything below is either read out of AWS’s own files or measured on this machine.

The short version: the emulator works better than its headline number suggests, the 56.46% pass rate is the wrong thing to be afraid of, and the class of failure worth fearing is much smaller and much more specific than “half the tests fail”.

What is actually in the box

AGENTS.md in the repository root is more precise about the architecture than the README is. It describes a workspace generated by something called EmulatorCore v0.1.0, with one Cargo crate per AWS service and one file per operation under src/ops/. The Smithy model ships with each crate and drives the wire protocol; vendor/ holds what AGENTS.md calls an “agent-immutable framework” that contributors are told not to fork.

Counting from a sparse clone:

That 1.04:1 test-to-source ratio on a 15.9-million-line codebase is not what a prototype looks like. It is also not what a finished product looks like, because 8,261 of the 40,130 operations — one in five — are stubs that answer HTTP 501 by construction. The repository tells you so in its own header comment, one line above a census marker that the codebase documents as the way to list them:

# the repository's own census, quoted from the scaffold header itself
grep -rl '^//! SCAFFOLDED, NOT IMPLEMENTED:' */src/ops/ | wc -l
# 8261

A fifth of the API surface is honest about being absent. That is a design choice worth respecting: a 501 is a much better answer than a fabricated success. The issues/ ledger shows the team has explicitly rejected fake inference — one open file describes an earlier attempt to fake a machine-learning response with a ~60-term lookup table and a digest rescaled into [0.90, 1.00), and says plainly: “Do not re-introduce either.”

The published ledger, read directly

scores.json.gz for v1.0.322 decompresses to 16.7 MB of JSON: a wrapper object with commit, schemaVersion, scoredAt, and a units array. Each unit carries an outcome, a kind (behavioral or wire), the services it touches, and — when it fails — a detail block containing the expected and actual status, the failing step, and the operation.

By kind: 32,402 behavioral, 10,535 wire. The scored services number 350.

Per service, the spread is enormous — and it does not follow the pattern most people would guess:

CloudFormation — the service most CI pipelines actually start with — is the worst-covered large service in the emulator at 18.9%, on the largest test population of any service. IoT is at 10.0%. Meanwhile DynamoDB and SQS, the services people usually assume are hardest to fake, are its best scores.

The failures are not spread evenly across intent

The evaluation suite’s manifest.jsonl (29,676 checks, 18.2 MB) labels each check. Joining it to the score ledger by check_id — 29,255 of them match — turns the single 56.46% into something much more useful:

Read that last row carefully. Of the 156 units labelled native data-plane, zero pass; 133 fail and 23 are skipped. The emulator is a control-plane emulator with a control-plane’s strengths: create, describe, list, tag, delete. When a test reaches for a native data-plane protocol, the answer is not “mostly right”, it is nothing.

The happy-path versus rejection gap (65.3% versus 60.6%) is narrower than I expected and points at the same thing from another angle: the emulator is as reliable at refusing valid requests as it is at accepting valid ones. That is consistent with a codebase where a fifth of the operations are unimplemented — a 501 refusal looks like a validation error if you are only checking the status class.

The one number that should worry you: 306

Aggregate pass rates hide the failure mode that actually hurts. For every failing unit, the ledger records expected and actual status. Filtering for the dangerous direction — AWS’s expectation is a 4xx client error and the emulator returned a 2xx success — gives 306 units. Of those, 248 came back as a plain HTTP 200.

These are the cases where your CI proves something false. The test asserts that the SDK raises on bad input, the emulator returns success, the assertion never fires, and the suite is green.

The operations concentrate the same way: CreateDistribution (11), CreateDomain (9), and AssociateSourceToS3TableIntegration (9). Two concrete entries, quoted from the ledger:

• BackplaneControlService / GetMethod: expected 404, actual 200.
• ApiGatewayV2 / CreateApi: expected 400, actual 201.

The mirror image matters too, and it is nine times larger: 2,755 units where AWS expects a 2xx and the emulator returned a 4xx. That is the failure mode you will actually experience day to day — a valid request refused, a test that cannot go green, an afternoon lost to a workaround. It is merely expensive, not dangerous, because it fails closed.

What the 80 skipped units are

Skipped units are easy to overlook and this set is too pointed to ignore. All 80 carry cause: runner-tool-missing, and their IDs start with _smoke.:

• _smoke.js-sdk-v3-drop-in
• _smoke.js-sdk-v3-batch-copy-and-query-operations
• _smoke.opentofu-provider-connects
• _smoke.opentofu-provider-assume-role-resolves-the-role-session

The smoke tests that would demonstrate SDK drop-in compatibility and OpenTofu provider connectivity did not run in the environment that produced the published scores. The product’s central promise — “you point your endpoint at the emulator and your code doesn’t change” — is, in the ledger, supported by tests that were skipped rather than passed. That is not a scandal; it is a gap between the claim and the evidence, and it is exactly the gap a reader should close themselves.

Coverage: 11,082 of 40,130

The manifest names the operations each check exercises. Deduplicating across all 29,676 checks gives 11,082 distinct service:Operation pairs. Against 40,130 operation files in the repository, that is 27.6% of the API surface with at least one fidelity test behind it.

The docs say what to do with the other 72.4%, and are unusually direct about it: “If no fidelity test covers a behavior, the emulator might already handle it correctly, but no fidelity test confirms it yet.”

Two smaller asymmetries in the same join are worth recording because nobody publishes them:

• 421 manifest checks have no scored unit in the v1.0.322 ledger — checks that exist in the suite but produced no result in this release.
• 67 of the 436 service crates have no matching scored service (matched by crate directory name, so this is approximate at the margins). For those services there is no published evidence in either direction.

The gap ledger is growing, not shrinking

issues/things-that-do-not-work/ holds 6,544 markdown files. issues/things-that-now-work/ holds 12,708. Each file carries structured front matter, which makes the backlog quantifiable rather than anecdotal:

Files are named with a UTC timestamp, so filing and resolution can be counted per day. That is where the most revealing trend lives:

In the first ten days resolutions outran filings by an order of magnitude. Then the direction flipped, and filings have outrun resolutions every single day for eleven consecutive days. The worst day is 2026-11-09, with 1,011 gaps filed against 10 resolved.

There are two honest readings and the data does not choose between them. Either the automated audit that generates these files got much more aggressive — plausible for a codebase written by agents against a spec — or the project is accumulating known-broken behavior faster than it fixes it. What is not ambiguous is the arithmetic: the open backlog grew by 3,877 files in eleven days, and 159 of the items filed are marked CRITICAL.

Only 277 of the 6,544 open files carry an Issue-Gap-Class; 6,267 are unlabeled. The resolved side is labeled far more consistently (4,579 of 12,708). The ledger is more legible about what has been fixed than about what is broken.

Twelve releases in five days, and 206 regressions

Between 2026-11-07 09:02 and 2026-11-11 00:32 the project published 12 releases, v1.0.250 through v1.0.322. The release bodies are long lists of fix: <slug> items — the v1.0.322 body is 13,099 bytes of them, and contains no fidelity number at all.

Only 5 of the 12 releases attach scores.json.gz; the asset first appears at v1.0.290. That is a real gap between what the README tells you to do (“check the fidelity scores published with each release”) and what the release page actually gives you for most releases.

The five scored releases do let you watch the pass rate move:

Between the last two there are 482 units that went fail→pass. There are also 206 units that went pass→fail, alongside 400 new unit IDs and 97 dropped ones. Some of those 206 will be expectation changes rather than code regressions — the unit set moved too, so the two cannot be separated from the outside. But “improves continuously, with updates many times a day” (the docs’ own phrase) cuts both ways when you are pinning a CI dependency: improvements and regressions ship through the same channel, at the same speed.

What I measured by actually running it

I extracted the Debian package without root (dpkg-deb -x), started the binary, and pointed boto3 1.44.1 at https://127.0.0.1:9297 with the emulator’s own CA.

Reality check first: the binary reports Version: 1.0.322, Commit: 7753983c6e495ec3612695b1336bf4936fc7b7fd — the exact commit the score file was generated from. The ledger and the artifact I ran are the same thing, which is not something you can assume with most projects.

What worked, first try, no fiddling:

That CloudFormation row deserves emphasis, because §2 above says CloudFormation scores 18.9%. The stack I created provisioned a queue, a bucket, and a parameter, reported the three resources as CREATE_COMPLETE, emitted the events in the right order, and resolved an output. The happy path is not where the 81.1% of failures live.

Now the part I got wrong, which is the most useful part of this article.

I went looking for gotchas with 13 negative-path probes — oversized messages, invalid bucket names, unknown CloudFormation resource types, Ref to nonexistent logical IDs, non-JSON templates, wrong key types, 500 KB DynamoDB items, and more. Three of them were “accepted where AWS rejects”, and I had three findings ready.

All three were my error:

1. SQS oversized message. I was sure the limit was 256 KiB. The emulator accepted 262,145 bytes. Then I read the current SendMessage API reference: the maximum is now 1 MiB. The emulator was right and I was working from a limit AWS had already changed.
2. Lambda inline zip. create_function with a raw non-zip body left the function Pending and Invoke failed InvalidZipFileException. That is what real AWS does with a payload that is not a zip. With a proper zip it went Active and returned the handler’s value.
3. SSM parameter name without a leading slash. The emulator accepted no-slash-name. AWS’s PutParameter docs require the leading slash for parameters in a hierarchy — a flat name is fine.

Then I pushed the SQS boundary to find the real edge, and the emulator enforced it to the byte, with AWS’s own message text, at 1,048,576 versus 1,048,577.

That is the actual lesson of this codebase, and it is a better one than “half the tests fail”. Thirteen targeted negative probes found zero confirmed divergences, while AWS’s own suite knows about 18,614 failing units. Manual spot-checking cannot tell you whether this emulator is safe for your stack — in either direction. Three confident findings dissolved on contact with current documentation, and each one would have been a wrong paragraph in a less careful review. The ledger exists precisely because intuition about AWS is stale, asymmetric, and expensive to be wrong about.

The limit AWS states about itself

Two documents in this project are more candid than the marketing copy, and both should be read before adoption.

The security page says, in bold: “Don’t use AWS Emulator to test whether your application is secure. Passing tests in the emulator don’t show that your permissions are correct or safe.” It then lists what not to test — IAM policies and permissions boundaries, resource-based policies, KMS key policies, cross-account and cross-region access, “any other security control” — and explains the mechanism: “The emulator never denies a request signed with its own key.” It does not check KMS key policies or S3 bucket policies, so “a request that AWS denies might succeed in the emulator. A request that AWS allows might fail.”

Two more that are easy to miss and expensive to discover later:

• “When the emulator runs code for you, such as function code, it doesn’t isolate that code from your machine. The code runs with your user’s permissions and can read and change your files.” Your Lambda tests run with your shell’s privileges, unsandboxed.
• “Ports for resources with their own endpoint don’t check request signatures”, and with --auth off the emulator “serves any request, signed or not.”

And the limitations page rules out the use case people will reach for next: all state is in memory and “the emulator deletes all resources when it stops”, behavior is verified against AWS “in us-west-2 only”, and “we haven’t tested the emulator for long-running use or load testing.” This is a test-run tool, not a local cloud.

How to decide, service by service

The docs point at an AI-agent skill for this. You can do it yourself in five steps, and the numbers above are what you will be reading.

1. List the operations your workflow calls — not the services, the operations.
2. Check coverage in manifest.jsonl from aws/aws-emulator-evaluation-suite. If your operation is not in the 11,082 named, you have no evidence and must test against real AWS.
3. Read the units array in scores.json.gz for your release, filtered to your service. Read the per-operation units, not the service average — AmazonS3 is 62.7% overall, but that number says nothing about your six calls.
4. Filter for the dangerous direction: expected.status >= 400 and 200 <= actual.status < 300. If any unit covering your operation is in that set, your error-path tests are not meaningful against the emulator.
5. Grep the crates for ^//! SCAFFOLDED, NOT IMPLEMENTED: in the operations you depend on. If it is there, the operation answers 501 and no test will change that.

Then pin the release by commit, not by “latest” — 206 units moved the wrong way in one five-day window.

FAQ

Is this LocalStack? No. It is a separate AWS-published implementation, Rust, one crate per service, CC0-1.0, installed as a native package. It is also much newer — the repository was created on 2026-09-11.

Does it need an AWS account or credentials? No. On first start it generates its own access key and writes it to ~/.aws/emulator/<port>/emulator_credentials with a CA bundle beside it, and it rejects unsigned requests. Each run gets new keys.

Why is the pass rate only 56% if the happy paths work? Because a fifth of the operations are 501 stubs and the suite tests the whole surface, including native data-plane protocols where 0 of 156 units pass. The failures are concentrated in services with the largest incomplete surfaces: CloudFormation, EC2, SSM, Auto Scaling, Athena, IoT.

Can I use it in CI? Yes, with two conditions: you only assert on operations with fidelity units that pass, and you keep a separate, real-AWS run for validation and security behavior. The docs explicitly exclude security testing.

What about CloudFormation specifically? It has the largest test population (4,206 units) and the worst pass rate among large services (18.9%), plus 689 open gap files mentioning it. My single three-resource stack worked end to end, so “CloudFormation is 18.9%” is not a statement that it is unusable — it is a statement that 3,391 specific behaviors diverge, and you should read them before trusting a template test.

Is the 15.9-million-line codebase really hand-written? No. AGENTS.md says the project is generated by EmulatorCore v0.1.0 with per-operation scaffolds, and 8,261 operation files are unmodified stubs with the census marker intact. Treat the line count as the size of the generated surface, not as evidence of effort.

The verdict

Judged as an emulator, this thing is better than its score. S3, DynamoDB, SQS, SSM, Lambda and a three-resource CloudFormation stack all behaved correctly in a first sitting, down to the error message text and a byte-exact size boundary. I went looking for defects with thirteen probes and found none I could defend.

Judged as a CI dependency, the numbers say something narrower and more useful. 27.6% of 40,130 operations have any fidelity evidence. 306 units confirm the emulator returns success where AWS returns an error — the exact failure mode that turns green CI into a production incident. And a fifth of the API surface will answer 501 before your test ever gets to assert anything.

The right way to use it is the way the project itself documents, which is unusual in this category: read the ledger first, pin the release, keep error-path and security assertions on real AWS, and treat the emulator as a fast loop for control-plane CRUD rather than as a substitute for a cloud.

If an emulator can tell you exactly which 18,614 behaviors it gets wrong, the least you can do is read the list. That list is the product — the binary is just the part that compiles.

For a second opinion on how much a project’s own disclosure files can reveal, see the OpenMausBot audit, where a sandbox that only switches on in the cloud turned out to be switched off by design, and the agent-desktop audit, where 44 commands report a permission error for a permission they never checked.