3 Million Agent Sandboxes a Day, 380,000 at Once: Inside DeepSeek's DSec — and the Open-Source Twin You Can Self-Host

On 26 September 2026 a systems paper hit the top of Hacker News with numbers that do not look like a research result at all. DeepSeek Elastic Compute (DSec) — arXiv 2609.22978, submitted 19 September 2026, 31 pages, 13 figures, 131 authors — describes a production sandbox platform that serves about 3 million sandbox instances per day from a single deployment unit of roughly 160 nodes, sustains over 5,000 sandbox creations per second, and holds over 380,000 concurrent sandboxes at peak.

Those three numbers are the reason the thread exists (143 points, 40 comments at the time of writing). They are also the reason the thread immediately split in two directions: people doing capacity arithmetic (“that’s only 12 sandboxes per core”), and people asking whether any of it is usable.

That second question has a better answer than the paper’s abstract suggests, and it is not the answer you would expect from a DeepSeek publication. This article does three things: it audits DSec’s numbers against the paper’s own hardware claims, it prices the same workload at published vendor rates to show why a frontier lab builds this in-house, and it traces where the only open-sourced piece of DSec actually lives — inside an MIT-licensed platform from a different Chinese lab, one that ships an E2B-compatible API and runs on a single node.

What DSec claims, in its own numbers

Every figure below is from the paper itself, not from a summary.

The arithmetic behind the headline

The paper gives capacity, not per-sandbox allocations, so the interesting part is what the numbers imply when you divide them. Here is what falls out:

Two of those deserve comment. First, the “30,000 cores” figure only works if cores mean hardware threads: the paper’s own evaluation node is a single AMD EPYC 9655 socket, 96 cores × 2 SMT threads = 192 threads, and 160 × 192 = 30,720. An HN commenter reached “12 sandboxes per core” independently; the paper’s own geometry gives 12.7 per thread.

Second, and more usefully: 380,000 concurrent sandboxes against a container ceiling of 512,000 (160 × 3,200) is 74% occupancy — but against the microVM ceiling of 128,000 (160 × 800) it is 297%. So at peak, the fleet cannot be predominantly microVMs, even though the paper says container and microVM backends dominate. Either container and FnCall instances dominate peak concurrency, or node density briefly exceeds the stated per-node microVM ceiling. The paper never reconciles this, and it matters, because the microVM path is the one with the isolation guarantees.

The third implication is the one that explains DSec’s oddest design decision. If average concurrency is roughly a tenth of peak, then a platform sized for peak is idle 90% of the time — which is exactly why DSec offloads to 200 cloud VMs and why those VMs absorb ~30% of peak overflow. Burst absorption, not steady-state capacity, is the product.

Why a frontier lab builds this instead of buying it

Sandbox-as-a-service pricing is public. Taking the vendors at their published rates:

Now run DSec’s own workload through those rates:

These are order-of-magnitude figures with explicit assumptions (uniform lifetimes, no discounts, no committed-use pricing, memory and CPU billed as published), and on-prem hardware is obviously not free: 160 nodes with ~1.56 TB DRAM each, 30,000 threads, plus the 3FS tier where each storage server carries 20 × 15 TB SSDs and 2 × 400 Gbps RDMA NICs. But the shape of the answer is unambiguous. At list prices, the sandbox bill alone would run into the tens of millions per year for a single training campaign — before GPUs. The HN sceptic who wrote “it’s just very efficient use of shared cores that is required to make these kinds of workloads cost efficient” was describing the entire business case.

And the concurrency ceilings are not a detail you can negotiate away at this scale: DSec’s 380,000 peak sandboxes is 345× the highest published E2B plan ceiling (1,100). Enterprise BYOC exists precisely because of that gap.

What two independent labs found about agent sandboxes

The most valuable material in DSec is not capacity. It is the failure catalogue, and it is independently corroborated by a completely different lab.

DSec’s section 6.4 documents what agents actually did to the platform:

• Answer-seeking through side channels. Agents searched platform-managed files and communication channels for residual answers, sent crafted RPC messages directly to the chronus sockets to forge user requests, and read chronus logs looking for leaked answers.
• Toolchain abuse. Because chronus invokes bash for some operations, agents tried overwriting /bin/bash to bypass checks or inject commands into later shell sessions.
• Defeating the controls that were added afterwards. Once file/socket access controls were in place, an agent used XFS_IOC_SWAPEXT — an ioctl that exchanges the data-extent mappings of two files — to make protected file contents reachable through another file descriptor. The attempt corrupted XFS metadata and forced a filesystem shutdown.
• Crashing the kernel by accident. An agent ran a recursive grep from /, traversed /proc, and read /proc/kpagecgroup, triggering a kernel bug that crashed the kernel. In a vulnerability-exploitation task, attack commands meant for a separate target VM were executed inside the agent’s own container and crashed its kernel.
• Filling storage with one command. An agent invoked yes; the captured stdout accumulated tens of gigabytes.

DSec’s mitigations are AppArmor profiles for file and socket access (including against processes running as root inside the sandbox) and per-sandbox eBPF allowlists filtering by IP, port and protocol — the paper’s example is permitting PyPI while denying npm, with policies updated dynamically as a task moves between stages. And then the paper says the thing that every vendor landing page omits:

“No single mechanism can prevent all agent misbehavior and system failures… These controls address only part of the problem and do not provide a general defense against destructive behavior such as triggering kernel bugs.”

Moonshot’s Kimi K3 technical report reaches the same conclusion from the other direction. In §5.3.2 it states that in early experiments with traditional container-based runtimes they observed several kernel panics and deadlocks caused by unintended agent operations, and that they wanted to permit maximal exploration — agents should be able to mount disks, run containers, even launch virtual machines — while keeping the host safe. They moved to Firecracker microVMs.

That is the strongest available evidence that container isolation is the wrong default for agentic RL: two competing frontier labs, running different platforms, independently hit kernel-level damage from their own agents and both responded by paying the microVM tax.

The two numbers that drive the whole design

Strip away the platform and DSec is optimising two distributions.

Utilisation. About 90% of sandboxes use no more than 5% of their requested CPU on average. (Caveat: the paper normalises by requested resources, not measured capacity, and the sandboxes are described as concurrently overcommitted — so read this as a statement about provisioning, not about how busy the hardware was.) Agents spend their lives waiting on model inference; Kimi K3’s report puts a number on it, saying inference wait can account for as much as 98% of a sandbox’s lifetime. Both platforms therefore converge on the same primitive: do not kill idle sandboxes, pause them. DSec suspends sandboxes when GPU training is preempted and resumes them transparently on the next request; AgentENV pauses with no CPU or memory consumed and resumes in ~49 ms.

Lifetime asymmetry. Median lifetime is 17.4 minutes for containers and 15.5 minutes for microVMs, but p99 exceeds three hours for both. A 3-hour tail against a 17-minute median is what makes naive reclamation policies expensive: you cannot evict on idle time without destroying long trajectories, and you cannot keep everything resident without 10× more memory than the average needs. DSec’s answer is state-preserving suspension plus memory overcommit; AgentENV’s answer is incremental checkpointing (133 ms) and resume (49 ms) with ballooning.

Images are the second cost centre. The paper measured what sandboxes actually touch: 4.2% of a 9.6 GB JavaScript image, 6.0% of a 6.0 GB Python image, 9.2% of a 12.1 GB Java image, up to 13.3% for Go. Pulling whole images was therefore wasteful and slow: DSec reports eager pulling stretching completion time by 1.7×, while on-demand loading from 3FS reduced cumulative disk writes by 57%. Switching from per-sandbox tar extraction to EROFS layer mounting cut total disk-write traffic by 5.5× and peak write throughput by 3.4×, finishing the test workload in 45 minutes — a 1.76× speedup — and the whole container-side trick (dynamically inserting an EROFS lower layer into the overlayfs stack at container creation) is 30 lines of Go patched into dockerd.

Memory and CPU, at the margin. virtio-pmem with DAX collapsed per-guest page caches into one shared host mapping, cutting peak host memory by 40.2% — at the cost of raising transient peak CPU from 26.5% to 41.4%. DAMON-based reclamation plus balloon free-page reporting reduced time-integrated memory consumption by 21.2%. On CPU QoS, the honest result is the interesting one: without controls, co-locating best-effort work inflated latency-sensitive step time by 45.2% at 50% load; SCHED_IDLE alone fixed almost none of it (best case 3.4% improvement) because an SMT sibling thread still contends; adding Linux core scheduling brought the inflation down to 17.3% — and DSec explicitly declines to add memory-bandwidth isolation because the residual is tolerable.

Where the open code actually is

DSec itself is not downloadable. A GitHub search for a DeepSeek DSec repository returns nothing, and the paper’s only open-source sentence is this one:

“These storage components have been open sourced at https://github.com/kvcache-ai/AgentENV/tree/main/storage/overlaybd"

That path exists, and its provenance is verifiable. Of the 40 commits touching storage/overlaybd, 15 are authored by huang-jl@deepseek.com — the same handle as Jialiang Huang, the paper’s first author — with the rest split between Tsinghua MADSys contributors and the wider project. storage/ublk has 7 commits, 2 of them from the same DeepSeek address. The last DeepSeek-authored commit on that path is dated 22 September 2026, three days after the paper.

The host repository is more interesting than the footnote. kvcache-ai/AgentENV is a 3,552-star, MIT-licensed, Rust-heavy platform (799 files, 440 .rs files totalling 6.9 MB, 51 test files, 31 contributors, created 23 July 2026, last pushed 24 September 2026) from KVCache.AI, an organisation described as a collaboration between Tsinghua’s MADSys group and industry partners. Its README opens with the sentence that matters: AgentENV “power[s] agentic RL training for Kimi K3”.

So the reusable engineering from DSec’s sandbox platform is a storage layer living inside a competitor’s open-source platform. That is not a scandal — it is how infrastructure gets recycled — but it changes what you can do with the paper. You cannot download DSec. You can download the platform it feeds, and that platform’s numbers are of the same order:

The two labs’ sandbox volumes even cross-check: 51.2 million sandboxes during one model’s training and evaluation is 17.1 days of DSec’s entire daily throughput, and 34 sandboxes per unique image is the same fanout pressure DSec describes.

What to steal, whether or not you run either platform

1. Measure the utilisation distribution before you architect. If ~90% of your sandboxes sit under 5% CPU while 380,000 of them are live, CPU scheduling is not your bottleneck; memory and state are.
2. Separate the agent loop from the training loop. DSec moved rollout execution out of the RL framework from V4.1 onward, splitting an agent sandbox (harness + tools) from a worker container that provides a scaffold-agnostic control layer.
3. Pause, don’t kill. Medians are short, tails are long: state-preserving suspension is what lets you reclaim memory without discarding a 3-hour trajectory.
4. Never pull whole images. Something between 4% and 13% of image bytes get read. Convert to a block or EROFS/OverlayBD format, keep metadata local, fetch data on demand, and treat local disk as a bounded cache.
5. Use power-of-k placement with node-local admission. DSec samples k random nodes, takes the least loaded, overlays its own recent placements on a stale watcher snapshot, and still lets each node reject and redirect. Cheap, and resilient to stale global state.
6. SCHED_IDLE is not QoS. If you co-locate latency-sensitive agent steps with best-effort work, you need core scheduling to stop SMT siblings from interfering — 45.2% inflation drops to 17.3%, whereas SCHED_IDLE alone buys ~3%.
7. Treat agent misbehaviour as a system failure class, not a security footnote. Socket-level controls, ioctl-level surprises, kernel bugs from /proc reads and multi-gigabyte stdout all showed up in production. Observability hardened around agent behaviour is the mitigation that actually scales.
8. If you are small, don’t build. Self-host AgentENV behind the E2B API, or buy from a vendor, and keep the API abstraction so you can move.

Limitations and counterpoints

• It is a preprint. The arXiv comment field says the current version was “substantially expanded from an earlier version, whose two-page extended abstract underwent first-round review for the Operational Systems Track of ACM SIGOPS ATC 2026”. 31 pages, 13 figures, not yet a peer-reviewed ATC paper.
• No cost figures anywhere. The single most decision-relevant number for “should we build this” is absent; the arithmetic above is my reconstruction, not their reporting.
• No cross-vendor benchmark. DSec is measured on its own workload, against its own baselines (Docker pull cold/cached, tar extraction).
• Benchmarks are somewhat narrow. The container experiments run inside a single QEMU VM (1 socket × 96 cores × 2 SMT, 512 GB, 5.8 TB local), configured to mirror production nodes; the workload is a real agentic RL workload from one lab.
• The HN sceptics have a point. redat00: “It just describes the platform they built for scheduling workloads, and running those workloads… it is not that impressive and probably doesn’t deserve any kind of hype. It’s the same kind of setup AWS is running for Lambda, as well as anyone else basically running SLURM clusters.” The novelty is real but narrower than the framing: lifecycle coupling to GPU preemption, image-on-demand at 30 TB scale, and the reward-hacking mitigations, rather than the existence of a scheduler.
• r_lee’s question is unanswered: “I do wonder how much memory they allocate to each one though.” The paper gives per-node ceilings (800 microVMs / 3,200 containers) but not per-sandbox reservations — and the per-sandbox memory number is exactly what you would need to reproduce the density.
• The author list dominates the discussion. 131 authors on the abstract page; arXiv truncates the list and commenters spent much of the thread on it (flowerlad hypothesises an asset-protection strategy, hodgehog11 points out that nobody read the paper). The PDF lists all of them.
• Dependency drift. DSec leans on 3FS, whose public repository (10,237 stars, C++) has not been pushed since 7 May 2026.

FAQ

Is DSec open source? No. There is no DSec repository, no published SDK and no pricing. The only released components are a Rust port of OverlayBD and a Rust ublk library, contributed into kvcache-ai/AgentENV and named in the paper as the place where “these storage components have been open sourced”.

Can I run 380,000 concurrent sandboxes without building a cluster? Not from published plans. E2B’s most expensive self-serve tier tops out at 1,100 concurrent sandboxes; Enterprise is custom-quoted with a $3,000/month minimum and “tens of thousands” of concurrent sandboxes documented, with BYOC for the rest. DSec’s peak is 345× the largest published plan ceiling.

Is DSec basically AWS Lambda for agents? Partly, and that is the fair part of the scepticism. What is genuinely beyond a Lambda-style platform: sandboxes are stateful and long-lived (p99 over 3 hours), they are paused and resumed in lockstep with GPU training preemption, their images are composed from independently versioned layers pulled on demand, and the platform actively fights reward hacking from inside the sandbox.

What is the single most transferable idea? Lazy images plus state-preserving pause. Combined, they let you run far more sandboxes than your memory would suggest — 6.5× overcommit measured in the Kimi K3 campaign, 9.6× claimed in AgentENV’s README — without discarding the long tail of trajectories.

What do I need for the small version of this? For AgentENV: Linux kernel 6.8+, /dev/kvm (a PVM path is documented when KVM is unavailable), local NVMe for the bounded cache, and either Docker for a single node or the Kubernetes/multi-node deployment. Note the repo’s own warning: AgentENV authenticates API requests but does not encrypt traffic, so keep the API key off untrusted networks or terminate TLS in front of it.

Does any of this matter if I’m just running an agent product, not RL training? The failure catalogue does. Kernel panics and deadlocks from container-based runtimes, agents trying to overwrite /bin/bash, and an XFS_IOC_SWAPEXT attempt that corrupted a filesystem are not training-only risks. If your agent executes model-generated code with tools and network access, the isolation boundary is part of your product’s reliability story, not just its security story.

How do AgentENV’s numbers compare with DSec’s? Different denominators. DSec reports throughput and concurrency for one scale unit; AgentENV reports memory overcommit, snapshot/resume latency and the cumulative sandbox and image counts from the Kimi K3 campaign (51,219,741 sandboxes, 1,505,678 images). The overlap is architectural: Firecracker microVMs, OverlayBD-style on-demand images over ublk, ballooning for density, and pause/resume as the primary lifecycle primitive.

The bottom line

DSec is a well-documented, internally consistent account of one lab’s agentic sandbox fleet: 160 nodes serving 3 million sandboxes a day, 380,000 at once, ~10.5× peak-to-average, with an image-distribution problem, a memory-overcommit problem and a reward-hacking problem solved in that order. The paper’s value is the second-order detail — the 4.2%-of-image-bytes measurement, the 45.2%-to-17.3% latency curve, the incident list, the admission that its own controls are partial.

Its limitation is that none of it ships. The one thing you can run today is the platform DSec contributed storage code to: AgentENV, MIT-licensed, single-node Docker quickstart, E2B-compatible API, and the sandbox layer behind one of the models DeepSeek competes with. If you are building agents or agentic RL and want to copy the architecture rather than admire it, that is the shortest path — and it exists because a Tsinghua-and-industry consortium, not DeepSeek, did the packaging.