A Kernel Per Bad Idea

The first time one of my agents wrote a script and wanted to run it, I ran it on my laptop, in my home directory, with my shell's environment. That is the security model most agent setups ship with, and it holds right up until the model is wrong, or until something it read talks it into being wrong.

Code an agent writes is untrusted for a reason that has nothing to do with the model being bad at its job. Nobody read it. It was written seconds ago by a process whose input may have included a page it fetched, a repo it cloned, or an issue a stranger filed. Prompt injection is what moves that from a thought experiment to a threat model: instructions and data arrive on the same channel, so "my agent wouldn't do that" is a bet on a text classifier rather than a boundary.

The question I actually had to answer is narrower than "is my agent safe". It's this: when the code does the worst thing it can do, what stops it, and is that thing a wall or a preference?

Why the obvious answers didn't fit#

A container was the first thing I reached for and the first thing I put down. A container is a view of the host kernel, assembled from namespaces and cgroups. Guest root and host root are the same kernel's idea of root, separated by features that all have to be configured correctly and stay that way. The kernel's syscall surface is the attack surface, and it is enormous. A container is a fine boundary between programs I wrote. It's a thin one between me and a program written by something that read a hostile web page.

A V8 isolate is the model Cloudflare Workers use, and it's genuinely excellent at what it does, which is JavaScript. My agents write Python constantly, Go when they're touching my own services, and Rust occasionally. Getting those three into an isolate means a Python interpreter compiled to WASM and a Go binary carrying its own GC: several mediocre runtimes instead of one good one.

A managed sandbox (E2B, Modal) is the answer I'd give most people, and it's the right one if you want zero operations. It wasn't mine for the same reason the rest of my stack isn't rented: the code, its filesystem, and whatever credentials it touches leave my box, and the bill is per-second on work that is bursty by nature. I already turned this VPS into a platform; I wanted the sandbox to live there too.

What I wanted was the thing containers can't give and isolates can't generalise: a kernel the code is welcome to destroy.

One kernel per run#

Firecracker is the VMM that runs AWS Lambda, and its whole point is that a microVM is cheap enough to make per-execution. Each sandbox gets its own guest kernel, its own vCPU allocation, and its own fixed RAM. That changes what a bad program can be. A fork bomb is a fork bomb inside a kernel that only has one vCPU worth of scheduling to give away, and the host never learns about it. A memory bomb hits memory.max and the guest OOM killer eats it. Recursion that never returns burns its own vCPU until the timeout and dies.

The system is called Microvm, it's about 12k lines of Go not counting tests, and the whole design falls out of one assumption: the code inside a sandbox is hostile. Not "might be". Is.

The second row is the part people skip. Firecracker is a process, written in Rust, and a process has bugs. So it doesn't run as me. It runs under the jailer, which chroots it, puts it in a fresh PID namespace, and drops it to a uid that owns nothing:

args := []string{
	"--id", inst.jailID,
	"--exec-file", r.cfg.FirecrackerBin,
	"--uid", fmt.Sprint(r.cfg.UID),
	"--gid", fmt.Sprint(r.cfg.GID),
	"--chroot-base-dir", r.cfg.ChrootBase,
	"--cgroup-version", "2",
	"--parent-cgroup", r.cfg.Slice,
	"--new-pid-ns",
}

The daemon refuses to start if you hand it uid 0, because a second barrier you disabled is not a second barrier. Firecracker also installs a seccomp filter that cuts it down to roughly forty syscalls, and I want to be precise about credit here: that filter is Firecracker's own default. My contribution is declining to pass --no-seccomp. It's a layer I inherited, not one I built.

The network is where an agent actually hurts you#

Everything above stops the guest from escaping into the host. None of it stops the guest from opening a socket, and for agent code that's the interesting direction. A sandbox that can reach the network can reach 169.254.169.254 and read the cloud metadata service, which on most providers hands out credentials to anything that asks. It can scan the LAN the host is sitting on. It can talk to my Postgres, which is only protected by being on a private address.

So egress is off unless you ask for it, and when you ask for it, it's filtered rather than open. The first half is structural: no network means no NIC at all in the VM config, so the guest boots with loopback and nothing else. There's no rule to get wrong because there's no interface.

The second half is where the topology does the work. Every sandbox gets its own /30, not a slot on a shared bridge.

Two sandboxes are never on the same link, so A cannot ARP its way to B; guest-to-guest traffic has to route through the host, where it meets the filter. That matters because it means isolation between two tenants is a property of the wiring rather than a rule I remembered to write. The nftables set is the second half:

const rulesetTemplate = `
table inet {{.Table}} {
	set blocked4 {
		type ipv4_addr
		flags interval
		elements = {
			10.0.0.0/8,         # RFC1918 private
			100.64.0.0/10,      # CGNAT: the ISP's own infrastructure
			127.0.0.0/8,        # host loopback
			169.254.0.0/16,     # link-local, and cloud metadata at 169.254.169.254
			172.16.0.0/12,      # RFC1918 private, and where the sandbox pool lives
			192.168.0.0/16,     # RFC1918 private: the LAN this host sits on
			224.0.0.0/4         # multicast
		}
	}
	...
}`

The sandbox pool lives inside 172.16.0.0/12, so blocking RFC1918 blocks sandbox-to-sandbox as a side effect of blocking the LAN. That is intended, not collateral. There's also a rule dropping anything arriving from a TAP into the host itself, so the host is not a service a sandbox may talk to: not SSH, not the API that created it, not the gateway on its own link.

pip install works. Reading my cloud credentials does not. Allowing the first without the second is the entire job.

One rule in that file exists purely because I ran it on a box with Docker on it. In netfilter, an accept verdict means "stop evaluating this chain", not "deliver the packet": every other base chain on the same hook still runs, and one drop anywhere is final. Docker sets the filter FORWARD policy to drop, so every sandbox packet died after my chain had already accepted it. The fix is two rules inserted at the top of DOCKER-USER, scoped to the ip family so the private-range drops in the inet table still apply.

Limits the guest cannot argue with#

Rate limits live in the VMM rather than on the host, and that placement is the whole point. tc and io.max act after the packets have been produced and copied, and they're a second stateful thing to keep in sync with every VM's lifecycle. Firecracker meters the virtual devices themselves, so there is no interface for the guest to reconfigure and no queue for it to jump:

The network row lands above its cap, which is the limiter behaving as configured rather than leaking. Each bucket is given a OneTimeBurst worth one second of traffic, so a transfer this short is mostly burst and averages above the sustained rate. Without it, a capped VM would pay the throttle on its very first read, boot included, and a limit meant for sustained abuse would show up as a slow start.

Network has a default cap where CPU doesn't need one, for a reason worth stating: a sandbox pinned to a quarter of a core can still push the host's uplink flat, and the first you hear about it is the abuse complaint. The default is ~100Mbit each way.

The ceilings nest. Per-sandbox limits are what a caller asked for; microvm.slice bounds every sandbox at once. Without the outer one, the host's safety would depend on every individual limit being computed correctly, which is a bet that eventually loses.

Killing is done through the cgroup, never by pid:

// The jailer clones into a new PID namespace, so the pid we started is
// not the VMM that ends up running. The cgroup is the exact boundary of
// everything this sandbox is running, and the kernel maintains that
// membership itself, so nothing can hide from it.
g.write("cgroup.kill", "1")

It's SIGKILL and there is no graceful path, because offering hostile code a shutdown hook is offering it a way to outlive its own stop.

No network path into the guest#

The host has to talk to the guest to start commands and collect output, and the obvious way to do that is a port. That would mean the sandbox has a listening socket, which means the isolation depends on the firewall in front of it.

Instead the control channel is HTTP over AF_VSOCK. Firecracker exposes the guest's vsock as a Unix socket in the jail with a CONNECT handshake, so a custom DialContext lets both sides use stdlib net/http, streaming included.

No gRPC, no protobuf, and no network path into the guest to secure in the first place.

That agent is PID 1, and PID 1 has a problem worth knowing about if you ever write one. It must call wait4(-1) to reap processes the kernel reparents onto it. But wait4(-1) reaps any child, including the ones os/exec is concurrently waiting on, and whoever calls it first consumes the exit status. The loser gets ECHILD and never learns how the process exited. Every exec reported waitid: no child processes instead of an exit code. The fix is that PID 1 re-execs itself as a supervisor child and does nothing but reap; the child does the real work and gets to keep its exit codes.

Secrets don't live in the guest#

An agent's code usually needs a token to do anything useful. Environment variables can be set per sandbox and per exec, with the more specific winning, and they are applied by the host on each exec rather than written into the VM at boot:

req.Env = mergeEnv(s.spec.Env, req.Env)

Writing them in at boot would leave credentials sitting in the guest's filesystem, readable by anything inside, long after the command that needed them finished. They're write-only at the API: never logged, and no endpoint gives them back.

Telling the agent why its code stopped#

This is the part I didn't expect to matter and now think is the most agent-specific thing in the system.

An agent that gets a non-zero exit code will do the reasonable thing: assume its program is wrong and start debugging it. If the real reason was that the sandbox hit its TTL, the agent is now rewriting correct code to fix a problem that isn't in the code. It'll do that for as many turns as you let it.

So the statuses distinguish outcomes that all look like failure and aren't the same failure at all:

Both SDKs turn that into one call. Err() returns nothing for a non-zero exit, because that's your program's own answer and not an error in the system, and returns an error for the endings that are not your code's doing:

exe, _ := client.Run(ctx, sb.Id, "python3", "main.py")
if err := exe.Err(); err != nil {
    // the sandbox holding execution exe_01J... was taken away
    // (its TTL, the idle reclaim, or the VM died); your code did not fail
}

Output is buffered on the host, in a 1 MiB ring per stream, and the ring keeps the tail rather than the head: a head-truncating buffer would faithfully preserve the startup banner and throw away the error. The moment you most need a run's output is when it was killed, and output buffered inside the guest dies exactly then.

Starting a command and watching it are two calls, which earns its keep twice. The execution belongs to its sandbox rather than to an HTTP request, so a dropped connection no longer kills a running job (it used to, because the request's context was the exec's context). And because the output is on the host, the stream replays from the beginning before it follows, so an agent reconnecting after a blip loses nothing.

What an agent actually calls#

Two shapes, and the difference is about backpressure. A sandbox is a VM you hold and run commands in; creating one fails when the node is full, so the queueing is your problem. A task goes on a queue, never fails for capacity, and waits for a slot anywhere in the fleet.

For an agent doing several steps that share state (write a file, run it, read what it produced) it's a sandbox. For a hundred independent runs it's tasks.

const sb = await client.sandboxes.create({ image: "python", network: true });
try {
  await client.files.write(sb.id, "main.py", 'print("hello")');
  const exe = await client.run(sb.id, "python3", ["main.py"]);
  console.log(exe.stdout);
} finally {
  await client.sandboxes.delete(sb.id);
}

Defaults are 1 vCPU, 256 MiB, a 15 minute TTL, a 2 minute idle reclaim and a 5 minute exec timeout. The finally is politeness rather than necessity: the TTL and the idle reclaim both exist because an agent that crashes mid-turn will never send the delete.

Everything a caller touches is generated from api/openapi.yaml, which is the contract for the server's wire types, both SDKs, and the docs. The conventions are Stripe's, because they're the ones a developer already knows, with Idempotency-Key on the creates. A request whose reply is lost cannot be known to have happened, so the caller's only options are to retry (and maybe run the work twice) or not to (and maybe never run it). A key gives them a third: retry and get the original answer.

The honest limits#

It needs /dev/kvm, so "VPS" is doing some work in the title. Firecracker needs hardware virtualization, which most cheap VPS tiers do not expose. When I wrote about rebuilding fluid compute on my own box this was the wall I hit. The answer wasn't clever, it was a different machine: a box with nested virtualization enabled, or bare metal. The numbers in this post were measured on a Pi 5, not a $5 droplet.

Cold starts are only good for two of the four languages. Measured, running real code:

Go's 27 seconds is a cold compile with no cache. This is the case a warm pool of snapshots exists to fix, and I haven't built it yet. Today an agent doing a tight write-run-fix loop in Go is going to feel every one of those seconds.

Filtered egress stops the LAN, not exfiltration. A sandbox with network: true can reach the public internet, which means it can POST your data to anything with a DNS name. The filter is about my infrastructure, not about the agent's data leaving. If that's your threat model, the answer is network: false, and an agent that needs pip install isn't getting it.

Two of the layers are weaker than the README implies, which I found out by reading my own code carefully for this post. The seccomp filter is Firecracker's default, inherited rather than authored, and nothing here pins or verifies the ~40 syscall figure. And pids.max is implemented in the cgroup package but never set by anything: no flag reaches it. The argument that it's redundant (host-side it would bound the VMM's threads, not guest processes, and a guest fork bomb is already contained by the vCPU and memory caps) is a real argument, but "the layer is inert as shipped" is the honest description.

I haven't done Firecracker's production host setup in full. The side-channel mitigations on that list (SMT, the various speculative-execution knobs) are the host's job, not the VMM's, and a microVM boundary does not make a shared CPU stop being a shared CPU. For hostile multi-tenancy that list is the price of entry. For my own agents on my own box, I've done part of it.

None of this is audited. It's one person's code. The e2e suite boots real VMs and checks the security claims, because a unit test can assert a firewall rule was rendered and only a booted guest can prove the packet doesn't get out. That's evidence, not a review.

What I actually bought#

The trade is straightforward once you say it out loud. I gave up managed convenience and the cheap VPS tiers, and I still owe myself a warm pool. What I got back is that the sentence "my agent can't do that" stopped being a claim about the model.

It's a kernel. The code can throw itself at it as hard as it likes, burn its vCPU, fill its RAM, fork until the guest scheduler gives up, try to reach a metadata endpoint that isn't routable from where it's standing. Then the VM goes away and the next bad idea gets a fresh one. The best part of building it was watching the questions get boring. They stopped being "did it escape" and became "was that a timeout or a TTL", which is a question with an answer in a table.