A Server That Bills Like a Function

Classic serverless has a wasteful default: one instance per in-flight request. When your function calls a database and waits 400ms for the answer, that instance is pinned and idle, and you are billed for every one of those 400ms. For anything I/O-bound, an AI call, a query, a slow third-party API, the waiting is most of the wall-clock, so you are paying mostly to wait. Vercel's fluid compute is the fix for that, and the part worth understanding is how little of it is actually about virtualization.

Two changes, not a new kind of VM#

Fluid compute is two decisions stacked on the same box.

The first is in-instance concurrency: one instance serves several requests at once. When request A blocks on I/O, the runtime's event loop advances request B in the gap, so an instance that used to handle one call at a time now handles many. The second is active-CPU billing: you pay for CPU time, the milliseconds your code actually runs, not for wall-clock. If a request spends 100ms computing and 400ms waiting on a query, you pay for the 100ms. Memory is still billed for the whole life of the instance, but at a far lower rate. Vercel's engineering write-up puts numbers on it: over 75% of Vercel Function invocations now run on fluid compute, and it saves up to 95% on compute bills.

Around those two moves sit a warm pool and a router. Vercel keeps at least one instance of a function running, which it calls "scale to one", so most requests never cold-start, and a resolver holds TCP affinity so a new request lands on an instance that is already warm. Those are the polish. The two moves above are the engine.

One caveat, which Vercel states too: in-instance concurrency only helps when the work is I/O-bound and there is real concurrency. If your function pegs the CPU, sharing an instance just makes two requests fight over the same cores.

The stack underneath, and why you can't copy it#

Vercel Functions run on AWS Lambda, and Lambda runs each sandbox inside a Firecracker microVM: a small virtual-machine monitor written in Rust that boots a guest in about 125ms with under 5 MiB of overhead, and can start up to 150 microVMs per second per host. With snapshots it restores a frozen VM in milliseconds, which is how the warm pools stay cheap. It is real engineering, and it is the part you cannot lift onto an ordinary VPS, because Firecracker needs hardware virtualization: access to /dev/kvm, which you only get on bare metal or with nested virtualization enabled.

That constraint sorts the whole isolation landscape, from lightest to heaviest:

Only the bottom two need KVM. Everything above runs on a plain VM.

The wall my box hit#

I run everything on a Hetzner Cloud box, the same one the rest of my stack lives on (its own story), so I checked what it can actually do:

[ -r /dev/kvm ] && echo "KVM usable" || echo "no /dev/kvm"   # -> no /dev/kvm
grep -oE 'vmx|svm' /proc/cpuinfo | sort -u                    # -> (nothing)
grep -q hypervisor /proc/cpuinfo && echo "this box is a VM"   # -> this box is a VM
stat -fc %T /sys/fs/cgroup                                     # -> cgroup2fs

No /dev/kvm, no vmx or svm CPU flags, and a hypervisor flag that confirms the box is itself a guest. Hetzner Cloud disables nested virtualization, so Firecracker will not start here at all; only Hetzner's dedicated and auction machines expose /dev/kvm. On a Cloud VPS your isolation ceiling is gVisor or WebAssembly, not real microVMs. That sounds like a defeat, and for running untrusted third-party code it is. For running my own code it changes almost nothing, because the clever, expensive microVM was only ever there to isolate other people's code.

Rebuilding the valuable 80% in Go#

Here is the point of the whole post: the two moves that make fluid compute cheap need no microVMs. Both are a few lines of Go.

In-instance concurrency is a worker pool with a limit. A weighted semaphore caps how many invocations an instance runs at once, its "soft limit", the same knob Fly and Knative expose. Because most of each invocation is spent waiting, the waits overlap instead of idling:

// one weighted semaphore per instance caps concurrent invocations
sem := semaphore.NewWeighted(6)
 
func handle(ctx context.Context, req Request) {
    sem.Acquire(ctx, 1)   // blocks here if the instance is already full
    defer sem.Release(1)
    run(req)              // run() is mostly I/O wait, so the instance
                          // keeps taking new work while this one blocks
}

I ran a small version of that: 24 invocations, each 5ms of CPU then 200ms of "I/O", on one instance with a soft limit of 6. Serially that is about 4.9 seconds. With the pool it finished in 823ms at a peak concurrency of 6, because the six in-flight waits happened at the same time. That is the entire trick.

Active-CPU billing is even smaller: one number the kernel already keeps for you. On cgroups v2 every cgroup has a cpu.stat file with a usage_usec counter, and it counts CPU time, not wall-clock. Read its delta around an invocation and you have the exact quantity Vercel calls active CPU:

// usage_usec is CPU time, not wall-clock: the I/O wait never lands here
func activeCPUusec(cgroup string) int64 {
    b, _ := os.ReadFile("/sys/fs/cgroup/" + cgroup + "/cpu.stat")
    for _, line := range strings.Split(string(b), "\n") {
        if v, ok := strings.CutPrefix(line, "usage_usec "); ok {
            n, _ := strconv.ParseInt(strings.TrimSpace(v), 10, 64)
            return n
        }
    }
    return -1
}
// before := activeCPUusec(cg); run(req); billed := activeCPUusec(cg) - before

To check it measures what I think, I ran a program on that same Hetzner box that burns 200ms of CPU and then sleeps 800ms, the sleep standing in for a slow query:

Wall clock 1001ms; active CPU 203ms from getrusage, and 204ms from the cgroup's usage_usec, which agree to within a millisecond. The 800ms of sleeping never shows up in the meter. Bill on usage_usec instead of wall-clock and a long-lived server starts to charge like a function.

What you don't get#

This rebuilds the value, not the whole system, and the gaps are worth naming.

• No hardware isolation on Cloud. Without /dev/kvm you cannot put untrusted code behind a hardware boundary. gVisor gives you a user-space kernel that works without KVM, but it taxes syscalls: a HotCloud study measured simple syscalls at 2.2x and pathological filesystem cases over 200x slower. To isolate other people's code, rent a bare-metal box or accept that overhead.
• Per-invocation CPU is not per-cgroup CPU. usage_usec is measured per cgroup, so once several invocations share one instance you can only prorate, not attribute exactly, unless you give each invocation its own cgroup and pay that cost.
• WASM is the lightest sandbox that needs no KVM (wazero is pure Go, no cgo), but not every language and library compiles cleanly to WASI, and WASI networking has been thin until recently. Check your target stack before promising "any language".
• You will not rebuild Vercel's router. TCP affinity at over 100K requests a second with cross-region failover is hyperscaler infrastructure. A single gateway with per-function affinity is plenty at my scale.

And the honest disqualifier: if your functions are CPU-bound, none of this helps. Instance-sharing only pays off when there is real I/O wait to reclaim. Otherwise one invocation per worker and horizontal scaling is the right answer.

The trade, restated#

Fluid compute reads like a virtualization breakthrough, and the microVM underneath is real work. But the part that cut the bills is an accounting decision: charge for CPU time instead of wall-clock, and reuse the I/O wait with a worker pool. Do those two things and a server you already own begins to bill like a function. The microVM is there to isolate code you don't trust; if the only code you run is your own, you already hold both pieces that matter, and one of them is a single counter in a file the kernel keeps whether you read it or not.