Skew Protection for Next.js

Deploy a Next.js app while someone has it open and their tab is now living in the past. The HTML they loaded points at JavaScript chunks with hashed names from the old build. The first time they click a link, the client router asks the server for an RSC payload, or a chunk it hasn't fetched yet, and the server that answers is the new build, which has never heard of either. The best case is a full page reload. The worst case is a broken screen and a console full of ChunkLoadError.

On Vercel this doesn't happen, because old deployments stay alive and skew protection routes each request to the deployment it came from. Secondbook runs on its own k3s cluster, where a rolling update replaces the old pods and that's the end of the old build.

So the question was concrete: how do you keep every open tab talking to the build that served it, on a cluster you run yourself, without keeping every build forever?

Why the usual fixes fall short

The first instinct is to keep old static assets around, in a bucket or a shared volume, so a chunk from the previous build still resolves. That covers files. It doesn't cover the requests that need the old server: an RSC payload is rendered by the build that owns the route tree, and a Server Action is an encrypted reference into one specific build's code.

The second instinct is sticky sessions. A cookie pins a browser to a pod, but the pod gets replaced in the rollout anyway, and the cookie knows which pod it liked, not which build it needs.

What you actually want is to route by build. And Next.js already labels every request with one, if you let it.

The id Next.js already sends

Set deploymentId at build time and Next does two things with it. Every /_next/static URL gets ?dpl=<id> appended, and every client navigation and prefetch carries an x-deployment-id: <id> header. Secondbook's CI sets it to the commit SHA:

// CI sets NEXT_DEPLOYMENT_ID to the commit SHA the image is built from;
// unset in dev means skew protection is off.
deploymentId: process.env.NEXT_DEPLOYMENT_ID || undefined,

The same SHA goes onto the image as a label, so the deploy script can tell which build an image contains without trusting its tag.

That makes the router's job almost boring. Each build runs as its own Deployment and Service, web-<id>, and the router reads the id off the request, header first, then the query string:

func deploymentID(r *http.Request) string {
	id := r.Header.Get(DeploymentIDHeader)
	if id == "" && strings.Contains(r.URL.RawQuery, "dpl=") {
		id = r.URL.Query().Get("dpl")
	}
	if !validID(id) {
		return ""
	}
	return id
}

Five answers to one question

Every request gets exactly one of five outcomes, and the router counts each in a Prometheus metric:

The code is as short as the table:

func (r *Routes) Pick(id string) (*Version, Outcome) {
	if r == nil || r.Serving == nil {
		return nil, OutcomeUnavailable
	}
	switch {
	case id == "":
		return r.Serving, OutcomeUnpinned
	case id == r.Serving.ID:
		return r.Serving, OutcomeLatest
	}
	if v, ok := r.byID[id]; ok {
		return v, OutcomePinned
	}
	return r.Serving, OutcomeFallback
}

The fallback row is what makes the whole thing safe to be wrong about. A tab whose build is gone doesn't break: it lands on the new build, notices the mismatch, and does a full page load.

The active build is a pointer

"Active" can't mean "the newest pods", because the newest pods might not be ready, and a deploy that routes traffic to a build before it can answer is how you turn a release into an outage. So the active build is an explicit pointer in a ConfigMap. A deploy doesn't move it. It stamps the new version with a promoted-at annotation, and the router moves the pointer once that version has ready endpoints, and never to one without.

That one indirection also makes rollback instant. Rolling back is stamping the previous version as promoted again. If its pods are still up, the pointer moves in the next reconcile and the old build serves immediately, with no image pull and no cold start.

How long an old build lives

Keeping old builds forever would be the easy answer and an expensive one, so each version ages through a short life, timed by annotations on its Service:

• For an hour, the previous build stays routable: tabs opened before the deploy keep working. A tab that comes back after that gets the fallback reload.
• Then it is scaled to zero, not deleted: the Deployment and Service stay, with the replica count saved in an annotation, so a rollback is a scale-up plus a pointer move.
• Only two builds are ever kept: a third one deletes the oldest.

The timers live on the cluster objects, not in the router's memory, so a router restart or a change of leader picks them up where they were.

A security fix can't wait an hour

The retention window is a feature until the old build has a hole in it. Then the hour of routable life becomes an hour of exposure. So a version can be promoted with revokes-older=true, from a commit subject containing [security] in CI or by hand, and the moment it is active and ready, every older version is unrouted on every replica at once and deleted after a short drain.

A revoked build is never served again, not even as the fallback when nothing else is ready. The comment above that rule in the planning code puts it better than I can: "a 503 beats serving a build with a known hole". Revocation waits for the fix to be ready first, though. Revoking the old build while the new one is still starting would take the site down to make a point.

Many replicas, one writer

The router runs as two replicas spread across nodes, and both route every request. Each one keeps the cluster state in memory through client-go informers, and the step that turns that state into a routing table is a pure function, so both replicas compute the same table from the same inputs.

Only one of them writes. A Kubernetes Lease elects a leader, and only the leader moves the pointer, stamps timers, scales down and deletes. Every write carries the object's UID and resourceVersion as preconditions, plus a check of the version label, because RBAC can scope a Role to a namespace and a verb but not to "Services that carry this label". The router enforces that in code instead.

What went wrong on the first day

Three things, all visible in the history.

• Some requests skipped the router: two edge routes, the ones serving /_next/static, still pointed straight at the old web Service. Those were exactly the requests that most needed pinning, since a static chunk served by the wrong build is a 404. The fix was moving them behind the router like everything else.
• One 502 on a brand-new version: the first connections reached a Service that kube-proxy hadn't finished programming yet, and got refused. The router, which still proxied then, learned to retry a refused dial up to five times, 200 ms apart, and nothing else.
• The router was a new single point of failure: if both replicas died, the site died with them. Traefik now health-checks the router every two seconds and fails over to a web-fallback Service that covers every web pod. Pinning is lost meanwhile and the site stays up. (Since requests stopped passing through the router, the question mostly went away: Traefik keeps the last routes.)

That last one I tested on purpose in production: with the router scaled to zero for 20 seconds, 100 of 100 requests still returned 200 through the fallback, and with one of the active build's pods deleted, 60 of 60 did.

What the router costs, and how to make it free

A proxy in front of every request has to justify itself, so I measured it. The first test is the router's own end-to-end harness: the real proxy and routing table, real HTTP backends on loopback, ten thousand sequential requests per case. Loopback keeps the network identical in both columns, so the difference is the router's own work.

The routing decision is not where that goes: Pick takes 6.6 nanoseconds and reading the id off a request about half a microsecond. A CPU profile under load put 61 percent of the time in socket system calls. The cost is the hop itself: every request crosses two TCP connections instead of one, and on a small box that halves throughput. Behind a real Traefik, with 32 concurrent clients, the proxy path topped out at 10,668 requests per second.

That pointed at the fix. Traefik can already match on a header or a query parameter, so the router doesn't need to carry the traffic at all, only to tell Traefik where it goes. It serves Traefik's HTTP provider instead: one route per live build, matching x-deployment-id or ?dpl=, and a lower-priority default route to the active build. Traefik sends every request straight to its build. Once that worked, the proxy had no reason left to exist, and v0.3 removed it: the router is Traefik-only now.

Same box, same Traefik, same load:

The second row is what the site would do with no skew protection at all, so in this mode the router costs nothing per request. The ceiling left is Traefik's own, sharing four cores with the load generator and the backend. The pointer, the hour of retention and revocation all work as before, applied as route changes on Traefik's next poll. And if every router replica dies, Traefik keeps the last routes it fetched: I cut the endpoint for eight seconds and it kept answering 200.

Every route keeps its protections

A real site is never one route. Secondbook has a catch-all with its rate limit, a /_next/static route with a looser one, and a route for traffic from inside the cluster with no limits at all. So the router doesn't invent routes: it takes templates, written exactly as Traefik routes, and expands each one into a default router to the active build plus a pinned router per older build, with the template's rule AND the build's id, one priority higher, and the same middlewares. A pinned chunk request still gets the static route's rate limit.

The first version of that got rate limits wrong in a way that only showed up under load. Traefik builds a separate middleware instance per router, and an in-memory rateLimit keeps its buckets per instance. The router used to generate a pinned router for the active build too, so a browser on the current build had two buckets per route, one for requests carrying the id and one for those without, and got twice its limit. Now the build that's serving gets no pinned router: its requests all land on the default one, one bucket, as without skew protection. A tab on an older build still counts apart for the hour that build stays routable; for an exact limit across builds, Traefik's Redis-backed rate limit keys its buckets by middleware and client, not by router.

The contract it creates

Skew protection moves the problem rather than deleting it. For up to an hour, two builds serve real users at the same time, and both talk to the same API and the same database. So the rule in Secondbook's docs is that changes between the web app and the API must stay backward compatible for at least an hour, and deploys go API first, web last, with rollbacks in the reverse order. Only the web build is versioned. A backend fix reaches every build at once, which is usually what you want from a fix.

Server Actions add one more requirement: an action posted from an old tab is encrypted with the old build's key, so the key has to be stable across builds (NEXT_SERVER_ACTIONS_ENCRYPTION_KEY), or those actions fail against the new one.

Where it isn't worth it

• It costs a second set of pods: after every deploy, the previous build runs for an hour. On a small cluster that is real memory.
• Revocation throws away client state: tabs on a revoked build reload into the fix and lose whatever was unsaved.
• A rollback to a build at zero replicas can be slow: if the node pruned the image, it has to be pulled again first.
• It needs Kubernetes: versions are discovered through the API server. On plain Docker you'd need another way to find them.
• It needs Traefik: the routes it publishes are Traefik's dynamic configuration. Another proxy would need its own translation.

The trade

A deploy used to be the moment every open tab got a little bit broken. Now it's the moment a second build starts serving new visitors while the first one quietly finishes with the old ones, and an hour later the cluster forgets it ever existed. That costs a second set of pods for an hour and a promise that the API stays compatible for that hour, and in exchange a deploy is no longer something the people using the site can notice.

The router is open source, extracted from Secondbook with its prefixes made configurable: pablofdezr/skew-router.