--- title: The 21st Visitor Broke Live Mail date: '2026-10-22' description: 'Buzontemporal''s live inbox held one Mailingest stream per visitor, and Mailingest allows 20 per account. Why raising the cap wasn''t the fix, and how Centrifugo was.' author: Pablo Fernandez tags: - Realtime - SSE - Centrifugo - NATS - Mailingest source: 'https://www.pablofr.com/blog/the-21st-visitor-broke-live-mail' --- [Buzontemporal](https://www.buzontemporal.com) is a temporary email site: you open the page, an address is waiting, and mail sent to it appears without a reload. That last part, the live inbox, had a number hidden in it that nobody had multiplied out. Twenty visitors could watch their inbox update in real time. The twenty-first got an error, and so did everyone after them. Nobody actually hit it. It was caught on paper on launch day, about an hour and a half before the site went public, which is the cheapest moment to find a bug like this one. But the count is worth walking through, because the limit that caused it was right, the obvious fix was wrong, and the real fix deleted more code than it added. ## One stream per visitor The first version was the straightforward one. The browser held a server-sent events (SSE) connection to Buzontemporal's API, core-api. For each of those, core-api opened its own SSE stream to [Mailingest](/blog/giving-an-inbox-to-each-agent), my mail platform, for that one mailbox, and relayed whatever came down it. Mailingest, for each stream, subscribed to the mailbox's subject on NATS, its message bus, and checked Postgres every 60 seconds that the mailbox still existed. ``` browser ── SSE ──▶ core-api ── SSE + API key ──▶ Mailingest (a tab) (relays it) one stream per mailbox ├─ NATS subscription ├─ Postgres check / 60 s └─ Redis lease (owner: 20) ``` *Before: every open inbox was a chain of two streams, and the far end held a subscription, a database check and a Redis lease.* Each link in that chain is reasonable on its own. Mailingest's stream resumed after a dropped connection with `Last-Event-ID`, replaying what the client missed from Postgres. It ended after an hour, so a revoked API key stopped receiving within the hour. core-api parsed the events, checked them against a schema and capped its own streams at 3 per client address and 2,000 in total. It worked in every test, because every test had a handful of visitors. ## The limit that did its job Mailingest doesn't let anyone hold streams without limit. Every stream takes a lease in Redis against three bounds at once, counted across every replica: ```rust:crates/temp_mail/src/http.rs /// Open event streams across every replica, bounded three ways: in total (per /// service), per mailbox and per owner (an account, or a visitor's client /// network). One global pool let a single client hold every slot — and every NATS /// subscription behind them — until its mailbox expired; a pool per replica let N /// replicas admit N times the bound. /// Streams one mailbox may have open at once. pub const MAX_STREAMS_PER_MAILBOX: usize = 3; /// Streams one owner may have open at once, across its mailboxes. pub const MAX_STREAMS_PER_OWNER: usize = 20; ``` The comment says why the per-owner bound exists: an earlier version had only a global pool, and one client could take every slot. Twenty streams per owner is plenty for one account watching its own mailboxes. When it's reached, the stream is refused with a `429 too_many_streams`. The problem is who the owner is. Buzontemporal talks to Mailingest with one API key, so every visitor's stream belongs to the same account. To Mailingest, ten thousand people with the inbox open look exactly like one customer holding ten thousand streams, which is precisely what the bound is there to stop. *Add visitors past twenty under each design. With one stream each, the 21st is refused; with Centrifugo, Mailingest's side doesn't grow with the visitors at all.* A limit counts something, and this one counted accounts. Buzontemporal's visitors aren't accounts. They're people behind one, and nothing at Mailingest's edge could tell them apart. ## Raising the number doesn't help The first fix that comes to mind is a bigger number, or a separate limit for trusted integrators. It would have worked for a day. What it hides is that every visitor costs Mailingest something for as long as their tab is open, whether or not any mail arrives: - **A NATS subscription per visitor.** 50,000 open inboxes are 50,000 subscriptions. - **A Postgres query per visitor per minute,** to notice a deleted mailbox. 50,000 open inboxes are about 830 queries a second, doing nothing but asking whether things still exist. - **Two connections per visitor,** one from the browser to core-api and one from core-api to Mailingest, each a socket and a buffer on both sides. The cost grows with people looking, not with mail arriving. Most temporary inboxes receive one message, or none, so almost all of that work is waiting. ## Option one: multiplex it yourself The well-known fix is to stop opening a stream per viewer. Open one stream from the source, and fan it out in memory: 1. **One stream per domain at Mailingest,** carrying the notices of every mailbox on it, each tagged with its mailbox. One NATS subscription for all of them, and no periodic checks: the stream ends when the credential is revoked, or after an hour. 2. **core-api opens it once** (once per replica) and keeps a map from mailbox to the visitors watching it: ```ts const listeners = new Map>(); function onNotice(notice: { mailbox_id: string /* … */ }) { const set = listeners.get(notice.mailbox_id); if (!set) return; // nobody is looking at that mailbox: it costs nothing const frame = encode(`id: ${notice.id}\nevent: message\ndata: ${JSON.stringify(summary(notice))}\n\n`); for (const w of set) w.write(frame).catch(() => set.delete(w)); } ``` 3. **Make the visitors' connections cheap:** a plain SSE route instead of going through the RPC layer, a `: ping` comment every 25 seconds because Cloudflare closes idle connections at 100, and a higher `ulimit -n`, since every connection is a file descriptor. 4. **Recover by hand.** When a visitor reconnects, replay what they missed from a small ring buffer per mailbox, or ask Mailingest for the list. The cost toward Mailingest becomes fixed, whether ten visitors are watching or fifty thousand. It's maybe a hundred lines, and no new service. I wrote it all down and then didn't build it, because of the last step. Recovery, heartbeats, per-client limits and, as soon as there's a second replica, counters shared through Redis, all of it is code that a push server already has. ## Option two: a push server [Centrifugo](https://centrifugal.dev) is an open source realtime server: clients connect to it, it holds the connections, and it delivers what it's given to the channels they're subscribed to. Two of its features decided it: - **It consumes from JetStream itself.** Version 6 can read publications straight from a NATS JetStream stream, which Mailingest already runs. There's no HTTP call per message, and a Centrifugo that was down catches up from where its durable consumer left off. - **It recovers.** Each channel keeps a short history, and a client that reconnects with its last position gets what it missed. | | In-house fan-out | Centrifugo | | --- | --- | --- | | New infrastructure | none | one more service to run and watch | | Scale | what each replica holds | a cluster, through Redis | | Recovery on reconnect | written by hand | built in (channel history) | | Authorization | the backend's own | a signed token per connection | | Code to own | fan-out, pings, limits, recovery | sign a token, publish a message | The in-house version is the better choice for a small project, or anywhere one more service costs more than a hundred lines. Here the service was cheap to add, because the bus it reads from was already there. ## How it's wired When the router stores a temporary message, it publishes the message's public notice (sender, subject, a 240-character preview, the size) to a JetStream stream named `REALTIME`. The headers tell Centrifugo which channel it's for and how to recognise it if it comes twice: ```rust:crates/events/src/durable.rs pub async fn publish_realtime( ctx: &Context, channel: &str, publication_id: &str, data: Vec, ) -> anyhow::Result<()> { let mut headers = async_nats::HeaderMap::new(); headers.insert(REALTIME_CHANNELS_HEADER, channel); headers.insert(REALTIME_IDEMPOTENCY_HEADER, publication_id); publish_message( ctx, Subject::Realtime.as_str().to_string(), data.into(), Some(publication_id), Some(headers), ) .await } ``` The message's id does two jobs. It's the `Nats-Msg-Id`, so if the router retries a delivery, JetStream drops the second publish within its two-minute window. And it's Centrifugo's idempotency key, so a publication that reaches Centrifugo twice is delivered once. If publishing fails, the delivery doesn't: the failure is logged with ids only and counted in a metric. The row in Postgres is the record, and the live notice is a convenience on top of it. ``` router ── stored, published once ──▶ JetStream REALTIME │ durable consumer ▼ core-api ── realtime_token ──▶ API Centrifugo rt.mailingest.com │ (cookie's mailbox) │ temp: │ { url, token } │ uni_sse ▼ ▼ browser ─── token for one channel ──▶ EventSource ``` *After: Mailingest publishes each message once; Centrifugo holds the connections; core-api only signs a visitor in.* ## A token for one channel The browser connects to Centrifugo directly, at rt.mailingest.com, so it needs something Centrifugo trusts. core-api asks Mailingest for it on the visitor's behalf, for the mailbox in the visitor's signed cookie, and Mailingest signs a JWT with a secret only it and Centrifugo share (not the one its own sessions use, so a leak of one can't forge the other). The token names the channel `temp:`, and it expires when the mailbox does or in an hour, whichever is sooner. The channel list is the whole permission. Centrifugo's `temp` namespace allows nothing from the client: no subscribing to other channels, no publishing, no presence, no history on request. A visitor's token can receive notices for one mailbox and do nothing else, and when it expires Centrifugo closes the connection. A deleted mailbox stops receiving within the hour, without a single Postgres check. On the browser's side there's no SDK. Centrifugo's unidirectional SSE transport works with a plain `EventSource`, with the token and, on a reconnect, the last position it saw: ```ts:apps/web/src/lib/live.ts const request = { token: access.token, ...(position ? { subs: { [position.channel]: { recover: true, epoch: position.epoch, offset: position.offset } } } : {}), }; source = new EventSource(`${access.url}?cf_connect=${encodeURIComponent(JSON.stringify(request))}`); ``` Every frame is checked against a schema before it's used. A new token is fetched 30 seconds before the old one runs out. Reconnects back off, up to 30 seconds. And after any reconnect, the inbox re-reads the message list anyway and drops duplicates by id, because the history only covers the gap if the gap was short. ## What it cost The history is short on purpose: 50 publications per channel, for 15 minutes. A longer one was proposed, and turned down, because nothing about a temporary message should outlive it by much. A message deleted from the inbox stays in the channel's history until those 15 minutes pass, which is the one place a notice lingers. Centrifugo runs as one node with its in-memory engine. A second node means moving every node to the Redis engine, because with the in-memory one each node only sees its own channels. The other costs are the ordinary ones for a server that holds connections: a raised file descriptor limit in the container and on the host, pings every 25 seconds to keep Cloudflare from closing quiet connections, and no read timeout in Traefik shorter than that. In exchange, Mailingest lost code. The change that brought Centrifugo in deleted 901 lines and added 819, and what went includes both per-mailbox SSE endpoints, their Redis stream slots, the `Last-Event-ID` replay, the per-mailbox NATS notices and the revocation fan-out that existed only to end those streams. core-api lost its stream gauge and both its stream limits, and every inbox in every language lost the "too many streams" message. Mailingest's other per-account limits are where they were. None of them counts viewers any more. ## Counting the right thing The bug wasn't the number twenty. It was that a limit sat where it could only count accounts, and the thing growing was people. Any platform that lets customers serve their own users through one API key has the same blind spot. A per-account limit is fair to the customer and wrong for the customer's audience, and raising it per customer just moves the ceiling. Moving the connections to a layer built for viewers fixed that. Mailingest now does one thing per message, publish it once, and nothing per person watching.