Every File in S3 Gets Its Own Key

The files in Sumalos are the worst kind to lose. People photograph their ID card so the assistant can fill in a contact. They forward supplier invoices with bank details on them, upload signed quotes, and keep the private key of the certificate their business uses with the tax agency. All of it lives in an S3 bucket at Tigris, on machines I don't run.

Tigris encrypts at rest, like every serious object store. That protects against a disk walking out of a data center, and nothing else: anyone holding the access key reads every object in plain text, and so does anyone who finds a bucket policy set wrong. And the access key lives in the app's environment, next to everything else.

So the question I wanted to answer: how do you make a leaked bucket useless, and still be able to change the master key later without rewriting a single object?

The answer is an old one, envelope encryption, and this post is about how it looks in a few hundred lines of TypeScript, including the parts that are still loose.

Three keys deep

Every file gets a fresh 256-bit key the moment it is stored. That key encrypts the bytes. It is then encrypted itself, "wrapped", by a key that belongs to the file's organization, and the organization's key is wrapped by a master key that never touches a file. The comment at the top of the module draws it in one line:

/**
 * Files in S3 (Tigris), encrypted before they leave the process. Each file has its own
 * key; that key is wrapped by its organization's key, and that one by the master key:
 *
 *   MASTER_KEYS ─wraps→ organization_keys.key ─wraps→ files.key ─encrypts→ S3 object, files.name
 */

The split is the point. The bucket holds only ciphertext. The database holds only wrapped keys. The master keys are in the process's environment and nowhere else. A leak of any one of the three gives nothing away; it takes the database and the master together to open a file, and with both of those an attacker already owns the app.

Wrapping is a few lines. The same function seals a key, a file name, or the certificate's private key:

/**
 * AES-256-GCM over a small value (a key, a file name): nonce ‖ ciphertext ‖ tag.
 * `aad` binds it to where it belongs (e.g. `file:<id>`), so a sealed value copied
 * onto another row does not open.
 */
export function seal(key: Uint8Array, plaintext: Uint8Array | string, aad: string): Uint8Array<ArrayBuffer> {
  const nonce = randomBytes(NONCE);
  const cipher = createCipheriv("aes-256-gcm", key, nonce).setAAD(Buffer.from(aad));
  const body = cipher.update(typeof plaintext === "string" ? Buffer.from(plaintext) : plaintext);
  return new Uint8Array(Buffer.concat([nonce, body, cipher.final(), cipher.getAuthTag()]));
}

The aad argument is the detail I'd copy into any project. AES-GCM authenticates "additional data" that isn't encrypted, and here that data is the row the value belongs to: file:<id> for a file key, name:<id> for its name, organization:<id> for an organization's key. Someone with write access to the database can't take a file key from a row they can read and paste it onto a row they can't, or move a sealed name between files: the tag no longer matches and unseal throws. A test checks exactly that, sealing factura.pdf under name:a and failing to open it under name:b.

A file larger than memory

A whole-file seal would be fine for a 200 KB receipt. It isn't fine for whatever the server generates later (an export, a year of PDFs zipped), and GCM over one huge buffer means holding it all in memory and getting one tag at the very end. So file bodies use a streaming format instead: the plaintext is cut into 64 KiB chunks and each chunk is sealed on its own.

This is the STREAM construction from Hoang, Reyhanitabar, Rogaway and Vizár, the same idea behind Tink's streaming AEAD. The nonce is what makes it safe to cut a file up. Because each chunk's nonce contains its position, chunks can't be swapped. Because it contains a flag for the last chunk, a file can't be cut short at a chunk boundary and passed off as complete. The encrypting side has to hold back one chunk to know which is last:

transform(chunk, controller) {
  size += chunk.length;
  pending = Buffer.concat([pending, chunk]);
  // A full chunk goes out only once more data follows: the last one must be flagged as such.
  while (pending.length > CHUNK) {
    controller.enqueue(sealChunk(key, prefix, index++, false, pending.subarray(0, CHUNK)));
    pending = pending.subarray(CHUNK);
  }
},
flush(controller) {
  controller.enqueue(sealChunk(key, prefix, index, true, pending));
},

The tests attack exactly those properties: a flipped bit, two swapped chunks, a file truncated at a chunk boundary and decrypted with the shorter size it claims to have, the wrong key. All four throw. The round-trip test runs sizes chosen to sit on the edges (0 bytes, 1, one chunk minus one, one chunk exactly, one plus one, three chunks plus 17), fed in uneven 10,007-byte pieces the way a network delivers them.

The cost is small and easy to work out: 8 bytes per file and 16 per 64 KiB chunk, about 0.02% on anything bigger than a few chunks. A 4-byte index caps a file at 2³² chunks, which is 256 TiB. Neither number bothers me.

Uploads stream straight through. The body goes through a hashing stream (for the file's fingerprint, so duplicates can be found without keeping a second copy), then the encrypting stream, then into a multipart writer:

for await (const chunk of (body instanceof Blob ? body.stream() : body).pipeThrough(hash).pipeThrough(encrypt)) await writer.write(chunk);

The writer sends 8 MiB parts with four in flight, so at most about 32 MiB of a file is in memory while it uploads, whatever its size. Users can upload up to 20 MB; what the server generates has no limit.

Reading a byte range

Fixed-size chunks pay off on the way out. A PDF viewer or a <video> asks for byte ranges, and with a single GCM tag over the whole file the server would have to fetch and decrypt everything just to authenticate the first page. With chunks, sealedRange turns a plaintext range into the sealed bytes that cover it:

export function sealedRange(size: number, start: number, end: number) {
  const first = Math.floor(start / CHUNK);
  const last = Math.floor(end / CHUNK);
  const from = HEADER + first * SEALED_CHUNK;
  const to = Math.min(HEADER + (last + 1) * SEALED_CHUNK, sealedSize(size));
  return { from, to, first, skip: start - first * CHUNK, length: end - start + 1 };
}

The route fetches the 8-byte header separately for the nonce prefix, then the chunks, decrypts each one (authenticating it with its own tag), and a slicing stream drops the bytes before the start and after the end. The test for this runs six ranges over a file of three chunks plus 17 bytes, including a single byte, a range across a chunk boundary, and the last five bytes, and compares each with the original.

Rotating the master

Here is where the envelope earns its name. The master keys come from one environment variable, a comma-separated list of <id>:<64 hex> entries, the first one current. Every wrapped organization key starts with the id of the master that wrapped it:

wrap(key: Uint8Array, aad: string): Uint8Array<ArrayBuffer> {
  const id = Buffer.from(current);
  return new Uint8Array(Buffer.concat([Buffer.from([id.length]), id, seal(keys.get(current)!, key, aad)]));
},
unwrap(wrapped: Uint8Array, aad: string): Uint8Array {
  const id = Buffer.from(wrapped.subarray(1, 1 + wrapped[0]!)).toString();
  const master = keys.get(id);
  if (!master) throw new Error(`master key ${id} is not in MASTER_KEYS`);
  return unseal(master, wrapped.subarray(1 + wrapped[0]!), aad);
},

So a rotation is: put a new key first in the list and restart. New organizations get wrapped under the new key at once. Old ones still open, because their wrapped key names the old master and the old master is still in the list. To retire the old master, each organization's key is unwrapped with it and wrapped again under the new one, one small row per organization, and then the old entry comes out of the list.

The file keys don't change, because they were never wrapped by the master. The S3 objects don't change, because they were never encrypted by anything but their own file key. A bucket with a million objects rotates by rewriting as many rows as there are organizations. The keyring test does the first half of this: it wraps a file key under k1, builds a ring of k2,k1, and opens it; then it builds a ring with only k2 and checks that the error names k1, so a missing master says which one is missing.

The honest part: the rewrap job isn't written yet. The keyring supports two masters at once and the test proves old wraps stay open, but walking organization_keys and rewrapping is still a step on paper. And rotation has a second catch I found while writing this post. A few other secrets are derived from the master variable by hashing the whole string with a purpose label: the key of the vault that holds the assistant's pseudonyms, the key that signs calls between the WhatsApp agent and the API, the key for WhatsApp link codes:

export function pseudonymKey(masterKeys = process.env.MASTER_KEYS): Uint8Array {
  return masterKeys ? new Uint8Array(createHmac("sha256", masterKeys).update(PSEUDONYM_PURPOSE).digest()) : new Uint8Array(randomBytes(32));
}

Adding k2 to the front of the list changes that string, so every derived key changes with it. For the request signing and the ten-minute link codes that's harmless. For the pseudonym vault it means every sealed value in it stops opening and every pseudonym changes. Files are safe across a rotation; those derived keys aren't, until they derive from one named master instead of the whole list. That fix is small, and it has to land before the first real rotation.

Deleting by forgetting

Envelope encryption also changes what deleting means. When an organization is deleted, its row in organization_keys goes with it by cascade, and with it the only way to unwrap any of its file keys. The objects in the bucket become unreadable at that moment, before a single one is removed; the delete that follows only frees the space:

/**
 * Deletes every object of an organization that is being deleted. Its rows and key go
 * with the organization (cascade), which already leaves the objects unreadable; this
 * frees the space. Returns how many objects went.
 */

That also covers the copies I don't control: a bucket snapshot, a replica in another region, a backup someone made. Without the organization's key they are noise. It's the same for one file: removing it deletes the object first and the row after, so a failure leaves a row to retry from, never an orphan key.

The organization key does other small jobs once it exists. The tax certificate's private key is sealed with it, so the database alone never holds it in clear. ID card numbers are compared through an HMAC under it, so two scans of the same card can be matched without storing the number, and nobody can reverse the digests with a list of every possible DNI.

Handing a file to a browser

None of this helps if the decrypted file then leaks through the web. Files are never public URLs. The app hands out a signed link, /api/files/c/<id>.<expires>.<hmac>, that opens the file for ten minutes, with a twist: the expiry is rounded up to the next five-minute step, so every link made within one step is the same URL and the browser's cache keeps working when the page signs it again. In practice a link lives between ten and fifteen minutes.

The response carries two headers that matter more than they look:

"content-security-policy": "sandbox",
"x-content-type-options": "nosniff",
"cache-control": `private, max-age=${Math.max(0, link.expires - Math.floor(Date.now() / 1000))}`,

Files come from users, and the app serves them from its own origin. An uploaded SVG or an XML file with a script in it, opened inline, would run on the app's origin with the user's session. sandbox gives the document an opaque origin with scripts off, and nosniff stops the browser from deciding that a file labelled as an image is really HTML. Permissions are decided before a link is ever signed; the link itself is only a short-lived bearer ticket, signed with a secret of its own rather than one derived from the master.

The uploads nobody finishes

A multipart upload that dies halfway (a crash, a deploy, a phone that loses signal) leaves parts in the bucket. They don't show up in a listing of objects, but they are stored and billed. On AWS a lifecycle rule, AbortIncompleteMultipartUpload, cleans them up. Tigris's lifecycle rules cover transitions and expiration, and that rule isn't among them, so the worker does it once an hour: it lists the open uploads and aborts any started more than 24 hours ago. Bun's S3 client can't list multipart uploads, so those two calls are signed by hand with aws4fetch:

for (const [, upload] of xml.matchAll(/<Upload>([\s\S]*?)<\/Upload>/g)) {
  const key = unescapeXml(tag(upload!, "Key") ?? "");
  const uploadId = unescapeXml(tag(upload!, "UploadId") ?? "");
  const initiated = Date.parse(tag(upload!, "Initiated") ?? "");
  if (!key || !uploadId || !(initiated < cutoff)) continue;
  const path = key.split("/").map(encodeURIComponent).join("/");
  const abort = await aws.fetch(`${bucket}/${path}?uploadId=${encodeURIComponent(uploadId)}`, { method: "DELETE" });
  if (!abort.ok && abort.status !== 404) throw new Error(`AbortMultipartUpload ${key}: ${abort.status}`);
  aborted++;
}

The parts are ciphertext too, so this is about the bill, not about secrecy. A sibling job does the same for whole files nobody points at: documents dropped into a conversation that was never saved, deleted after a day's grace.

What the bucket still says

Encryption hides contents, not shapes, and it's worth being exact about what a leaked bucket still reveals. Object keys are <organization>/<kind>/<id>, so a listing shows how many files each organization has and of what kind (document, image, thumbnail). The ids are UUIDv7, which carry their creation time. Sealed sizes give the plaintext size exactly, since the overhead is a formula. Names are sealed, but the database keeps the MIME type and the SHA-256 of each file's plaintext in clear, for finding duplicates; someone with a database dump and a copy of a specific file could confirm it's stored.

For an invoicing app I can live with all of that. Nobody learns what an invoice says, whose ID card it is, or which bank account is on it. They learn that an organization uploaded 40 documents in March.

What I like about the design is how little it asks of the rest of the code. Nothing outside files/ ever sees a key; callers store a stream and read a stream. The bucket can be copied, mirrored, backed up or leaked, and none of it matters without one row in Postgres and one variable in the environment. And the next time I need to change that variable, I'll rewrite a few hundred rows, after fixing the derived keys, and leave the bucket alone.