A Bucket That Can Say No

S3 launched in March 2006, so "put your data in a bucket" is not new advice. What is new is the number of systems that now call a bucket their system of record rather than their backup. turbopuffer says object storage is the source of truth and that it has no other dependency in the critical path. Chroma shipped a write-ahead log that keeps nothing on local disk. LanceDB tells you concurrent writers are safe with no commit coordinator. Apache Kafka voted in March 2026 to build a topic type whose durable storage is a bucket.

All of that arrived in the last two years, on a storage service that has barely changed. So the question worth answering is not whether object storage is cheap, which everybody already knew in 2006. It is what a bucket could suddenly do in 2024 that it could not do in 2023.

The answer is small enough to fit in an HTTP header.

What a bucket could not do#

Object storage gives you a key and some bytes, and a PUT replaces whatever was there. That is fine for photos and terrible for truth. Suppose two writers both read the current manifest of a table, both append their own data files, and both write the manifest back. Each PUT succeeds. The second one silently erases the first writer's work, and nothing anywhere reports an error. The bucket has no opinion about which version you thought you were replacing.

Every system that wanted a commit log on S3 hit that wall, and they all solved it the same way: rent the atomicity from somewhere else. Delta Lake, Iceberg and Lance on S3 all used a DynamoDB lock manager. The bucket stored the bytes and a real database stored the answer to which bytes were current.

That sidecar is the thing the whole architecture was supposed to avoid. You moved your data to object storage for durability you don't operate, then bolted on a stateful service you do operate, in the one place where its failure takes the whole log down with it.

The header that changed it#

AWS shipped conditional writes on 20 August 2024: If-None-Match on a PUT, which writes the object only if that key does not exist yet. Three months later, on 25 November 2024, came the other half: If-Match against an ETag, which writes only if the object is still the exact version you read.

Both are one flag on a request you were already making.

# write it only if nobody has claimed this key yet
aws s3api put-object --bucket log --key manifest.json \
  --if-none-match "*" --body manifest.json
 
# replace it only if it is still the version we read
aws s3api put-object --bucket log --key manifest.json \
  --if-match '"a3f19c8e5b2d4071"' --body manifest.json

The second command is compare-and-swap, the primitive every lock-free data structure is built on, except the compare happens in S3 and the failure comes back as a 412. The bucket can now refuse a write. That single ability is what the lock manager was there to provide.

A 1995 algorithm with a new instruction#

The clearest thing anyone built on it is Chroma's wal3, a write-ahead log that lives entirely in object storage. Their description of the mechanism is worth reading twice, because of how ordinary it sounds:

That is a lock-free linked list, and the paper they point at is John D. Valois, Lock-free linked lists using compare-and-swap, from PODC 1995. The algorithm was thirty years old when they shipped it. Nothing about the theory needed to be invented. What was missing until August 2024 was an implementation of compare-and-swap that a bucket would honour, and once that existed the textbook applied unchanged, with a PUT where the CPU instruction used to be.

The practical result was fast. Chroma took wal3 from prototype to production traffic in four months, and open-sourced it, so the claim is checkable: the first commits land in the repository in early March 2025. LanceDB did the smaller version of the same thing, deleting its DynamoDB dependency in March 2025 and cutting a manifest commit from three I/O operations to one.

The shape everyone lands on#

Read three of these systems and you find the same four decisions, arrived at separately.

• Object storage is the system of record. Not a cold tier under a real database: turbopuffer is explicit that this is not tiering, where cold data eventually drains to S3.
• Files are immutable. Nothing is edited in place, because a PUT replaces a whole object. Compaction rewrites; it never patches.
• A tenant is a prefix. turbopuffer calls a namespace a prefix; Chroma gives each collection its own log at its own path. Isolation costs a string concatenation instead of a cluster.
• The serving nodes are caches. They hold local NVMe copies of things the bucket already has, so losing one costs latency and no data. Neon says it plainly: the pageservers are just a cache of what is stored in the object storage.

The older route to the same economics was tiering, and Elastic's frozen tier shipped it back in March 2021: searchable snapshots in S3, read through a local cache sized at roughly a tenth of the data, with storage savings the blog puts at up to 90 percent against the hot tier. It works, and it is still a tier. The 2024 systems are making a stronger claim, which is that the bucket is the database and the disks are scratch space.

I built a small version of one of these to understand it, which is its own story, about how you avoid dragging vectors across the network on every query. This post is about the layer underneath that: why the bucket is allowed to be the source of truth at all.

Kafka voted for it#

The direction stopped being a vendor argument on 2 March 2026, when Apache Kafka accepted KIP-1150, Diskless Topics, with nine binding votes in favour. The proposal is for topics that write through to object storage and use local disks for caching instead of durable storage.

Two things keep that in proportion. It is a motivational KIP, which the vote result says out loud: it establishes consensus on the direction without prescribing an implementation. And no code has shipped, because the implementation lives in two sub-proposals that are still under discussion. What was ratified is the idea, by a project with no product to sell.

Where it stops#

Every source I read concedes the same limit, and it is the write path.

turbopuffer's own tradeoffs page says writes take up to 200 milliseconds to commit, and that consistent reads have a floor around 10 milliseconds. Its design writeup is just as direct about who that rules out: relational databases are the row for transactions and CRUD at sub-millisecond latency, while a search engine's write workload is closer to a data warehouse, high throughput with no transactions and relaxed write latency. KIP-1150 makes the matching concession, that diskless topics will incur higher produce latency and classic topics remain appropriate for low-latency uses.

Which is why the systems that do serve OLTP quietly keep object storage off the commit path. Neon is the instructive case, because from a distance it looks like the same architecture. A Postgres transaction there is committed once a Paxos quorum of separate safekeeper nodes has acknowledged the WAL record, and Neon's docs state that queries do not read from object storage at all. S3 provides durability and cost, and nothing on the hot path.

So "separation of storage and compute" and "object-storage-native" are different claims, and collapsing them misreads all four rows of that table.

The second limit is the one the primitive itself imposes. Compare-and-swap is optimistic, so contention is the normal operating condition of a busy log, and every writer that loses a race has to read the manifest again and retry. wal3's error enum is the most honest description of that cost I have found in any of these codebases:

#[error("log contention, but your data is durable")]
LogContentionDurable,
#[error("log contention, and your operation may be retried")]
LogContentionRetry,
#[error("log contention, and your data may or may not be durable")]
LogContentionFailure,

Three flavours of losing a race, and the third one is an outcome where the system cannot tell you whether your write survived. That is the true ceiling on this architecture, and nobody has published where it sits: I could not find a single number for the write rate at which compare-and-swap on one key stops scaling. LanceDB's docs get closest, warning that too many concurrent writers can fail because a writer only retries a commit so many times.

The numbers I could not check#

I ran this research as a verification pass, where each extracted claim faced three independent attempts to refute it and needed a majority to survive. Twenty-five claims went in and seven died, and the pattern in which ones died is the most useful thing I learned.

Nothing architectural failed. The conditional-write timeline, wal3's design, Neon's commit path, KIP-1150's status: all confirmed against primary sources. What failed was almost entirely the arithmetic. Latency percentiles attributed to systems that never published them. A widely repeated claim that S3 Express One Zone storage costs eight times S3 Standard. The tidy story that diskless Kafka exists to dodge cross-availability-zone network charges, which the KIP itself does not say.

And the cost figures that did survive all carry the same asterisk. turbopuffer's storage ladder, the one that puts its design at 70 dollars per terabyte per month against 3,600 for a RAM and SSD architecture, is a real published table, but it is cost of goods rather than list price, it assumes half the data sits in cache, and it counts only stored gigabytes. It excludes request costs entirely, and requests are exactly what a write-heavy workload pays for. Elastic's 90 percent has the same shape, excluding the GET charges that fill its cache.

So the engineering record here is good and the economic record is thin. Every system in this post batches, group-commits and compacts specifically to control request costs, which tells you the pressure is real, and not one of them publishes the resulting blended cost per gigabyte written. If you are choosing an architecture on the cost argument, you are choosing on vendor models that omit the line item their own design is shaped around.

What one header bought#

Object storage did not get faster. It got the ability to refuse, and refusal turns out to be the whole difference between a place to keep bytes and a place to keep truth. Once a PUT can fail because someone else moved first, a bucket becomes a coordination primitive, a thirty-year-old algorithm finally has somewhere to run, and the stateful sidecar you kept around to sequence writes stops being necessary.

What you pay is a commit measured in hundreds of milliseconds, a contention ceiling nobody has mapped, and a bill that no published model fully accounts for. That is a good trade for a search index and a bad one for a checkout, which is roughly what everyone building these systems already tells you, in the small print under the cost table.