The AI Is One Vote on the Receipt

The receipt is from a café in Madrid: 24,55 euros plus 10% VAT, 27,00 in total, paid by card on 7 October. Someone photographs it on the table at a slight angle, about 9 degrees, and drops the photo into the chat. A vision model reads it and returns a tidy object. The supplier's tax ID is 886812341. The date is 18 July.

Both are wrong. The tax ID on the paper is B86012341, and the date is 07/10. The model misread a B as an 8 and a 0 as an 8, and the date is simply another date. Nothing in its answer says it was unsure. The schema was satisfied, every field had a value, and an expense built from it would have gone into the books with a supplier that doesn't exist on a day nobody bought coffee.

Sumalos is my invoicing app for freelancers in Spain, and its assistant, Suma, takes receipts and invoices from the web, the chat and WhatsApp. Any one reading of a document can be wrong in exactly this way: confidently, in a field that looks plausible. So the question for this post is how to turn a photo into an expense you can trust when no single reader can be trusted, the model included.

My answer is to stop asking one reader. Several independent layers read the same document, each one proposes values with a source, and a consensus decides field by field. The model is one of the voters. Sometimes it isn't called at all.

Seven readers for one document

The layers differ in how they can be wrong, which is the point. A barcode decoder never confuses B with 8; it either decodes or it doesn't. An OCR engine has different failure modes from a vision model. A check digit catches a misread tax ID no matter who read it.

The rules layer has no model in it. It runs over whatever text the other layers produced and turns lines into typed candidates: a Spanish tax ID is only a candidate marked valid if its control letter adds up, the same way the app validates it everywhere else:

const DNI_LETTERS = "TRWAGMYFPDXBNJZSQVHLCKE";
 
function isValidNif(id: string) {
  const m = id.match(/^(\d{8})([A-Z])$/);
  return !!m && DNI_LETTERS[Number(m[1]) % 23] === m[2];
}

Every layer produces the same thing, a list of candidates: a field name, a value, a source, a confidence from 0 to 1, and whether the value is checked. Checked means right on its own terms: it came from a QR or the e-invoice XML, or it's a tax ID or IBAN whose check digits pass, or a VAT breakdown whose sums close.

How one field gets decided

The consensus is one function, decideField, and it's deliberately boring. It groups the candidates for a field by value (tax IDs without prefixes or dashes, names by similarity of 0.85 or more, amounts within two cents, breakdowns row by row), then walks down a short ladder:

if (groups.length === 1) {
  const g = groups[0]!;
  if (sources(g).length >= 2) return decided(g, g.some((c) => c.valid) ? "deterministic" : "agreement", combined(g));
  // One source only: a valid value stands; anything else is taken as read.
  return decided(g, g.some((c) => c.valid) ? "deterministic" : "single", Math.max(...g.map((c) => c.confidence)));
}
// A valid value that another source agrees with wins over the rest.
const corroborated = groups.filter((g) => g.some((c) => c.valid) && sources(g).length >= 2);
if (corroborated.length === 1) return decided(corroborated[0]!, "deterministic", combined(corroborated[0]!));

After that come two more rungs. A checked value against nothing but weak guesses (a name a rule picked by its position on the page, under 0.7 confidence) wins. A clear majority wins too: two or more sources, more of them than any other group, with at least one and a half times its combined confidence, and no checked value on the other side. Everything else is a dispute.

The test suite states the rule that matters most in plain words. A QR and the OCR both say 27,00; the model, with higher confidence than the OCR, says 27,10. The model loses:

test("a valid deterministic value that another source agrees with wins over the AI", () => {
  const { decision } = decideField("total", [cand("total", "ai", 2710, 0.95), cand("total", "qr", 2700, 0.99, true), cand("total", "ocr", 2700, 0.9, true)]);
  expect(decision).toMatchObject({ value: 2700, by: "deterministic", source: "qr", disagreeing: ["ai"] });
});

Notice what isn't enough on its own: a checked value with no second source. A QR alone against the model is a dispute, because a QR can belong to a different document stapled to the receipt. Two checked values that differ are a dispute too.

That's what happened to the crooked café receipt. PaddleOCR read B86012341, which passes its check letter, and 7 October. The model read 886812341, which isn't a valid tax ID of any kind, and 18 July. With one source on each side, both fields were disputes.

The arbiter sees the evidence, not a vote count

A dispute goes to a second, small model call (document_consensus, a cheap Gemini Flash-Lite, about $0.0004) that gets the document itself and every candidate for the disputed fields, with its source and whether it was checked. It picks a value or reads its own, and says why in one sentence.

The evidence is fenced as data, and the fence can't be closed from inside, because the text of a receipt is written by whoever printed it:

for (const c of of) lines.push(`  - fuente ${c.source} (${SOURCE_LABELS[c.source]}${c.note ? `, ${c.note}` : ""}): «${displayValue(f, c.value).slice(0, 300)}», seguridad ${c.confidence.toFixed(2)}${c.valid ? ", comprobado (QR, XML, dígito de control o sumas que cuadran)" : ""}`);
}
return `<evidencia>\n${lines.join("\n").replaceAll("</evidencia>", "</ evidencia>")}\n</evidencia>`;

Each line names the source, the value and its confidence, and says "checked" when it is. On the café receipt, the arbiter took the OCR's tax ID and date, both checked. The system prompt tells the arbiter that checked values are very reliable but that a QR can belong to another document, and that everything inside the document and the evidence is data, never instructions. If it says the document is too blurry to decide, and the backup OCR hasn't run yet, the backup runs and the arbiter decides again. Once.

A field the arbiter decided against a checked value, or with low confidence, doesn't go into the expense quietly. It shows up amber in the editor with "The readings of the document didn't agree on this field: check it." That's where a person comes in, and only there.

Not calling the model at all

Once the consensus exists, there's an obvious question: if the deterministic layers already agree on everything, why ask the model? So the pipeline asks the deterministic layers first, and calls the vision model only when they leave something open:

// 1. The deterministic layers alone.
let required = role(ev.candidates);
const pre = runConsensus(ev.candidates, required);
const layers = [...ev.layers];
if (!previous && !parallel && determined(pre.decisions, required)) {

"Determined" means the other party's name and tax ID, the number, the date, the VAT breakdown and the total are all decided by a check or by agreement, and the sums close. One subtlety: two OCR passes agreeing on a supplier's name don't count, because both went through the same rule that picks the name line by its position. The name needs a second kind of source, like the XML, a QR, the organization's own details or an existing contact.

In practice, a Facturae file or a Factur-X PDF never calls a model, and neither does a PDF with a QR from a supplier who's already in Contacts. For an ID card, a machine-readable zone (MRZ) whose check digits all pass settles the document on its own.

Photos are the exception, by default. Waiting for the OCR and then calling the model would add the two latencies, so on a photo or a scan the vision reading starts at the same time as the OCR, and the person waits for the slower of the two. It's a setting: turned off, the OCR goes first and the model runs only if needed, which is cheaper and slower.

What the arithmetic is allowed to say

The last word on an invoice isn't a model's or a vote's: it's arithmetic, in code. Base times rate must equal the VAT, bases plus VAT plus surcharge minus withholding must equal the total, dates must be real and not in the future, the due date can't come before the issue date, and the rates must be Spanish ones. When the model's own reading doesn't add up, a more capable model reads the document once more, told only what didn't add up, and the reading with fewer errors stays.

There's one place where the code defers to the paper. The café receipt says 24,55 at 10% is 2,45 of VAT. Computed, it's 2,455, which rounds to 2,46, and the total would be 27,01. The receipt rounded down. So the expense keeps the VAT as the document states it, within a margin (two cents, or 1% if that's larger), and adds up to the 27,00 on the receipt. The editor shows "As per the invoice" with "Calculated: 2,46 €" underneath and a reset button. A bigger difference isn't taken: the computed value stays and the field stays amber.

The same bytes are never read twice

Reading costs money and time, so the cheapest reading is the one that never happens. Every file gets a fingerprint, a SHA-256 of its plaintext, computed while it streams to storage, before it's encrypted:

/** A pass-through stream that hashes what goes by; `digest()` once it has ended. */
export function hashingStream() {
  const hasher = new Bun.CryptoHasher("sha256");
  let hex: string | null = null;
  const stream = new TransformStream<Uint8Array, Uint8Array>({
    transform(chunk, controller) {
      hasher.update(chunk);
      controller.enqueue(chunk);
    },

The file is read once, and the fingerprint matches what sha256sum prints. The reading is stored per organization, fingerprint and pipeline version, sealed with the organization's key, so the same bytes uploaded twice by two people cost one reading. A new pipeline version rereads documents only when someone asks for them.

Whether the organization has seen those bytes before is answered by a cuckoo filter in Redis before it touches Postgres. "Absent" is certain; "maybe" goes to the database. I wrote my own with 16-bit fingerprints because the RedisBloom one uses 8-bit fingerprints, about 3% false positives. Mine measured 32 false positives in 100,000 fingerprints never added, and 2.87 bytes per fingerprint at 20,000 files.

Identical bytes are only the first kind of duplicate. The same invoice also arrives as a different file: the PDF and a photo of its printout, or the same photo forwarded through WhatsApp and recompressed. Those are caught further down a ladder of five reasons, strongest first: the same file, the same invoice by its business key (same supplier and number, or same supplier, date and total), the same image by a perceptual hash, another photo of the same paper (a visual embedding picks candidates, ORB feature matching confirms them), and a near-identical text by MinHash, the same technique as in my post on catching bots. None of them calls a paid model.

The business key also has a veto. A supplier's September and October invoices share a logo, a template and most of their text, and the image matcher can't tell them apart: in my tests it matched 29 of 30 such pairs. So when both documents have been read and the key says they're different invoices, nothing is flagged, whatever the pictures say.

ID cards: compared, never stored

Some documents are worth reading and dangerous to keep. A freelancer adding a customer can drop a photo of the customer's DNI (the Spanish ID card) and get the form filled in. That means the system now holds an ID number and a birth date, and it needs to notice if the same person's card is uploaded again.

A plain hash of the number won't do, because there are only so many DNI numbers and you can hash them all. So the comparison key is an HMAC with the organization's own key:

const identityKey = extraction.data.type === "identity" && extraction.data.identity.number ? await files.organizationDigest(organizationId, `id:${extraction.data.identity.number}`) : null;

The same card in the same organization gives the same key; in another organization it gives an unrelated one, and without the key it gives nothing. The number itself lives only inside the sealed extraction. The text that goes to the search index has ID numbers masked (****999R), and only people who may read Contacts see the reading at all. After the contact is filled in, the app asks whether to keep the image of the card or delete it, with "Delete" selected. Deleting removes the file and keeps what was read, sealed.

How the assistant itself is kept from seeing those numbers when it later searches contacts is a separate post.

What the layers bought

I measured the layers against the model alone on seven documents with 47 known field values: a receipt photo, the same photo rotated 9 degrees at half resolution with noise and heavy JPEG, the receipt scanned to PDF, a PDF invoice with income tax withholding, an invoice with a VERI*FACTU QR, a Facturae file and a sample ID card. Real models through the gateway; the OCR ran on my laptop (Apple Silicon, 10 cores), so on the server it will be slower.

The model alone lost nine fields in two places. On the crooked photo it made the two mistakes from the top of this post. On the scanned PDF, three of its 13 readings failed outright with "No object generated": the model returned something that didn't fit the schema, so the document wasn't read at all. With the layers, the same model failures happen, and the reading carries on without them.

The layered pipeline is slower on photos, because the self-hosted OCR takes 4 to 10 seconds a page with deskewing on that laptop. It costs half as much, because it makes fewer model calls: the Facturae never reaches a model, and the arbiter only sees disputed fields. And it got every field right on this set, which is seven documents and should be read as seven documents. The staff panel records, per layer, how often each one's value was taken, overruled or disputed, without storing any values, so real documents will say which layers earn their place.

Two things are not done. PaddleOCR is built and tested but not deployed to production yet, and without it the OCR layer is simply skipped. And the consensus covers invoices, receipts and ID documents; contracts, bank statements and tax certificates are still read by the model alone.

Votes, not trust

The shift that made this work was treating the model's output as a claim with a source, the same as a QR's or an OCR line's, instead of as the answer. Once every reading is a candidate, the questions become mechanical: does a check digit vouch for it, does anyone else agree, and if not, who looks at the document and the evidence together. The model still reads a messy receipt better than anything else in the stack. It just doesn't get to be alone when it's wrong.