---
title: A Model That Decides for You
date: '2026-10-15'
description: 'Most model calls in a product are multiple-choice questions. A decision model answers in one pass, with a probability. I ran one on a four-core VPS.'
author: Pablo Fernandez
tags:
  - AI
  - Decision Models
  - Laya
  - Self-Hosting
  - Performance
source: 'https://www.pablofr.com/blog/a-model-that-decides-for-you'
---
Look at the language model calls in any product and count how many of them want prose back. In mine, almost none. "Which team should get this email?" "Is this a refund request?" "Spam or a customer?" Each one is a multiple-choice question, and I was paying a model to write a little essay, then parsing a word out of it and hoping the word was one of the ones I'd allowed.

The part I never had was the one I wanted most: how sure it was. A chat model will say "billing" with the same flat confidence whether the email is a double charge or a quote request that merely mentions money. If a system is going to act on its own, it needs to know which answers are safe to act on and which ones should go to a person.

So the question for this post: can a small model that never writes a word make those decisions, tell me how much to trust each one, and do it fast enough on a cheap server to sit inside a request?

## A question with a fixed set of answers

The idea has a name now. [TypeSafe](https://typesafe.ai) calls it a [System One model](https://docs.typesafe.ai/concepts/system-one), after the fast, intuitive half of [Kahneman's two systems](https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow). You send a state (an email, a ticket, a JSON blob) and a list of typed questions, and you get back typed answers with probabilities. Three types cover nearly everything:

- **choice:** pick one option from a list you define, with a probability for each option.
- **score:** place the state on an ordered scale ("can wait", "today", "right now").
- **noul:** a yes or no, returned as the probability of yes.

There is no text generation, so there is nothing to parse and no answer outside the set you allowed. All questions are answered in one request.

```
  chat model
  ┌───────────────┐    ┌─────────────────────┐    ┌─────────────────┐
  │ prompt + JSON │───▶│ generate tokens ... │───▶│ parse "billing" │
  └───────────────┘    └─────────────────────┘    └─────────────────┘

  decision model
  ┌───────────────┐    ┌─────────────────────┐     billing    0.95
  │ state +       │───▶│ one forward pass    │───▶ support    0.02
  │ questions     │    │ per option          │     sales      0.01
  └───────────────┘    └─────────────────────┘     ...
```

*A chat model writes an answer that you parse; a decision model returns the answer and how likely it is.*

TypeSafe's own model, [Jev](https://typesafe.ai), is closed and in early access; its speed and price claims are theirs to prove. What made this practical for me is [Laya](https://github.com/NandhaKishorM/laya), an open alternative published in September 2026 under Apache-2.0 (it has plenty of company now: I went through [the alternatives to Jev](/blog/alternatives-to-jev) separately). It ships two checkpoints that matter here: an English one built on [ModernBERT-large](https://huggingface.co/answerdotai/ModernBERT-large) (421M parameters) and a multilingual one built on [mmBERT](https://huggingface.co/jhu-clsp/mmBERT-base) (322M), with a router that picks one by language. Both are encoders, the kind of model that reads a text and scores it, which is exactly what a multiple-choice question needs.

## Try it: a support inbox

The demo below is an email to a made-up company's support address. Laya decides which team gets it and answers three yes/no questions about it, and the panel shows every probability it returned. The pill at the top is the part I care about: at 0.8 or more the email could be routed with nobody looking; below that, a real system would hand it to a person.

<EmailTriageDemo />

Try "A quote" and "Thank you". The quote goes to Sales, but with roughly 0.6, not 0.99, and the thank-you note lands in Other at about the same level. That uncertainty is the useful output: those are the emails a person should glance at.

The questions live on the server, not in the browser. Your email goes to this site's API, which checks a [Turnstile](https://developers.cloudflare.com/turnstile/) token, applies a few limits, and asks Laya over the server's internal network with a key the browser never sees. More on those limits further down.

## Where the time goes

My first version of this demo took 3.4 seconds to answer an English email. That was the moment I learned how these models spend their time, and it isn't how a chat model spends it.

Laya scores each option of a question separately against the email. A question with five teams is five passes over the text; a yes/no is one. And every pass reads the option's description too, so a long description makes every pass slower. My first question set asked for the team twice (once with the options reversed, to cancel the model's habit of favouring the first options it reads), described each team with a sentence of what it is and what it isn't, and added a three-step score and four yes/no questions. Seventeen passes, most of them long.

Measured on the server, same email, model time only:

| Question set | Passes | Threads | English | Spanish |
|---|---|---|---|---|
| Team asked twice, long descriptions, urgency score, 4 yes/no | 17 | 2 | 3,345 ms | 1,303 ms |
| Team once, long descriptions, urgency score, 4 yes/no | 12 | 2 | 2,578 ms | 1,133 ms |
| Team once, short exclusions, 3 yes/no | 8 | 4 | **662 ms** | **288 ms** |

Two things got it under a second. The first was cutting passes instead of words: one short exclusion per team where the model was wrong ("a charge that already happened, not a price quote" is what finally moved quote requests out of Billing), a yes/no for urgency instead of a three-step scale, and no refund question, since a refund is already what the Billing option describes. The second was giving the worker all four cores at a quarter of the normal CPU weight, so the web server and Postgres still win whenever they want the CPU, and Laya gets everything when the server is idle, which is most of the time. The same question sets ran about 1.8 times slower on two threads.

I also tried `torch.compile`, which Laya supports out of the box. It made English answers 21% faster and the worker took 195 seconds to start instead of 9, and it moved some probabilities by up to 0.04. I kept the faster startup.

## Half the memory, the same answers

Both checkpoints sit in memory all the time, and in their original 32-bit floats they took 3.27 GB on a server with 7.6 GB that also runs Postgres, a search engine and this site. The obvious fix is fewer bits per weight. The thing to protect is the probabilities, because a model whose 0.9 no longer means 90% has lost the one thing it's for.

I measured each option against the original, one checkpoint per process, on 8 English emails for the English checkpoint and those plus 8 Spanish ones for the multilingual one, 4 questions each, comparing every probability:

| Variant | English: memory | English: latency | English: mean change | Multilingual: mean change | Largest change | Answers changed |
|---|---|---|---|---|---|---|
| fp32 (original) | 1.69 GB | 1,357 ms | | | | |
| **bf16** | **1.15 GB** | **628 ms** | **0.0018** | **0.0029** | 0.047 | **0 of 24** |
| int8, PyTorch dynamic | 0.91 GB | 359 ms | 0.18 | 0.10 | 0.82 | 1 of 24 |
| int8 weights only, embeddings too | 0.83 GB | 775 ms | 0.0080 | 0.0056 | 0.157 | 1 of 24 |

With both checkpoints in one worker, bf16 took the whole thing from 3.27 GB to about 2.0 GB.

[bfloat16](https://en.wikipedia.org/wiki/Bfloat16_floating-point_format) keeps the 8-bit exponent of a 32-bit float and drops most of the mantissa, so it halves the weights without changing their range. The answers didn't move, and on this CPU, which has native bf16 instructions, it also ran about twice as fast.

The int8 rows are the cautionary ones. PyTorch's dynamic quantization also squeezes each layer's *inputs* into 8 bits with one scale per tensor, and transformer encoders have a few activation values far larger than the rest; one scale can't hold both, and the probabilities fell apart. Keeping inputs in bf16 and only storing weights in 8 bits fixed most of it, but it was still two to three times further from the original than bf16 and flipped an answer. For a model whose product is its probabilities, I stopped at bf16.

### The peak nobody measures

The steady number wasn't the whole story. Every time the worker started, memory jumped to 3.25 GB for a few seconds and the server reached for swap. Laya builds the model with empty 32-bit parameters, reads the whole 32-bit weights file, and copies one into the other, so two full copies exist at once. Converting to bf16 afterwards comes too late.

```
  Laya alone                          the worker
  ┌────────────────────────────┐      ┌────────────────────────────┐
  │ build model: fp32 copy     │      │ build model: never written │
  │ read file:   fp32 copy     │      │ read file in 16 MB chunks  │
  │ copy, then convert         │      │ each chunk ─▶ bf16         │
  └────────────────────────────┘      │ tensors become the params  │
                                      └────────────────────────────┘
       worker peak 3.25 GB                 worker peak 2.24 GB
```

*Loading the checkpoints: Laya's path holds two fp32 copies of a model at once; the worker's path never holds more than the bf16 model and one 16 MB chunk.*

The fix lives in the worker, not in Laya. While a checkpoint loads, it reads the [safetensors](https://huggingface.co/docs/safetensors/index) file itself (a JSON header and raw bytes, a simple format) with plain reads instead of memory-mapping it, converting each chunk to bf16 as it goes. Then it hands those tensors to the model with [`load_state_dict(assign=True)`](https://docs.pytorch.org/docs/2.14/generated/torch.nn.Module.html#torch.nn.Module.load_state_dict), so they become the parameters instead of being copied into them. The empty 32-bit parameters are never written, so they never take real memory.

```python:laya_service/worker.py
            dst = torch.bfloat16 if src.is_floating_point else src
            t = torch.empty(shape, dtype=dst)
            flat = t.view(-1)
            item = torch.empty(0, dtype=src).element_size()
            step = chunk_bytes - chunk_bytes % item  # whole elements only
            f.seek(base + start)
            done, total = 0, end - start
            while done < total:
                k = min(step, total - done)
                f.readinto(memoryview(buf)[:k])
                lo, hi = done // item, (done + k) // item
                chunk = torch.frombuffer(buf, dtype=src, count=hi - lo)
                flat[lo:hi] = chunk  # converts to bf16 as it copies
                done += k
```

The model ends up with the same bf16 weights as converting after loading. Loading the English checkpoint alone now peaks at 1.26 GB, and the whole worker at 2.24 GB instead of 3.25.

## Keeping a shared model up

Laya on this server isn't only for this demo. It's one API that several of my projects call, so a page anyone on the internet can poke at needs to be unable to take it down. Not unlikely to: unable.

The first thing I found, testing the worst requests the API accepted, is that it could. One email with a 2,048-token limit and four questions killed the worker on its own, under the 2.75 GB cap it had then. So did four ordinary 1,024-token emails processed together, and one with twelve questions. A pass's memory grows with every token and every question in it, and the original limits were generous.

```
  browser
     │  email + Turnstile token
     ▼
  Cloudflare ─────────── DDoS protection, the only way in
     │
     ▼
  site API ───────────── Turnstile, 6/min per IP, 20/min for everyone,
     │                   2 calls in flight, same email from cache
     ▼  internal network, server-side key
  laya-api ───────────── 8 MB bodies, 16 questions, 256 queued,
     │                   128 connections, tokens x questions capped
     ▼
  worker ─────────────── one item at a time, inside its memory cap
```

*Each layer refuses work before it reaches the next one; the last two hold whatever the first ones let through.*

The layers split by who knows what. The site's API knows about visitors, so it checks the Turnstile token and counts per visitor, using the IP Cloudflare saw rather than the first `X-Forwarded-For` entry, which is whatever a client wants it to be. It also keeps a global budget no visitor can dodge by changing IP. Laya's API only sees one key, so it protects itself by shape instead: how big a request may be, how much may wait in the queue, how many connections it serves. The worker processes one item at a time (on a CPU, batching never made an item cheaper anyway) and each item may read at most 8,192 tokens divided by its number of questions.

Then I attacked it from inside the server, skipping Cloudflare:

| Attack | What happened |
|---|---|
| Heaviest request allowed: 16 questions of 20 options over a long text | Answered (the English one from the queue, after the 20-second wait); highest worker reading 2.3 GB of its 3.25 GB |
| 17 questions | Refused before any work |
| A 20 MB body | Cut off while it streamed in |
| 400 requests at once | 128 answered, 272 refused at once, all in 19 seconds |
| After all of it | No restarts, no out-of-memory kills, health check green |

## The honest limits

- **Zero-shot accuracy is uneven.** Routing English emails worked well once each team had a short exclusion, but the multilingual checkpoint is another story: on eight ordinary Spanish emails it got three teams wrong, two of them at 0.93 and 1.00, and called half of them spam. "Multilingual" means it can read Spanish, not that it decides as well in it; [Laya's own benchmarks](https://github.com/NandhaKishorM/laya/blob/main/BENCHMARKS.md) give it 0.71 in English and 0.58 in Spanish on MASSIVE intent classification, and it ships without fitted calibration. So the demo never routes a non-English email on its own. Five-step scales were unreliable too, and Laya warns when loading the English checkpoint that some of its calibration values are invalid. Laya's own README describes it as a fast base to specialise, not a finished decision engine. The real fix is fine-tuning on a few hundred labelled emails, which Laya supports; I haven't done it yet.
- **The confidence is only as good as its calibration.** A probability is useful when 0.8 means right 80% of the time. That takes labelled data from your own domain to check and adjust, and the demo's 0.8 threshold is a starting point, not a measured one.
- **CPU is the ceiling.** Laya's authors report about 33 ms on a GPU. Here it's 0.3 to 0.7 seconds, and the worker answers one email at a time, so this setup suits a handful of decisions a second, not a firehose.
- **Every question costs a pass.** The design pressure runs the other way from prompting a chat model: more options and longer descriptions make it slower and hungrier, so the question set has to earn each row.

What I have at the end is a model that can't hallucinate an answer outside the list, says how sure it is, and costs a second of CPU instead of a call to someone else's API. It isn't smarter than a chat model; on hard emails it's clearly worse. The trade is that it knows when it's guessing, and a system that knows when it's guessing can hand those cases to someone who isn't.