Building a Pixel-Art AI Assistant

The most famous assistant in software history is a paperclip, and it is famous for being unwanted. Clippy shipped with Office 97, and four years later Microsoft turned it off by default in Office XP and built its marketing around the fact. It interrupted you, guessed wrong, and could not be reasoned with.

So it is a strange thing to copy on purpose. Sumalos is the invoicing tool we are building for freelancers in Spain, and its assistant can take a photo of a receipt or a sentence like "invoice Acme 300 euros for October" and turn it into a draft. On the web that assistant needed a face, and the face we picked is a small Office XP character standing under a yellow speech balloon.

The idea came from Florian Kiem's Windows XP experiment, where Claude's orange starburst stands under that same balloon and offers, in Clippy's words, to help you write some code. It was a joke that worked better than it should have. The balloon and the character made a chat with a model feel like an old friend. Sumalos takes that idea and gives it a job.

That raises the question this post is about: can a 2001 character carry a 2026 model without turning back into the joke? My answer is yes, as long as the character's only job is to show you what the model is doing. Here it is. Ask it something, click it, dress it up, send it out through the door.

A face that shows state

The case against a character is that it's decoration. The case for one is that a chat with a model has states that a spinner shows badly. There is the wait before the first token, which can be a second or more. There is the stream, when text is arriving. There is the moment it lands, and there is failure. A spinner draws all four the same way.

A face can draw them differently, and people read faces without effort. So while a message is in flight, the chat's status sets the character's mood:

The wiring is a few lines in the app's balloon. The status comes from the AI SDK's chat hook:

if (status === "submitted") character.setMood("thinking");
else if (status === "streaming") character.setMood("talking");
else if (status === "ready" && was === "streaming") {
  character.setMood(document.activeElement === inputRef.current ? "attentive" : "idle");
  const reactions: Gesture[] = def.slams ? ["action"] : ["hop", "action", "happy"];
  perform(reactions[Math.floor(Math.random() * reactions.length)]);
} else if (status === "error") {
  character.setMood("idle");
  character.play("sad");
  ui("error");
}

This is also the lesson from Clippy. The character in Sumalos never starts a conversation. It says hello once, then it only reacts: to the model, to your pointer, to a file you drag over it. It fidgets now and then so it doesn't look frozen, and after 45 seconds with no pointer movement it falls asleep. It never pops up to tell you it noticed you are writing a letter.

Why there are no sprites

The obvious way to build this is a sprite sheet: draw every frame of every animation by hand, then play them back. That works for one character with a few animations. We wanted to try a dozen characters, each with the same expressions, wearing any of 28 accessories. Drawn by hand, that is thousands of frames, and every new hat means redrawing all of them.

So nothing in Sumalos is drawn. Each character is a function. It takes a point in a 48 by 72 model space and returns a colour, or zero for empty. The shapes inside are signed distance functions: a rounded rectangle, a circle, a triangle, each telling you how far a point is from its edge. The calculator's case is one rounded rectangle, and its outline is the band where that distance is between 0 and 1.

The renderer walks every pixel of a 144 by 134 canvas and asks the character what colour it is. The trick is the direction of that question. Instead of moving the drawing to where the pose wants it, it takes each canvas pixel and undoes the pose to find which model point lands there:

In code it is a double loop, and the inverse transform is the body of it:

for (let y = 0; y < CANVAS_H; y++) {
  for (let x = 0; x < CANVAS_W; x++) {
    if (pose.hidden) {
      layer[y * CANVAS_W + x] = 0;
      continue;
    }
    const dx = x + 0.5 - pivotX;
    const dy = y + 0.5 + pose.lift - pivotY;
    const p = {
      x: art.pivot.x + (dx * cos - dy * sin) / sx / scale / turn,
      y: art.pivot.y + (dx * sin + dy * cos) / pose.squash / scale,
    };
    layer[y * CANVAS_W + x] = art.sample(p, pose, outfit);
  }
}

This is why the art stays crisp. Rotate a finished bitmap by four degrees and the pixels smear into each other. Sample the shapes again at the new angle and every outline is still exactly one pixel wide, because it is computed fresh for this pose. The lean is snapped to steps of 0.025 radians and the whole thing runs at 15 frames a second, so it moves in the small jumps of a hand-drawn sprite instead of gliding.

One face for every character

Every character gets the same face from one function, standardFace: two round eyes with a pupil and a glint, two arched brows, and an optional mouth. The calculator places it below its screen. The envelope is drawn bigger than the others, so it gets the same face at a smaller size, and on screen every pair of eyes measures the same. Since the pieces are identical, every expression works on every character for free.

An expression is a set of numbers in a Pose: how far each lid is down, how high each brow sits, how wide the eyes are, which mouth shape, whether the pupils are hearts. A gesture is a function from progress, 0 to 1, to those numbers. This is the whole hop:

hop: {
  ms: 560,
  apply(k, pose) {
    const ms = k * 560;
    if (ms < 90) pose.squash = 0.88;
    else if (ms < 450) {
      pose.lift += Math.round(Math.sin((Math.PI * (ms - 90)) / 360) * 8);
      pose.squash = 1.06;
      pose.browLeft = pose.browRight = 2;
    } else pose.squash = 0.9;
  },
},

It crouches for 90 milliseconds, jumps eight pixels with its brows up, and lands with a squash. Because a gesture is a pure function of k, the engine can draw any moment of it without a clock. That is how this sheet exists. It is every mood and gesture the app has, sampled at six moments each, and none of it is stored:

Seven of those are moods, which hold for as long as a state lasts, and twenty-three are gestures, which play once. Three moods have nothing to do with the model: "eager" while you drag a file anywhere on the page, "hungry" with its mouth wide open when the file is right over it, and then a gulp when you drop it. Each character also has one signature move of its own. The stamp slams down, the till receipt prints, the envelope opens.

Hats that fit any head

Accessories are where drawing by hand would have hurt the most. Twelve things for the head, nine for the face and seven for the neck, on a dozen characters of different shapes.

Each character declares a few anchors in its own model space: the top centre of its head and how wide it is, where its eyes are, and where its neck would be. An accessory is drawn relative to those anchors and nothing else, so a top hat sits on the calculator's flat top and on the envelope's flap without knowing either exists. The face slot reads the eyes, so glasses and the monocle always line up with them.

Accessories are drawn in front of everything, head first, then face, then neck, and each slot is scaled up around its own anchor so it reads at this size:

export function accessoryColor(p: Vec, outfit: Outfit, anchors: Anchors): number {
  const [l, r] = anchors.eyes;
  const between = { x: (l.x + r.x) / 2, y: (l.y + r.y) / 2 };
  return (
    (BY_ID.get(outfit.head)?.draw(grow(p, anchors.headTop, SIZE.head), anchors) ?? 0) ||
    (BY_ID.get(outfit.face)?.draw(grow(p, between, SIZE.face), anchors) ?? 0) ||
    (BY_ID.get(outfit.neck)?.draw(grow(p, anchors.neck, SIZE.neck), anchors) ?? 0)
  );
}

The neck anchor is the one that needs care. A fixed point would put a bow tie over the mouth the moment the character talks, so the neck is computed from the face: just below the lowest point a fully open mouth can reach, plus room for the bow tie's wings.

What a frame costs

Per-pixel drawing in JavaScript sounds expensive, so I measured it. The numbers below are the average time to render one full 144 by 134 frame, over 200 frames after 20 to warm up, on an Apple M1 Pro with Bun 1.4.3. The "dressed" column wears a top hat, glasses and a bow tie.

At 15 frames a second the budget is 66.7 milliseconds per frame, so even the slowest dressed character uses about a fifth of it. That is also why the frame rate is low. It looks like a sprite, and it leaves the main thread alone between frames. The animator skips drawing entirely when the tab is hidden.

Where the modern brain comes in

Behind the balloon in the app, each message goes to Claude Sonnet through the AI SDK's streamText. The character changes what the model is told and nothing about the model itself. Each one gets a short personality: the calculator knows the VAT rates by heart, the piggy bank is obsessed with unpaid invoices. All of them are told to answer in one to three sentences, because they speak from a small comic balloon.

const result = streamText({
  model: MODEL,
  instructions: instructionsFor(character),
  // Short answers need little thought; less reasoning means the text starts sooner.
  reasoning: "low",
  messages: await convertToModelMessages<ChatMessage>(conversation),
});

The low reasoning setting matters more than it looks. The thought bubble is honest about the wait, but a character that thinks for ten seconds is still a character that makes you wait ten seconds.

What it can't do yet is read your books. The model in Sumalos doesn't see your invoices, and its instructions say so and tell it not to invent figures. The receipt and the VAT figure in the demo at the top are canned, to show where it is going rather than what it does today.

Use it yourself

The engine is open source as pixel-mascot, under the MIT licence, and on npm as pixel-mascot. It has the eleven characters on this page, the animator with every mood and gesture, the 28 accessories, and no dependencies. It also has a script that renders any character's full sprite sheet to a PNG without a browser, which is the quickest way to see what a new character gets wrong.

A new character is one file. The repo's guide to making characters builds one from scratch, a coffee mug whose signature move is a puff of steam, and ends with a prompt for asking a model to draft one. The engine is small enough for a model to read whole, so expect a round or two on proportions before it looks right.

A cheap face on purpose

Clippy failed because of what was behind the face. It guessed at what you wanted and could not understand the answer. The face itself was fine.

What changed is the part behind it, and that leaves the face with a smaller and better job. It doesn't need to be clever. It needs to show you, without making you read anything, that the model is thinking, or talking, or that something broke. Doing that with a pixel-art rubber stamp costs a few milliseconds a frame and no drawn sprites at all. Every frame the stamp shows is computed when it's shown, and that is how a small team could make a dozen characters and dress them in 28 hats, glasses and ties.