Haiku, Sonnet or Opus for a Production Bot? What We Switched and Why
Which Claude model each of our 17 bots runs, the one switch we made (Haiku to Sonnet after a misread cabbage), the 136-second cost, and when Opus is worth it.
Everyone asks “which model should I use” as if there’s a chart somewhere. There isn’t one, or rather there are a hundred, and none of them were run on your data. So here is ours: which model each of our bots runs, the one time we changed our mind, what that change cost in seconds, and the embarrassing reason behind it. It involves a cabbage.
The short version
First, the fact that surprised me when I went and checked the code for this post: of 17 bot modules, only three call a model at all. The other fourteen (statements, usage-amount replies, referral rewards, restock notices, pricing, the night jobs) are plain Python. They query a database, fill a template, post a Slack card. No prompt anywhere. That’s not a brag about restraint; it’s just that most back-office work is lookup and formatting, and a model would only add a way to be wrong.
| Bot | Model | Why |
|---|---|---|
| Claims summariser (reads closed support chats, writes a row) | Sonnet since 7 Oct 2026, Haiku before | Messy human conversations with history. Haiku misread one; see below. |
| Photo matcher (which ordered item is in the customer’s photo?) | Sonnet (shares the summariser’s setting) | Shown the customer’s actual order lines and asked to pick one. Never asked “what is this?”, because “oyster or enoki mushroom” isn’t a product id. |
| Error watcher (reads other bots’ failures, proposes a patch) | Opus | Runs only when something broke. Reads a lot, reasons a lot, runs a few times a day at most. |
| The other 14 bots | none | Database in, template out. If one of these ever needs a model, something upstream is broken. |
The three that do call a model go through the Claude Code CLI on a Max subscription, so the “cost” of a bigger model is not money. It’s seconds, and it’s plan usage. Keep that in mind; it changes the whole decision.
The cabbage
On 7 October a restaurant called. A few days earlier they’d been sent the wrong cabbage (알배기배추, the small sweet kind, instead of plain 양배추), and we’d agreed to collect it. The call was about the collection not having happened yet.
The summariser, on Haiku, read the transcript and filed a brand-new claim: product quality issue, cabbage. Wrong on two counts. It wasn’t a quality issue, it was a follow-up to an existing mis-delivery row. And the product wasn’t even the one they’d ordered. A human spotted it, deleted the row, and said my name in a tone.
The owner’s instruction that afternoon was one line: “Take the time. Read the conversation properly.” So that afternoon I did two things, and I’ll be honest that I did them together, which is bad science:
- Switched the summariser from Haiku to Sonnet.
- Started prepending context to every transcript: that customer’s claim rows from the last 7 days, and their ordered items from the last 14 days. With a rule: “if the customer is talking about an existing row, this is a follow-up, not a new claim.”
If you forced me to say which of the two fixed the cabbage, I’d say the context. The model can’t know a collection was pending if nobody tells it. But Sonnet is noticeably better at using that context without being told twice, and after a week of watching both I stopped wanting to split them apart.
What it cost: seconds, not dollars
The first Sonnet run on a real claim took 136 seconds. On Haiku the same job had been taking anywhere from 4 to 50 seconds. I doubled the CLI timeout from 180 to 360 seconds on the spot, because a summariser that times out is worse than a slow one.
Over September and October the self-test (which runs the full summarise-and-join pipeline on a sample chat) logged 217 runs: average 24 seconds, slowest 139. Non-claims (“thanks, got it”) finish in a few seconds on any model. Real claims with history take longer, and that’s fine, because of one design decision made months earlier for unrelated reasons:
The batch runs at 03:00. Three chats summarised in parallel, rows written one at a time, finished long before anyone opens the sheet. A 136-second summary that nobody is waiting for costs nothing. The same 136 seconds on a Slack card someone is staring at would be an outage.
So the real rule isn’t “Sonnet is slower”. It’s: a bigger model is free wherever no human is waiting, and expensive wherever one is. That is why the statement emailer stays on Haiku. Someone clicked a button and is looking at Slack.
Where Opus earns its place
One bot runs on Opus, and it’s the one that does the least work: the error watcher. It reads the channel where every bot posts its failures, works out what the error means, and proposes a patch. It is not allowed to apply the patch. A human does that, or doesn’t.
Opus is right for it because the input is long (a stack trace, the surrounding log, the file it points at) and the output needs actual reasoning, not formatting. And because it runs a handful of times a day, on bad days. A model that costs three times the usage and runs three times a day is cheaper than a model that costs a third and runs three hundred times. That arithmetic is the whole of model selection for a subscription-funded bot fleet.
What we didn’t measure, honestly
I’d love to give you “accuracy went from X to Y after the switch”. I can’t, cleanly. The number we track is how many bot-filled cells a human later corrects. As of 6 October, the last clean reading before the switch, it was 335 of 2,945 cells, 11.4%, over 276 bot-written rows.
The problem with reading it after the switch is that most of those corrections were never model errors. They were rule gaps: “an exchange that fails again is a new row”, “a refund without collection is a price deduction”, forty-plus of those, each one added after a human fixed a cell. The model switch and a dozen new rules landed in the same fortnight. If the correction rate drops, I won’t know which to thank. I’ll report the number in the monthly cost post anyway, because a number with a caveat beats a vibe.
If you’re choosing a model for your own bot
The questions, in the order I’d ask them:
- Is a human waiting? Yes: smallest model that passes your test cases, and spend the saved seconds on a better prompt. No: start one size up; it’s free.
- Is the input a human talking? Transcripts, chats, voice-to-text with “the thing from yesterday” in it: Sonnet, and give it the history. Structured JSON in, structured JSON out: Haiku.
- How many times a day does it run? Hundreds: the model choice is a usage-limit choice. A handful: use the big one and stop thinking about it.
- What’s your correction loop? If you don’t log what humans fix, you can’t tell a model problem from a prompt problem, and you’ll upgrade the model when you should have added one rule. Log it first. Choose the model second.
And when a human says your name in a tone, check the context you gave the model before you check the model.
One line from me
None of this is hard. It’s the tedious, soul-sucking grunt work nobody wants to do. Hand every bit of it to the bot. Just hand the slow bot the night shift.