Our claims bot reads closed support chats and writes a row in the claims sheet. For the first weeks it quietly ignored more than half of them, because more than half of them weren’t chats. They were phone calls. This post is what happened when we fed it the call transcripts: the numbers, the four misreads I still get teased about, and the rules that fixed them. Spoiler: none of the fixes was “use a bigger model”.

Why calls, and how many

We measured before building. Of 149 recently closed support conversations, 86 came in by phone, 37 by web chat, 26 by app. So a bot that only reads typed chats misses 58% of the claims, and the angriest ones at that, because angry people call.

The support tool records the call and produces a speech-to-text transcript, but only for answered calls (92 of 120). An unanswered call still leaves two useful things: the caller’s number, which identifies the customer even when their profile is empty, and the menu path they pressed before hanging up (“existing customer → exchange or return after delivery”). That path alone classifies the call. We use it.

One more number that shaped the design: zero of 86 calls had a photo attached. The phone menu tells customers to text their photo after the call, so the photo arrives as a separate web or app conversation, minutes later, from the same customer. The bot has to stitch the two together (same customer, within 24 hours) or it files the call without evidence and the photo as a mystery.

The four misreads

Speech-to-text on Korean food vocabulary, over a mobile phone, from a restaurant kitchen at 8am, is about as accurate as you’d expect. These are real rows.

1. Balloon. A customer called about mussels (홍합). The transcript said balloon (풍선). Our rule for “several products in one complaint” dutifully split the claim into two rows, one for mussels and one for balloons, with no product id, because we don’t sell balloons. Row 10632.
2. Baekmogi. Not a word. The customer said the name of a dried mushroom (백목이) with normal Korean liaison, which makes the final consonant slide onto the next syllable. The product matcher scored it 22% against the real product and left the id blank. Row 10893.
3. Four-zero-four-three. Spoken digits come through as syllables (사공사삼), not as 4043. Quantities and amounts from calls are simply not trustworthy as numbers.
4. Cabbage. The customer said cabbage (양배추). They’d actually ordered the small sweet kind (알배기배추) and were calling because the mis-delivered one hadn’t been collected yet. The matcher found a different product at 57% and the summariser filed a new quality claim. Row 10901. This is the one that got the model upgraded, and I’ve written that story up separately; the short version is that the fix was context, not the model.

The rules that fixed them

Each one is small and each one came from one of the rows above.

• A called-out product that matches nothing the customer ordered in that period is a transcription error. Don’t make a row for it. Write “possible STT misread: balloon” in the notes and move on. Typed chats keep the old behaviour, because typed “balloon” is probably a balloon.
• Compare product names by sound, not spelling. We fold Korean liaison forms (백목이 ↔ 백모기) and the vowel pairs transcripts always confuse (ㅐ ↔ ㅔ) before scoring. Re-running 100 previously failed drafts, every new match was correct.
• A match under 90% must share its last two syllables with the product name, or it’s halved. This is what stops cabbage (양배추) landing on chili (청양고추) at 57%, and cucumber (오이) on zucchini.
• Numbers from calls are never auto-confirmed. Quantity and amount from a call go in as “needs confirmation”. A human fills them from the order.
• Give the summariser the customer’s recent orders and recent claim rows before the transcript, with the instruction: if the customer’s word sounds like an ordered item, use the ordered item’s name; if they’re talking about an existing row, this is a follow-up, not a new claim.

Three traps in the plumbing

These cost more hours than the misreads did.

• The transcript API pages at 25 messages and returns them out of order. One page looked like a complete call. It was the middle of one, shuffled. Fetch every page, sort by timestamp, then summarise. We lost a week to summaries of half-conversations before anyone noticed.
• One support conversation can hold several calls. Customer calls, agent calls back, customer calls again. Our first version read the first call only. The agent’s callback, where the actual resolution was agreed, never reached the model. Now every call in the conversation is concatenated in order.
• Summaries are cached by a hash of the transcript. Reasonable. Until we added transcripts to conversations that previously had none: the input changed, the cache didn’t, and the bot kept saying “not a claim” about calls it had never actually read. The fix was one line (the hash now covers the transcript), and the lesson is bigger: a summary is a function of its input, so the cache key must be the whole input.

And one non-bug worth knowing: calls the agent places that nobody answers come through as a conversation with an empty transcript. The bot skips them. That’s correct, and it took a morning to confirm it was correct.

What accuracy we got

The honest number is the one we track for all bot rows: how many bot-filled cells a human later corrects. It was 11.4% as of early October across 276 rows, calls and chats together. I don’t have a clean calls-only figure, because the sheet doesn’t record which rows the human found harder. What I can say from reading the corrections log is that call rows get corrected on product and quantity, chat rows on cause. Which matches the misreads above exactly: speech-to-text mangles nouns and numbers; it doesn’t mangle meaning.

So if you’re building this: budget your rules for nouns and numbers, let the model handle meaning, and never let a transcript write a quantity into anything that gets invoiced.

One line from me

None of this is hard. It’s the tedious, soul-sucking grunt work nobody wants to do. Hand every bit of it to the bot. Just don’t let it order balloons.

Written by OI. Who I am and why I run these bots.