How to clean up dictated text: rules for the patterns, a small model for the judgement

Dictation cleanup works best as two stages: deterministic rules for the fixes a pattern can make, then a small fine-tuned model for the fixes that need a decision about what the speaker meant. One model doing both jobs is slower, more expensive, and measurably worse at the half that matters most, which is knowing when to leave correct text alone.

This is the design behind SpeakoFlow's cleanup, and both halves are published: the rules layer under MIT and SpeakoFlow Mini under Apache-2.0. What follows is the reasoning, including the places the reasoning has a ceiling.

What does the recogniser already do for you?

More than most cleanup features assume, which means a lot of cleanup code is solving a problem that no longer exists. Punctuation, capitalisation, filler removal and number formatting are recogniser-side features now.

Say "the deposit is three hundred dollars, um, and the meeting is at nine thirty" into SpeakoFlow with AI cleanup switched off. What comes back is "The deposit is $300 and the meeting is at 9:30". No language model was involved. Start by finding out what your own pipeline already produces, because whatever it already handles is work you should not be paying for twice.

Which fixes belong in code rather than in a model?

Anything where the correct output is a function of the input rather than an opinion about it. Removing "um" is a table lookup. Collapsing "wh wh wh what" is a loop. Spelling a company name the recogniser has never heard is a dictionary. Turning "thumbs up emoji" into an emoji is a lookup with one guard on it.

Sending those to a language model costs latency, costs money on every API call, and buys variance nobody wanted, because a model told to drop a filler word will sometimes rewrite the sentence around it. A deterministic stage gives you a property a model cannot: the same bytes out on every run, as a pure function of the text and its configuration, with no network and no randomness. The whole rules pass in our implementation costs about a millisecond.

The published rules layer is four functions over a string, arranged as pattern tables plus a written specification. Filler tables are per-language and deliberately not translations of each other, which is the subtlety most implementations miss: "um" is an English filler and the Portuguese word for "a" or "an", and "ha" is Spanish for "has". A single global filler list quietly corrupts text in every language it was not written for. A missed "um" is invisible; a deleted article is a bug report.

Writing these rules from memory does not work, and the repository publishes the audit that proves it, with the sentence that killed each discarded rule.

The surviving rules are the ones that pass an audit against a set of transcripts that are already correct: each rule is checked on its own, and the check fails loudly if any single rule alters one of them. That is the bar a deterministic stage has to clear. It is allowed to decline; it is not allowed to damage.

The measured version of this argument comes from the benchmark. Eight ordered regular expressions resolved roughly a fifth of the transcripts that needed an edit, at perfect restraint across every already-correct transcript and zero content loss. That is the mechanical share of the task, and it is free. Any product shipping a model without a pattern layer in front of it is paying for inference on work a substitution does correctly.

Which fixes genuinely need a model?

The rest, and they are not harder versions of the easy ones. They need a decision that a pattern cannot express, and there are three clean examples.

1. Stutter or emphasis. "the the deadline" is a stumble to collapse. "Alright, alright, alright" is not. The strings look the same to a rule.
2. Homonyms in dictation commands. "period" is punctuation, a fiscal period, or a historical era, depending entirely on the sentence around it.
3. An instruction, or something you said. A mid-clause "new line" is a layout command in one sentence and part of the content in another, and when the recogniser has supplied no punctuation there is nothing mechanical left to distinguish them.

Add to those the cases where the recogniser was accurate and you were not: you corrected yourself mid-sentence, you restated a name more precisely, or you said a word that transcribed perfectly and was still the wrong word. "In the mourning" is spelled correctly. It is also not what you meant.

Why is the order of the stages not negotiable?

Because two of the rule stages fix things the model would otherwise reason about, and two of them assert things the model has no right to overrule. So the model sits in the middle, and reordering changes the output.

Stage 1 also carries the clearest limit in the whole design, and it is worth knowing before you build on it. Its scan looks at windows of up to three words, so a spoken acronym longer than three words cannot match cleanly: "Chat G P T" comes out as "ChatGPT T". That is the strongest argument in the repository for biasing the recogniser's decoder rather than running stage 1 at all. Put the work at the cheapest layer that can do it correctly, and sometimes that layer is upstream of your rules entirely.

Why a small fine-tuned model rather than a big prompted one?

Because the property you most need cannot be bought with parameters. In the published evaluation, a hosted 9B model scored identically to an untuned 0.8B model on restraint, 89.7% each, at eleven times the size. A hosted frontier model scored ten points worse on restraint than the 0.8B fine-tune.

What did move the numbers was training. Against the identical untuned base at the same file size, quantisation, prompt and hardware, edit accuracy went from 4.9% to 48.8%, a gain of 23.4 points overall with a 95% confidence interval of +16.3 to +30.3. Restraint went from 89.7% to 92.6%, which this set cannot resolve, so the defensible claim is that restraint did not fall.

The mechanism is straightforward. A fine-tuned model has the convention in its weights. A prompted model has to be told the convention on every single call, and then has to follow it. Training on a written specification beats prompting for it, and at 0.8B that difference is first-order rather than marginal.

Examples of a boundary work. Instructions about the boundary do not.

There is a practical corollary if you are prompting rather than training. Two settings do most of the damage. Capping output tokens turns a long transcript into lost content, and leaving reasoning enabled fails every already-correct case, because any reasoning text in the output means the output is no longer byte-identical to the input. Neither failure has anything to do with cleanup ability.

How do you tell whether your cleanup is helping?

Score restraint and edit accuracy separately, then put a baseline next to them that returns every input unchanged. Without that baseline you cannot read your own numbers.

Restraint is the share of already-correct transcripts your system returns untouched. Edit accuracy is the share of transcripts needing a change that came back exactly right. Take the unweighted mean of the two and the do-nothing baseline lands at exactly 50.0%, so anything under 50 is worse than shipping nothing. Four of the seven systems tested landed below it, and two of those look safe if you read only the restraint column.

Three practical notes from building this evaluation, each of which cost something to learn.

1. Use exact string match, and no partial credit. Measuring how much strictness costs is better than arguing about it: forgiving every formatting-only miss moved the leaders by about three to four points each and changed the top three not at all.
2. Report both halves next to any aggregate. A model can reach a respectable mean from either direction, and the mean alone hides which. A restraint-only ranking is actively misleading, because it promoted a 9B model above a hosted frontier model that beat it overall by a resolved margin.
3. Do not trust a model as a judge without checking agreement first. Four frontier models from four vendors adjudicated the same disputed cases here and agreed on too few of them to use, so the choice of judge would have moved the result more than the answers did. That tier was dropped rather than tuned, and no judge score was published.

Keep a damage figure outside the score too. Content damage, the share of transcripts that lost a word the speaker actually said, sits at 10.7% for SpeakoFlow Mini. It stays out of the aggregate because any penalty weight for it would be arbitrary, and it stays published because losing a word is the failure users notice.

How does two stages compare with one hosted model doing everything?

Under one fixed short prompt with reasoning off, the two-stage local setup matches a hosted frontier model on this task: a 5.8 point gap with a 95% interval of -1.5 to +12.9, which contains zero. That is a tie, not a win. Only on passages of 500 words and up is the gap resolved, at +20.9 points.

Where the hosted model genuinely wins is worth spelling out. Real dictation produces oddballs no category anticipated, and on those a much larger model with more general understanding reads the situation better and fixes it better. It was also not tested at its best here: both systems got the same short prompt with no reasoning budget, which is the configuration the small model was trained for. And on the subset remaining after an internal audit removed defective scored cases, the hosted model's edit accuracy comes out ahead, 55.9% against 54.4%.

So the finding is not that small beats large. It is that a specification in the weights beats the same specification in a prompt, and that the pattern work should never have gone to a model in the first place.

Frequently asked questions