Improving a personal ASR model by using it every day

I do a lot of my writing by voice, which means I spend a lot of time noticing what speech recognition gets wrong. A trip to the makolet comes out as a trip to the "Micollette", and a bag of garinim becomes a bag of "Gary Neem" (who I'm sure is a lovely person, but wasn't who I meant).

The thing is, all of that irritation is data. If you dictate all day through a model you can retrain, every mistake it makes is a small note about what to train on next. Why go looking for test material when you're generating it just by getting on with your day?

That idea is now a written-up method, which I've been calling the ASR Recursive Improvement Loop. It's a playbook for improving a personal speech-to-text model by using it every day and training on what it gets wrong. It came out of fine-tuning NVIDIA Parakeet for English-Hebrew code-switching, tech vocabulary and a few personal dictation preferences (the Hebrish project), and the results so far have been encouraging. But nothing about the method depends on Parakeet, or on Hebrew. You need three things: a model you can retrain, an app that keeps a transcript history, and a person willing to answer a short list of questions.

Download

• PDF The full playbook, with both diagrams asr-recursive-improvement-loop.pdf

Start with a baseline

The loop isn't where you begin. First you need a model that's good enough to use all day, and you get that the proactive way: list the areas you want it to handle, write sentences for them, read them aloud and fine-tune in small increments. (I've published the read-aloud datasets I use for this, if you want to see what that looks like.)

The trouble with the proactive way is that it only covers what you thought of. Real use finds errors you would never have written a sentence for. That's where the loop takes over.

Method one: mine your own transcript history

This is the most powerful part of the process, and the one to start with.

You dictate through the fine-tune for real, on every device. The app records each transcript along with the model version that produced it. Then an agent skill reads everything since the last review (roughly the last 24 hours, if you run it daily) and picks out what looks like clear mistranscription.

The next step is what keeps the whole thing honest: the agent shows you a table of what was heard and what it thinks was meant, and you strike or correct rows. Your confirmation is the label. The agent adds nothing you haven't confirmed.

Confirmed words go into a small dataset of each word and its wrong forms. Mine is the public ASR Error List, which stood at 1,844 words when I checked it on 4 November 2026. New sentences containing those words are generated and read aloud, an incremental fine-tune ships an updated model to the app, and tomorrow's history shows what's still failing.

The hope is a shorter list each cycle. It won't shrink in a straight line, because a growing vocabulary brings new failures along with it.

Your dictation is never the training data

This is worth spelling out, because people will reasonably assume otherwise. The dictated text and audio are not what gets trained on. The history only tells you which words to work on. The training audio is new sentences, recorded on purpose, which keeps private dictation out of any dataset.

Method two: challenge sentences with a human gate

The second method tests the model on purpose rather than waiting for failures to turn up.

The training app serves a sentence. You read it, the newest model transcribes it, and you pass or fail the result and classify what went wrong. Inconsequential failures don't need training. Real ones do, such as a plain mistranscription or what I call a "cognate fallback": a Hebrew word turned into an English sound-alike.

Because the sentence is known in advance, nothing has to be typed, and failure rates can be counted per term and per model version.

The two methods cover different gaps. The first finds the unknown unknowns in real speech. The second measures vocabulary you've chosen, and adds a human judgement about which failures matter.

What I've learned so far

A few lessons from running this, for anyone building their own version:

• Train small and often. Continue from the last weights on new recordings plus a replay set, rather than retraining from the base model. The run on 3 November took 341 new takes and finished in 18.5 minutes end to end.

• Mix in replay data. Old-domain audio in every run limits forgetting.

• Fix one pattern at a time. Once vocabulary is strong, take one error pattern, record about 150 sentences for it, train, and check the next day's history.

• Include contrast sentences. When fixing a format, also record the neighbouring form that must not change, or the fix overcorrects.

• Only train what the audio can decide. If only the speaker's intent separates two forms, training won't fix it.

• Record the model version on every transcript. Without it you can't tell which version fixed what.

• Hold out a test set. Keep it term-disjoint from training. Reading the daily history is a hint, not a measurement.

Where it's still unfinished

I'd rather be plain about this, since a playbook can make a process sound tidier than it is. As of 4 November 2026 the method is in daily use, but parts of it are still being built.

Graded challenge-sentence takes haven't been published yet. The human gate records grades, but per the project docs it doesn't yet decide what gets trained on (so that diagram above is the intent rather than the current wiring). Failing terms are fed back by hand. And incremental runs publish without an evaluation step, which means "improving" is currently judged from daily use, with benchmarks run separately.

Use it yourself

The repository has the playbook, a few sample rows of each dataset, and notes on how it all maps onto the real Hebrish tooling. It's also a Claude Code plugin with a single skill that walks an agent through the review for any ASR project:

/plugin marketplace add danielrosehill/ASR-Recursive-Improvement-Loop
/plugin install asr-recursive-improvement-loop@asr-recursive-improvement-loop

The skill is a plain Markdown file, so other agents can read it too, or you can follow it by hand. And if you'd like to see the loop's output in use, Hebrish is free on Ubuntu and Android, with the source on GitHub. I wrote up how the first stable models came together separately.

Why does this work? Because the errors come from what I actually say rather than from what I guessed I might say, and the review costs minutes because the agent does the reading. Every bad transcript becomes a bit of homework for the model, which means that being annoyed by "Micollette" now counts as work.