When I last wrote up the Hebrish models in depth, on 21 October, Sabich was at v1.0 and had learned from just over ten hours of audio. Seventeen days and a lot of reading aloud later (the OnePlus has had a busy month), it's at 31.3 hours and has stopped being retrained from scratch.
This is a progress report on three things: how much audio there is now, how it's blended, and the question I most wanted answered. Does teaching a speech model a pile of tech vocabulary cost it accuracy on the Hebrew words it was fine-tuned for in the first place?
So far, it doesn't.
Where things stand
The training history on hebrish.app now runs to Sabich v2.9, dated 4 November. Versions up to v2.4 (1 November) were each trained from stock Parakeet TDT 0.6B v2. From 2 November, the line went rolling: each update continues from the previous weights for a few epochs, rather than starting again. In the apps it's now simply called Sabich, with no version number to keep track of.
The composite total at v2.9 is 31.27 hours of distinct audio across 18,517 clips. That's 27.9 hours of my own voice and 3.36 hours of public LibriSpeech. Counting the deliberately repeated clips, a full pass through the blend comes to 33.56 hours. (I'm quoting the distinct figure as the honest one, since a repeated clip isn't new audio.) Apart from the LibriSpeech, every clip is still my voice, which is the biggest limitation of the whole project, and I'll come back to it.
The blend
Every clip goes into exactly one of five ingredients. At v2.9 they look like this:
Code-switching: 12.65 hours (40.5%). English with Hebrew words in it, which is what the project exists for.
Other English: 5.82 hours (18.6%). Task sets, numbers, accents and everyday English, all in my own voice.
Tech vocabulary: 5.03 hours (16.1%). Model, product and CLI names, acronyms and brands.
Priority words: 4.40 hours (14.1%). Words my dictation got wrong, each repeated up to three times in a run, with the extra copies capped at 10% of the run's audio.
LibriSpeech replay: 3.36 hours (10.7%). Public audiobook English, held at about 10% so the model keeps its general English.
The shape of that chart is the interesting part. Code-switching has doubled in hours (from 6.22 at v1.0) while falling from 60% of the mix to 40%. Tech vocabulary has gone from 1.07 hours to 5.03, and priority words didn't exist at v1.0 at all. In other words, the mix has moved away from the thing the model was built for. If tech vocabulary were going to crowd out the Hebrew, this is how you'd expect it to happen.
Does tech vocabulary hurt the Hebrew?
The cleanest comparison I have is Jachnun v1.0 (code-switching only) against Sabich v1.0 (code-switching plus tech and everyday dictation). Both were trained from stock Parakeet. Here they are on public test set v2, which is 129 clips, Jachnun first:
Code-switching WER: 5.85% and 5.73% (stock Parakeet: 13.72%)
Hebrew terms written exactly right: 61.1% and 63.4%
Terms that weren't in the training data: 51.8% and 53.3%
Tech WER: 6.03% and 4.77%
English control WER: 2.42% and 2.12%
The held-out split agrees: 5.42% and 5.30% on code-switching (422 clips), and 46.9% and 48.1% on held-out Hebrew terms. Sabich did come out marginally worse on my own plain English (1.56% to 1.62%) and on LibriSpeech (1.20% to 1.30%), but those are differences of a few words.
So the tech scores moved the right way on both tests, and the Hebrew didn't pay for it.
The later versions point the same way. Between Sabich v1.0 and v2.3, the tech ingredient more than tripled to 3.58 hours, and on the same test set code-switching WER fell from 5.73% to 4.96%, term recall rose from 63.4% to 67.0% and tech WER fell from 4.77% to 3.02%. (v2.2 actually beat v2.3 on that test set, at 4.91% and 70.1%, while v2.3 won on its own held-out split. Both differences are small, and the two tests simply disagree.)
Now for the caveats, because this is an initial result and not a controlled experiment:
There's one scored run per model, on one speaker reading aloud. Every gap between Jachnun and Sabich is within noise at these sample sizes. I'd claim "no sign of harm", not "tech vocabulary improves the Hebrew".
It isn't a clean ablation. Sabich v1.0 also had a little more code-switching audio than Jachnun (373.1 minutes against 339.2), and across the v2 series the code-switching data grew alongside everything else.
Benchmarks stop at v2.3. The first rolling update was checked internally against v2.4 before release, but no evaluation numbers are published for the rolling line, so the current model in the apps is unscored in public.
What's next
The obvious gap is scoring the rolling line, to see whether continuing from previous weights holds up as well as retraining did. The other is voices: the models are trained on my recordings only, and a model that has only heard one Irishman in Jerusalem is going to be a very good model for exactly one person. If you speak English with Hebrew mixed in and would like to change that, hebrish.app has the details, along with the benchmarks and the models themselves.
The fear with any fine-tune is that every new thing the model learns gets paid for somewhere else. Seventeen days in, the bill hasn't arrived.
