Hebrish v1.0: benchmarked

Three days ago I wrote up version 3 of my English-Hebrew code-switching fine-tune of Parakeet. Since then the project has picked up a name, a website (hebrish.app) and, as of today, a release I'm prepared to put my name to. This post is about that release and how it scored.

A naming change

Hebrish is English dictation that writes Hebrew words correctly. Makolet, Bituach Leumi and teudat zehut come out in one consistent romanised spelling, instead of "Micollette" and friends. Under the hood it's a fine-tune of NVIDIA's Parakeet TDT 0.6B v2.

Until now I'd been calling every training run a new version. That gave me five "versions" in a few weeks, which oversold things. So the first five checkpoints (published on Hugging Face as v1 to v5) are now PoC-1 to PoC-5: proofs of concept. The sixth checkpoint, training run r6-en30, is v1.0, nicknamed "Jachnun". It's the first release I'll stand by. What it needs now is real people trying it.

What changed since v3

v3 is now PoC-3. After it came PoC-4 and PoC-5. Both were trained, but neither was ever benchmarked. v1.0 is the first checkpoint since PoC-3 to go through the full set of tests.

The training mix for v1.0:

• 339.2 minutes of code-switched speech

• 95.3 minutes of my own plain English

• 48.5 minutes of LibriSpeech replay

That's about 30% English. It's there to stop the model forgetting what it already knew. Each version is a full fine-tune from stock Parakeet on all the data collected so far, not continued training from the previous checkpoint. The run took 41.0 minutes on one L40S GPU on Modal. The best epoch was 18 of 20, and the run cost about US$0.80.

The public benchmark

The public benchmark is a separate English-Hebrew test set, and every model is scored on the same clips. This round used test set v1, clean subset: 84 clips containing 172 Hebrew term occurrences, scored on 19 October 2026.

Term recall is the share of Hebrew terms written correctly. The figure in brackets is for terms the model never saw in training.

• Hebrish v1.0: 73.3% of terms (70.1% unseen), 3.3% WER

• Hebrish PoC-3: 67.4% (63.2%), 4.0% WER

• Hebrish Nemotron streaming fine-tune, checkpoint 2, scored offline: 62.2% (54.7%), 4.7% WER

• Whisper Hebrish v1: 54.1% (56.4%), 6.5% WER

• Whisper Large V3: 45.9% (46.2%), 7.5% WER

• Stock Parakeet TDT 0.6B v2: 32.6% (32.5%), 9.5% WER

On the plain English control clips, v1.0 scored 2.2% WER, against 2.5% for PoC-3, 2.7% for stock Parakeet, 2.4% for Whisper Large V3, 4.7% for Whisper Hebrish and 2.5% for the Nemotron fine-tune.

I also checked for contamination. None of the 145 benchmark clips are in the training data.

The held-out split

The second test is the held-out split of my own dataset, at revision 550bfcaa0f84: 387 code-switched clips. Some of the Hebrew terms in it never appear in training.

• Code-switching WER: stock 11.3%, PoC-3 6.0%, v1.0 5.1%

• Held-out terms (never in training) correct: 25.9%, 45.7%, 47.8%

• Test terms also seen in training correct: 27.7%, 63.3%, 72.7%

So did the extra data help with new words? Not measurably. v1.0's gain over PoC-3 on held-out terms is 2.1 points, about 6 terms out of 293. That's within noise. The 9.4-point gain on seen terms is not. It's the same lesson as v3. More audio makes the model better at words it has heard, and recognition of brand-new words sits just under 50%.

English didn't get worse

On my own plain English, WER went from 3.1% with stock Parakeet to 1.7% with v1.0. On LibriSpeech test-clean it's 1.23% against 1.20%, which is noise. The English in the training mix is doing its job.

The 50-cent benchmark

Scoring is cheap. The benchmark ran as six Modal L40S jobs in parallel, between 19:44 and 19:46 UTC. Each took about two and a half minutes of wall time. Together that's under 15 GPU-minutes, or about US$0.50 at most. (That's less than a single training run, so there was never a good reason to leave PoC-4 and PoC-5 unscored.)

Limitations

• One speaker's voice. All of the training and test audio is me reading. These numbers rank models on my voice, and nobody else's.

• Read speech. Not spontaneous conversation.

• A small test set. A difference of two or three points of recall is within noise.

• Not all Hebrew. 22 of the benchmark's 139 terms are English tech labels.

• Possible leaks. Two held-out terms (mandatim and Beit Safafa) may have leaked into training. I'm checking.

Try it

The full results are on hebrish.app/benchmarks, and the method is written up in the v1.0 white paper (PDF). The model is on Hugging Face, and every scored model is on the leaderboard Space.

What I need now is other voices. If you're an English speaker in Israel, get the apps from hebrish.app, dictate a few messages, and tell me where it falls over.