Can speech-to-text spell ‘makolet’? I tested seven models on Hebrish

English speakers in Israel talk in a mix of English and Hebrew ("I need to grab some bamba from the makolet"). Linguists call hopping between languages mid-sentence code-switching; around here it goes by Hebrish. Dictation software hates it. Earlier today I released the second version of my Hebrish dictation model, a fine-tune of NVIDIA's Parakeet v2 speech recogniser that is supposed to hear "makolet" and write makolet rather than "Micollette". The numbers in that post were good. They were also numbers about one model compared with its own starting point, which is a bit like a student marking their own homework (it can be done honestly, but nobody should take your word for it).

The question I actually care about is a different one. If you're an English speaker in Israel and you dictate a lot, which model should you use? Stock Whisper? Stock Parakeet? My old Whisper fine-tune from last year? The new one? To answer that I needed every model on the same audio, scored the same way, by code anyone can rerun. So I built a small benchmark, and a leaderboard to go with it.

What it measures

The test set is 145 short clips of me reading English sentences with Hebrew words in them ("I need to pick up my teudat zehut from Misrad Hapnim", that sort of thing). It's frozen on Hugging Face at a fixed commit, so a result scored today and a result scored next year are scored against exactly the same audio.

The headline number isn't word error rate. WER is the standard metric, and it's in there, but it flatters models on this particular problem. A fifteen-word sentence with one mangled Hebrew word scores about 7% WER, which sounds respectable. But that one word was the whole point of the sentence (nobody needs help transcribing "I need to pick up my"). So the leaderboard ranks by term recall: the share of Hebrew words the model wrote with the expected romanised spelling, as a whole word, ignoring case.

There are three supporting numbers. Unseen-term recall counts only Hebrew words that never appear in the public training data, which is the honest test of whether a fine-tune learned anything general. Lenient recall also accepts reasonable alternative spellings (tsaharon for tzaharon), because a stock model was never taught my spelling convention and shouldn't be punished too hard for picking a different one. And an English control set of 24 clips with no Hebrew at all catches the classic failure where a fine-tune learns Hebrew and forgets English.

Two awkward discoveries

Building the benchmark turned up two problems before a single model had been run, and I think they're worth mentioning because they're exactly the kind of thing that quietly corrupts a leaderboard.

The first was contamination. I checked whether any test clips had leaked into anyone's training data, and they had. 26 of the older desktop-microphone clips in the test split are, file for file, recordings that Whisper-Hebrish (my Whisper fine-tune from November 2025) was trained on. Another 7 phone clips are new recordings of sentences it was trained on. Nobody did anything wrong here: those clips were imported into the newer dataset long after Whisper-Hebrish was trained. But scoring Whisper-Hebrish on its own training audio would have flattered it badly. So the headline uses only the 84 clips that appear in no published training set, and the other 37 are reported separately.

The second was subtler. I fed the reference transcripts into the scorer as if they were model output, which should give a perfect score. It didn't. Four references spelled a Hebrew word one way ("heder madregot") while the label said another ("cheder madregot"), so no model could ever have got those right. That's now fixed, and the "oracle" check stays as a test. If there's a general lesson, it's to check the benchmark before you check the models.

The results

How many Hebrew words each model gets right: my fine-tune release 2 (built on Parakeet v2) 66%, release 1 62%, Whisper-Hebrish 54%, Whisper Large V3 46%, Whisper Large V3 Turbo 40%, stock Parakeet v2 33%, stock Parakeet v3 31%

On the 84 clean clips (WER is word error rate, the share of words a model gets wrong, so lower is better):

• My fine-tune, release 2 (built on Parakeet v2): 65.7% of Hebrew words right, 4.2% WER

• My fine-tune, release 1 (built on Parakeet v2): 62.2%, 4.6% WER

• Whisper-Hebrish (my 2025 Whisper fine-tune): 54.1%, 6.5% WER

• Whisper Large V3 (stock): 45.9%, 7.5% WER

• Whisper Large V3 Turbo (stock): 40.1%, 8.9% WER

• Parakeet v2 (stock): 32.6%, 9.5% WER

• Parakeet v3 (stock): 31.4%, 10.5% WER

A few things stood out.

The fine-tune does what it says. It doubles stock Parakeet's hit rate on Hebrew words and more than halves its WER, and the English control barely moves (2.7% for stock, 2.4% for the fine-tune). That last part is the one I was most nervous about. My own tests in the fine-tune's project, scored on the full test split, tell the same story from the other side:

Hebrew words transcribed correctly, stock Parakeet v2 against my fine-tune release 2: 29% to 50% for words never seen in training, 38% to 75% for words seen in training

Stock Whisper is noticeably better at Hebrew words than stock Parakeet. My guess (and it is only a guess) is that Whisper's multilingual training has heard a fair amount of Hebrew, while Parakeet v2 is English-only. Parakeet v3 doesn't close the gap either: its 25 added languages are European, so Hebrew isn't among them.

Whisper-Hebrish beat every stock model on Hebrew, but it paid for it in English. Its English control WER is 4.7%, against 2.4% for the stock Whisper it was trained from. That's the textbook result of fine-tuning on code-switched speech alone, and it's precisely why the Parakeet fine-tune was trained with about 30% plain English mixed in.

And releases 1 and 2 of my fine-tune are effectively tied on words they've never heard (61.5% against 60.7%, which is a difference of one or two words). The extra 43 minutes of audio in release 2 helped with the words it had recordings of. It didn't help with new ones, which tells me where the next round of recording should go.

What it can't tell you

This is a benchmark of one speaker: me, with an Irish accent, reading aloud into a phone in Jerusalem. It ranks models on my code-switching, and the fine-tunes were trained on my voice, so of course they do well on it. With 84 clips and 172 Hebrew words, a gap of two or three percentage points is a handful of words and shouldn't be read as a real difference. It says nothing about spontaneous speech, other accents or Hebrew spoken by native speakers.

I'd still rather have a small, honest, reproducible benchmark than a large impression. And the format is built to grow: a second speaker would become a new edition of the test set, and every model can be rescored against it.

Try it, or add a model

Everything is public:

• The leaderboard, with per-word results and the actual transcripts side by side

• The test set on Hugging Face

• The runner and scorer on GitHub

Any NeMo or Whisper checkpoint on Hugging Face runs with one command:

ehcs-bench run --model whisper:openai/whisper-large-v3-turbo --name "Whisper Large V3 Turbo"

For anything that can't run there (a cloud API, a phone app, a GGUF build), you transcribe the 145 clips however you like and score the transcripts with ehcs-bench score. A pull request with the result file puts it on the leaderboard.

If you speak English with Hebrew mixed in and would be willing to read a few dozen sentences into your phone, I'd love to hear from you. That's the thing that would turn this from a benchmark of my voice into a benchmark of ours.