Sabich v1.0: a second Hebrish model, with everything in it

Yesterday I released Jachnun v1.0, the first proper release of Hebrish, my English-Hebrew code-switching fine-tune of Parakeet. Today there's a second Hebrish model. It's trained on everything I've recorded, and it's called Sabich v1.0.

What Sabich is

Sabich is Jachnun's data plus everything else I've recorded. That means the code-switched speech, and also the tech vocabulary, brand names, acronyms, numbers and currency, Irish-English phrasing, disfluencies and self-corrections. It's named after the Israeli pita with a bit of everything in it. (The code-switching models get Israeli food names.)

It sits next to Jachnun in the Hebrish apps. Pick it if you dictate English with Hebrew words in it and a lot of tech English too.

One base, two recipes

One thing I should have explained earlier: these models aren't trained on top of each other. Every one starts from the same base, stock NVIDIA Parakeet TDT 0.6B v2, and is fine-tuned on all the data collected so far. Sabich isn't Jachnun with extra training. It's a fresh run on a bigger blend.

So the family tree looks like this:

• PoC-1 to PoC-5 (15 to 18 October): proof-of-concept checkpoints.

• Jachnun v1.0 (run r6-en30, 19 October): 339.2 minutes of code-switched speech, 95.3 minutes of my own English and 48.5 minutes of LibriSpeech replay.

• Sabich v1.0 (run r7-all, 20 October): 560.5 minutes of my own recordings, 373.1 of them code-switched, plus 62.5 minutes of replay.

The blend is the only difference between them. Sabich trained for 20 epochs on one L40S GPU on Modal, which took 52 minutes. (Off to one side there's Boxty v1.0, a sibling for English without Hebrew, for the Parlance app. It isn't part of Hebrish.)

The question

Adding data isn't free. A model that also has to learn tech vocabulary, numbers and accent could get worse at the thing Hebrish exists for: writing Hebrew words correctly. If that happened, I'd keep the two apart. If it didn't, both could ship in one app.

The public benchmark

Same benchmark as Jachnun: the public test set, clean subset, 84 clips. This is round 4.

• Term recall (Hebrew terms written correctly): Sabich 73.8%, Jachnun 73.3%.

• Recall on unseen terms: Sabich 68.4%, Jachnun 70.1%.

• WER: Sabich 3.27%, Jachnun 3.32%.

• English control WER: Sabich 1.99%, Jachnun 2.17%.

On code-switching that's a draw. Sabich is half a point ahead on all terms and Jachnun is ahead on unseen ones. On a test set this size, both differences are noise.

The held-out split

The second test is the held-out split of my own dataset. Every model was scored at the same revision, with normalised WER.

• Code-switching WER: Sabich 5.30%, Jachnun 5.42%, stock Parakeet 11.82%.

• Held-out term recall (terms never in training): 48.1%, 46.9%, 23.8%.

• Tech WER: 1.54%, 1.74%, 2.51%.

• LibriSpeech WER: 1.30%, 1.20%, 1.23%.

Both fine-tunes more than halve stock Parakeet's code-switching error rate and roughly double its recall on terms they never saw. Between the two, it's level again on code-switching. Tech vocabulary is where Sabich gains, modestly: 1.54% against 1.74%. On general English all three are within a tenth of a point of each other.

What that settled

Adding the extra data didn't cost code-switching accuracy, and it bought a little on tech vocabulary. That was the question. It's why both models now ship in the same app, and you choose between them in the model picker.

Limitations and the next step

• Small test sets. A difference of a point or two is within noise.

• A modest gain. The tech improvement is small. Sabich is a sensible default if you dictate a lot of tech English, not a big step up.

So far every recording is my own voice. Recordings from more speakers are the next step. If you speak English with Hebrew mixed in, contribute yours.

Get it

Sabich is in the Hebrish apps, from hebrish.app. The details are on hebrish.app/models.

The app build is open on Hugging Face as Hebrish/Sabich-sherpa-onnx-int8. The main model repo, Hebrish/Parakeet-TDT-0.6B-v2-English-Hebrew-Codeswitching-All-Subsets, is gated. The benchmark is Hebrish/English-Hebrew-Codeswitching-ASR-Benchmark.

Try it, dictate a few messages, and tell me where it falls over.