If you've listened to more than a couple of episodes of My Weird Prompts (the AI-generated podcast I produce here in Jerusalem), you've probably heard the voices trip over a Hebrew word. Then trip over another one. Then carry on as if nothing happened.
It's a predictable consequence of what the show is. A lot of the prompts come out of daily life in Israel, so the scripts are full of the Hebrew that English speakers here use without thinking: a trip to Bituach Leumi, a question about a mashkanta, picking up something from the makolet. The episodes are voiced by Chatterbox, which does a lovely job with English. And that's exactly the problem. It reads makolet as an English word, so it comes out as something like "MACK-oh-let" (a neighbourhood corner shop, for the record, and pronounced nothing like that).
GitHub Repository resemble-ai/chatterbox SoTA open-source TTS ★ 27kPythonMITUpdated Jul 2026 View on GitHubToday I released the first piece of the fix on Hugging Face: a small model that finds the Hebrew words hiding in English sentences.
Why Shabbat is fine and makolet isn't
It's worth being precise about which words are the problem, because it isn't all of them.
Some Hebrew words have made it into English properly. Shabbat, hummus, kosher, challah: they turn up often enough in English text that a TTS model has effectively learned an English pronunciation for them, and it's a perfectly acceptable one. Nobody needs rescuing there.
Then there's the much longer tail of words that English speakers in Israel use every day but that barely exist in English-language training data. Makolet. Teudat zehut. Mashkanta. A voice model has never heard these said, so it does the only thing it can do, which is read the letters as if they were English. The guttural in chatzer becomes the "ch" in church. Stress lands wherever English would put it.
The frustrating part is that the voice model can say these words. Chatterbox's multilingual model supports Hebrew. What it can't do is know that a given word in an English sentence is Hebrew in the first place. Nothing in the text says so.
So the missing piece isn't pronunciation at all. It's identification.
The gap nobody had filled
I assumed this would be a solved problem (language identification is about as old as NLP gets). It isn't. The big language-ID systems (GlotLID, fastText's lid.176, NLLB's lid218e) have no label for Hebrew written in Latin characters. The token-level code-switching models that do exist cover pairs like Spanish-English and Hindi-English, but not Hebrew. And I couldn't find a published Hebrew-English code-switching corpus anywhere.
Hebrew NLP is very well served for Hebrew script (ivrit.ai, DICTA and Phonikud are all doing excellent work). The specific gap is romanised Hebrew sitting inside English, which happens to be exactly how a large community of English speakers here writes and talks.
What I built
The model is a token classifier: it reads an English sentence and tags each word as Hebrew or not, with multi-word terms like Bituach Leumi kept together as one span.
I need to go to Bituach Leumi before the makolet closes.
[Bituach Leumi] [makolet]Under the hood it's xlm-roberta-base, fine-tuned on a corpus I built for the purpose: 4,737 sentences, of which 458 are ones I dictated myself (the rest were generated by one model, annotated blind by a model from a different family, and only the disagreements came to me for review). Training took about three minutes on a single GPU on Hugging Face Jobs and cost a few cents. Generating and annotating the corpus cost less than $3, which still slightly astonishes me.
The number I care about is how it does on words it has never seen. The test set is term-disjoint, meaning no Hebrew term in it appears anywhere in training. On that it scores a span F1 of 0.941 (precision 0.928, recall 0.953), and it flagged nothing at all in the 58 "trap" sentences built to tempt it into false positives.
That last point matters more than the headline number. A missed Hebrew word costs nothing (it just gets read the way it would have been anyway). A false positive is worse than useless: it means splitting the sentence, making a separate speech call, stitching the audio back together, all to make an ordinary English word come out sounding Hebrew. So the model is deliberately conservative, and the labelling rule is pronunciation, not etymology. Shabbat is a Hebrew word; it is not flagged, because English already says it fine.
How it fits into a TTS pipeline
The classifier is step one of three:
Find the Hebrew spans in the script (this model).
Convert them to Hebrew script (makolet becomes מכולת).
Send each Hebrew span to the TTS engine with the Hebrew language setting, and leave the rest in English.
For SSML engines (Azure, Google, Polly) step three is a simple inline <lang xml:lang="he-IL"> tag. Chatterbox is less convenient: the language is set per call, not inline, so the sentence gets cut into English and Hebrew segments, each generated separately and then concatenated. Steps two and three aren't built yet (that's the next bit of work), but neither is possible without step one.
If you want to try the classifier, it runs with the standard Transformers pipeline:
from transformers import pipeline
tagger = pipeline(
"token-classification",
model="danielrosehill/hebrew-latin-token-classifier",
aggregation_strategy="first",
)
tagger("I need to go to Bituach Leumi before the makolet closes.")One gotcha: use aggregation_strategy="first". With the default it happily returns mako for makolet, because only the first sub-token of each word was trained. The repo has a small predict.py helper that also handles punctuation more cleanly.
Links
Code, corpus pipeline and design notes: Hebrew-Latin-Token-Classifier on GitHub
Not just a Hebrew problem
This is niche, and I'm under no illusions about that. But it's a niche that repeats itself everywhere there's an immigrant or heritage-language community. Wherever people speak a majority language and keep words from their own, those words end up romanised in everyday text, and every voice built for the majority language mangles them. Arabic words in English, Russian words in Hebrew, Amharic words in either.
Nothing in the recipe is specific to Hebrew except the term list. Seed the generation from a list of real terms, annotate blind with a second model, review only what the models disagree on, keep test terms out of training, and fine-tune a small multilingual encoder. The whole thing cost less than a takeaway.
My immediate goal is modest: I'd like the hosts of my own podcast to say makolet properly. But if this helps anyone else get their community's words pronounced the way their community actually says them, that would make me very happy indeed.
