Datasets for fine-tuning speech recognition on your own voice

If you want a speech recognition model to get your vocabulary right, you have to record yourself saying it. I have spent the autumn doing exactly that, one sentence at a time, on my phone. The recordings now live on Hugging Face as a family of small datasets, each one for a different kind of vocabulary: tech terms, Linux commands, smart-home gear, my own name and so on.

This post is the map of those datasets: what is in each, which one to use for what, and how they relate. They all share one speaker (me, with an Irish accent, in Jerusalem), one phone and one licence, so they are not a general-purpose corpus. They are a worked example of how to build a personal one. The first one to look at is Tech-Sentences, the biggest of the set.

Where the sentences come from

The sentences are written first and read aloud second. New terms start as sentences written by a language model, which is the job of the Contextual Sentence Creator, then I record each one on my phone and keep the transcript as the ground truth. The recordings feed the Parakeet and Whisper fine-tunes I have written about, including Fritter for English tech dictation and the Hebrish models covered in the Hebrish write-up.

Everything is recorded into one master dataset, Read-Aloud-Voice-Sentences, which carries a category label on every take. The datasets below are slices of it. A script copies the rows for one category into its own repository unchanged, with the same ids, columns and train/test split, so fixes are made in the master and re-synced. The master dataset itself is gated: you accept its terms on Hugging Face before you can load it.

Two details are the same in every slice. The test split is term-disjoint, meaning a term held out for testing never appears in training, so a test score measures whether the model learned the word in general, not just that exact sentence. And every recording was made on a OnePlus Nord 3 5G with its internal microphone, with echo cancellation, noise suppression and auto gain turned off. Sometimes I change my speaking speed on purpose, or say a word the way people commonly mispronounce it, so a take that sounds off against its card is usually intentional.

The sentence datasets

All of these are CC-BY-4.0, English, and one speaker. Row counts are train plus test as reported by the Hugging Face dataset viewer on 3 November 2026; they grow as I record more.

• General English (General-English-Sentences, about 1,560 rows): plain everyday English such as emails, messages and homophone pairs, with no Hebrew, no names and no tech vocabulary. Use it to adapt a model to one speaker's voice and accent, or as a control so a fine-tune doesn't only learn jargon.

• Tech (Tech-Sentences, about 4,100 rows): dictation-style sentences dense in software, DevOps, cloud, self-hosting and AI vocabulary, with product names in the vendor's own casing (GitHub, PostgreSQL, kubectl). It includes long-form dictation of instructions to AI coding agents. Sentences that code-switch into Hebrew were removed from it on 19 October 2026.

• AI interactions (AI-Interactions-Sentences, about 560 rows): instructions dictated to coding agents such as Claude Code and Codex, plus the git and GitHub workflow around them: subagents, worktrees, compaction, plan mode, hooks, MCP servers, rebases.

• Linux admin (Linux-Admin-Sentences, about 525 rows): KDE Plasma, GNOME, systemd, storage, networking and homelab. The focus is names a recogniser writes as an ordinary English word: Konsole as console, Okular as ocular, Thunar as thunder, LUKS as looks.

• Smart home (Smart-Home-Sentences, about 120 rows): Home Assistant, Zigbee, Z-Wave, Matter, Thread, ESPHome, MQTT and device brands, including product names that are also plain words (Matter, Thread, Hue, Nest, Assist).

• Personal vocabulary (Personal-Vocabulary-Sentences, about 180 rows): my own name, my company and my project names, including the podcast My Weird Prompts and its hosts Corn and Herman Poppleberry. It is the most personal set and the least reusable; it is there as a pattern for building your own.

• Homophones (Homophone-Context-Sentences, about 156 rows): sentences where context decides the spelling (their and there, principal and principle, and 58 more sets).

• Paragraph dictation (Paragraph-Dictation-Passages): passages of two to four paragraphs, such as emails, prompts and meeting notes, dictated in one take with natural pauses. It has only three passages so far. It is not used for any Hebrish model; it exists for a separate model that puts paragraph breaks back into transcripts, and its `paragraph_text` column holds the labelled version.

The Hebrew set

English-Hebrew-Codeswitching-Sentences is the one behind the Hebrish models: English sentences with everyday Hebrew words and Israeli names dropped in, read aloud. It is the most downloaded of the family, with about 1,300 downloads. Like the master dataset it is gated, so you accept the terms on Hugging Face before loading it. The earlier, smaller set, English-Hebrew-Mixed-Sentences, is still up. For how these were used to train Jachnun and Sabich, see the Hebrish write-up.

An older set and the voice-note sets

Tech-Sentences-For-ASR-Training (titled TechVoice on its card) was my first attempt at a technical-speech dataset: 205 recordings, about 39 minutes and 10,400 words, MIT-licensed, with train, validation and test splits. Its card sets a goal of five hours and it is marked work in progress; the newer Tech set above covers the same ground.

Separately, there are three small sets of real, unscripted speech, which is a different job from read-aloud sentences:

• Annotated-Voice-Notes: a deliberately tiny, hand-picked set of real voice notes, each with its audio, a raw AssemblyAI transcript and a written note on why it was included. At the time of writing it holds a single recording.

• Voice-Notes: 65 voice-note transcripts from a Voicenotes export, screened by a language model for private content, with identifiers replaced by surrogates and every note approved by me by hand. The transcripts are raw ASR output, kept uncorrected on purpose, for evaluating a voice-note intent classifier.

Which one to use

• Adapting a model to a voice or accent without adding vocabulary: General English.

• Developer dictation: Tech, plus AI interactions if you dictate to coding agents. Add Linux admin or Smart home if you work in those areas.

• Names that collide with ordinary words: Linux admin and Smart home are good examples of the pattern; Personal vocabulary shows it for one person.

• Code-switching between English and Hebrew: the Codeswitching set.

• Paragraph or voice-note research: Paragraph dictation and the voice-note sets, with the caveat that they are tiny.

Limits

These are all one person, one accent and one phone. Models trained on them get better at that speaker's speech; Fritter's own numbers show it ends up roughly level with stock models on other people's. Several sets are small, especially Smart home, Personal vocabulary, Homophones and the paragraph and voice-note sets, so treat any score from their test splits as indicative only. They are CC-BY-4.0 with sentence-text attribution recorded per row in a `prompt_source` column, as set out in the master dataset's licence section.

If you want to build your own, the pattern is the useful part: write sentences around the words your recogniser gets wrong, record them in your own voice, hold out whole terms for testing, and keep one master dataset with a category on every take. The wider picture of my Hugging Face account, with the models these fed, is in the October 2026 roundup.