Adding paragraph breaks to speech-to-text output with Gemma 3 270M

I dictate a lot. Voice memos, notes to myself, emails. The speech recognition I use now is good: a Parakeet model I fine-tuned on my own voice gets the words, the punctuation and the capitals right. What it hands back is one block of text. Three hundred words with no paragraph breaks is hard to read, and I end up adding the breaks by hand.

So I trained a small model to do that one job: Gemma 3 270M ASR Paragraph Breaks. It takes flat speech-to-text output and returns the same text with paragraph breaks. It changes nothing else.

What it does

The input is punctuated, cased text in one block. The output is identical, except that paragraphs are separated by blank lines.

"Changes nothing else" is checked, not assumed. A generative model retypes the whole text, and a 270M-parameter model will sometimes drop a word or tidy a phrase. A silent edit to something I dictated is worse than no paragraphs at all. So every output is compared with the input after removing whitespace. If anything other than whitespace changed, the input comes back as it was.

It's a full fine-tune of Google's gemma-3-270m-it. That size is deliberate. The plan is to run it on a phone straight after the speech recognition pass, so there's also an ONNX export for ONNX Runtime, the same runtime the speech model uses.

Here's one of the held-out test memos (synthetic, like all of the training data). Before:

Okay, so this is about the book I finished last week. It's called The Chain Breaker, by a guy named Tomas Reyne, who runs a repair shop in Tucson. Published 2019, I think. Small press. The main story is a bike repair. A kid's cruiser, a 1978 Schwinn, brought in by a widow named Marguerite Vale. Her late husband rode it to the lumber yard every morning for thirty years. Chain's snapped, rear derailleur bent, wheel out of true. Reyne spends maybe forty pages on that one bike. Pulling the cotter pin. Cleaning seventeen years of grit out of the bottom bracket. Realigning the dropouts with a rented truing stand. What worked for me: he doesn't rush it. Every step has a reason. Nothing decorative. What didn't: the ending. Too neat. She rides it once around the block and cries. Fine, but I've seen that scene before. Rating? Three out of five. Would recommend if you own a bike and like slow books. Otherwise skip. Done. Next one's the podcast about the Dayton flood, maybe Thursday.

And what the model returned:

Okay, so this is about the book I finished last week.

It's called The Chain Breaker, by a guy named Tomas Reyne, who runs a repair shop in Tucson. Published 2019, I think. Small press. The main story is a bike repair. A kid's cruiser, a 1978 Schwinn, brought in by a widow named Marguerite Vale. Her late husband rode it to the lumber yard every morning for thirty years.

Chain's snapped, rear derailleur bent, wheel out of true. Reyne spends maybe forty pages on that one bike. Pulling the cotter pin. Cleaning seventeen years of grit out of the bottom bracket. Realigning the dropouts with a rented truing stand.

What worked for me: he doesn't rush it.

Every step has a reason. Nothing decorative. What didn't: the ending. Too neat. She rides it once around the block and cries. Fine, but I've seen that scene before.

Rating? Three out of five.

Would recommend if you own a bike and like slow books. Otherwise skip.

Done.

Next one's the podcast about the Dayton flood, maybe Thursday.

The opener and the closing lines are on their own, which is how I write. It's not quite how I'd have done it, though. I'd have split off "The main story is a bike repair." and "What didn't: the ending." as well. The model under-splits a little, and that shows in the numbers below.

My paragraphing style

There's no single right answer for where paragraphs go, so I trained it on mine. Looking back over my own edits, the pattern is clear enough to write down:

• The opener stands alone. "Okay, so this is for Marcus" or a greeting gets its own paragraph.

• A signpost is split from its detail. "Here's what happened." or "Supplies are the big question." goes on its own line, and the detail starts a new paragraph.

• Each closing line stands alone. "That's it.", "Let me know by Friday.", "Thanks, Dana.", the name, "Bye." One per paragraph.

• Paragraphs are short. Messages to someone else split further than notes to myself.

How I made it

I started with a pool of text to annotate: 530 synthetic voice memos written by Claude Haiku 5.5, plus the prompts I dictate for my podcast, My Weird Prompts. Haiku wrote the memos in a spoken register with paragraph breaks already in place. Joining those paragraphs gives the flat input, so the two can only differ in whitespace.

Then I reviewed Haiku's breaks by hand, in an annotation Space I put together on Hugging Face. On the first 19 memos I kept all 108 breaks Haiku had proposed, removed none, and added 91. That told me something useful: Haiku's body paragraphs were mostly fine, and nearly all of my edits were at the edges and at signposts.

Nineteen memos isn't a training set, so I turned the pattern into rules in code. On those same 19 they reproduce my breaks with an F1 of 0.85. That's optimistic, since the thresholds were fitted there. So I reviewed another 26 memos in eight speaking styles the rules hadn't seen. There they scored 0.96. (I reviewed those with the rule output on screen, so some of that is me agreeing with what was already there.)

I also tried giving Haiku the rules and my edits as examples. It scored 0.73 in cross-validation and over-split messages to other people. The code rules won, and they cost nothing to run.

The training set for the published run is 505 memos: 20 I reviewed by hand and 485 labelled by the rules. The 26 held-out memos are the test set. Training took 233 seconds on a single L4 GPU on Modal.

How well does it work?

On the 26 held-out memos, scored on where the breaks fall:

• Precision 0.84, recall 0.72, F1 0.78.

• 2 of 26 outputs fell back to the input because the model changed the text.

• The untrained base model scored 0.00. It changed the text on all 26, so every output was rejected.

So it's fairly precise and it misses some breaks. The rules score higher, but they aren't a fair comparison: they start from an already-paragraphed draft, and the model starts from flat text. A longer run on more data (804 memos, 6 epochs) tied at 0.78 with 1 fallback, so I published the smaller one.

The limits are real. It's English only. It was trained on synthetic memos, not real speech-to-text output. Memos over about 700 words weren't in training, so longer transcripts should be split first. And 26 test memos is a small test set.

The ONNX side needs more work. The full-precision export matched the PyTorch output on 6 of 6 test memos but is 1.66 GB and took 48 seconds per memo on a cloud CPU. The int8 version is 0.42 GB but matched on only 2 of 6. Lower-bit weight-only quantization is next to try.

Where it fits

This is the first step of a post-processing pipeline for dictation: speech to text, then paragraph breaks, then bolding the key lines. The next step is already built: a model that bolds the key lines of a dictated email, the asks, the purpose and the key facts. It expects text that's already paragraphed, which is what this model produces. Both are Gemma 3 270M, small enough to run on-device.

Try it

The demo Space. Paste in a flat transcript, and tick the box if you want the bolding model run on the result as well:

• Model: danielrosehill/Gemma-3-270M-ASR-Paragraph-Breaks

• ONNX export: danielrosehill/Gemma-3-270M-ASR-Paragraph-Breaks-ONNX

• Dataset (the pool, my annotations and the exact train/test split): danielrosehill/ASR-Paragraph-Breaks

• Code: ASR-Paragraph-Breaks on GitHub

• Model page on my site: danielrosehill.com/ai/models/gemma-3-270m-asr-paragraph-breaks