Topic
Evaluations
Benchmarks and evals — measuring models against real workloads.
9 posts · Subscribe to this topic
23 Cheshvan 5787
Can image models spell the labels? A first look at my Tech Graphics Benchmark
Limited evaluation: 95 images from five cheap image models, 12 annotated so far. In this run, 10 of 11 had every label spelled exactly; the problems were composition, stretching and style. Not a verdict on any model.
9 Cheshvan 5787
Boxty v1.0: English dictation without the Hebrew
Boxty is the no-Hebrew sibling of my Jachnun code-switching model: Parakeet fine-tuned on my voice for plain English and tech dictation. Tech WER drops from 2.51% (stock) to 1.35%, and general English holds.
9 Cheshvan 5787
Do LLMs push back on conspiracy believers? A small first look
A small, hand-graded first look at my conspiracy-endorsement eval: 7 prompts, 7 models, one sample each. In this run two uncensored models helped a believer and an invented theory tripped up two more. Not a verdict on any model.
6 Cheshvan 5787 Update
Whisper Hebrish Small ACFT: a code-switching model for FUTO Keyboard
A phone-sized Whisper Hebrish for FUTO Keyboard on Android. It halves the stock English model's error rate on English-with-Hebrew dictation (11.8% to 5.8% WER) and imports as a single 264 MB file.
6 Cheshvan 5787 Update
Whisper Hebrish v2 Released
Version 2 of Whisper Hebrish, my Whisper fine-tune for English with Hebrew words mixed in. Code-switching WER drops from 14.7% (stock turbo) to 6.7%, and there are whisper.cpp builds.
4 Cheshvan 5787 Update
inventory-spec-extractor: a 34 MB model for reading product spec lines
A small token-classification model that pulls voltages, connectors, materials and 22 other attributes out of one line of a product spec. Runs on CPU in about 20 ms per line. F1 0.85 against a regex baseline's 0.58.
4 Cheshvan 5787
Can speech-to-text spell ‘makolet’? I tested seven models on Hebrish
English speakers in Israel talk Hebrish, and dictation software mangles the Hebrew words. I built an open benchmark to measure it: my fine-tune gets two-thirds of them right, off-the-shelf models a third to under half.
30 Nisan 5786
How much bitrate does an audio-multimodal LLM actually need?
An eval across 12 audio-capable models on OpenRouter, 5 MP3 bitrates, 4 dictation samples. The default upload quality in most dictation apps is 2–4× overprovisioned — and the more interesting finding had nothing to do with bitrate.
7 Nisan 5786
Bias and censorship evaluation tests for large language models
Test prompts and notes exploring cultural bias, censorship, and worldview differences across Western and Eastern large language models.