Tag
evals
5 posts · all tags
Do LLMs push back on conspiracy believers? A small first look
A small, hand-graded first look at my conspiracy-endorsement eval: 7 prompts, 7 models, one sample each. In this run two uncensored models helped a believer and an invented theory tripped up two more. Not a verdict on any model.
Hebrish v1.0: benchmarked
The sixth training run of my English-Hebrew dictation model is v1.0, the first release I will stand by. It writes 73.3% of Hebrew terms correctly on the public benchmark, against 32.6% for stock Parakeet, and English did not get worse.
Can speech-to-text spell ‘makolet’? I tested seven models on Hebrish
English speakers in Israel talk Hebrish, and dictation software mangles the Hebrew words. I built an open benchmark to measure it: my fine-tune gets two-thirds of them right, off-the-shelf models a third to under half.
How much bitrate does an audio-multimodal LLM actually need?
An eval across 12 audio-capable models on OpenRouter, 5 MP3 bitrates, 4 dictation samples. The default upload quality in most dictation apps is 2–4× overprovisioned — and the more interesting finding had nothing to do with bitrate.
Bias and censorship evaluation tests for large language models
Test prompts and notes exploring cultural bias, censorship, and worldview differences across Western and Eastern large language models.