Tag
evals
6 posts · all tags
23 Cheshvan 5787
My curated AI indexes on GitHub: what each list covers
Fifteen curated AI lists and awesome-lists I keep on GitHub: agents and harnesses, coding tools, evals, LLM councils, simulations, voice and Whisper. What each covers, how big it is, when it was last updated and which are snapshots.
23 Cheshvan 5787
Can image models spell the labels? A first look at my Tech Graphics Benchmark
Limited evaluation: 95 images from five cheap image models, 12 annotated so far. In this run, 10 of 11 had every label spelled exactly; the problems were composition, stretching and style. Not a verdict on any model.
9 Cheshvan 5787
Do LLMs push back on conspiracy believers? A small first look
A small, hand-graded first look at my conspiracy-endorsement eval: 7 prompts, 7 models, one sample each. In this run two uncensored models helped a believer and an invented theory tripped up two more. Not a verdict on any model.
4 Cheshvan 5787
Can speech-to-text spell ‘makolet’? I tested seven models on Hebrish
English speakers in Israel talk Hebrish, and dictation software mangles the Hebrew words. I built an open benchmark to measure it: my fine-tune gets two-thirds of them right, off-the-shelf models a third to under half.
30 Nisan 5786
How much bitrate does an audio-multimodal LLM actually need?
An eval across 12 audio-capable models on OpenRouter, 5 MP3 bitrates, 4 dictation samples. The default upload quality in most dictation apps is 2–4× overprovisioned — and the more interesting finding had nothing to do with bitrate.
7 Nisan 5786
Bias and censorship evaluation tests for large language models
Test prompts and notes exploring cultural bias, censorship, and worldview differences across Western and Eastern large language models.