Can image models spell the labels? A first look at my Tech Graphics Benchmark

I keep needing graphics for blog posts and repo READMEs, and I keep wondering whether an image model can now make one without getting the words wrong. Diagrams-as-code (Mermaid, Graphviz) are accurate and, to be honest, a bit dull. An image model can give a diagram some character, but text rendering has long been the weak spot, and in an explainer one wrong label is fatal.

So I started the Tech Graphics Benchmark: a small, periodic check of whether current image models can produce a banner or a restyled whiteboard with every label spelled exactly and every arrow where it was asked for. This is a first look at the data from its first four rounds. It is a limited evaluation: 95 images generated, 12 of them annotated so far. In this run, the text was spelled exactly in 10 of the 11 annotated images where that check applied; what let the images down was mostly something else.

Setup

The benchmark has two modalities, graded separately because they fail differently. Text-to-image gets a written description with exact label strings. Image-to-image gets a photo of one of my whiteboards (from my Whiteboards dataset) and is asked to restyle it, for example into a chalkboard or a comic page, without dropping, inventing or rewording any label. There is also a Hebrew track, gated by a one-word screen, that has not been run yet.

As of 2026-10-28 there are 18 test prompts (ten text-to-image, five image-to-image, three Hebrew), and five more were retired along the way, for example anime-style asks that didn't match anything I would publish. The prompts, with the expected strings for each, are in Tech-Graphics-Benchmark.

Four rounds were generated on fal on 2026-10-28, all with the same small panel of cheap models: FLUX.3 Klein, GPT Image 2.5 Flare, Ideogram V4.5, Qwen-Image 3 and Nano Banana 2.1, with Seedream 5 Flash added for the whiteboard (edit) round. That is 95 images in total, about $2.51 at list price. Every image is in Tech-Graphics-Benchmark-Runs, shuffled and with opaque IDs so that I was blind to the model when grading.

Grading is by hand, in a private Gradio Space. Each image gets pass/fail ticks (text spelled exactly, text legible, structure, meaning preserved, where they apply to the prompt), 1 to 10 ratings for style, whether I would publish it, and value for money, and free-text notes. The raw annotations are in Tech-Graphics-Benchmark-Ratings.

What the data shows so far

Only 12 images have been annotated: 4 from the informal first round (two prompts that I then retired) and 8 from the banners round. The whiteboard and informational rounds are generated but not yet graded, so there is nothing to say about image-to-image yet, and nothing at all about Seedream 5 Flash (5 images, none annotated).

Text. 10 of the 11 annotated images with a text check had every expected string spelled exactly. The one miss was a Qwen-Image 3 image of a bus-stop scene, which I noted as probably a technical failure rather than a bad reading of the prompt. In this run, then, exact text was not the main problem for any of the five models, which is not what I expected when I started.

Where the images failed instead. A Qwen-Image 3 diagram passed every tick but the text was stretched to fill the frame, which made it unusable (my note: the benchmark needs an aspect-ratio criterion). A FLUX.3 Klein sloth-and-capybara banner had the right terminal text but an impossible arrangement of the two animals around the laptop, so I rated it 1 out of 10 for publishing. Two images in the informal round, from Ideogram V4.5 and Nano Banana 2.1, passed every validation tick and I still rated them 2 and 1 out of 10, because the anime style looked like something for a kindergarten wall rather than a tech blog. Text fidelity alone didn't make an image usable.

What I liked. The neon README title with a pixel-art sloth (`tgb-t2i-08`) got the highest ratings in this run: 8 for GPT Image 2.5 Flare, 7 for Nano Banana 2.1 and 7 for FLUX.3 Klein, with 5 for Ideogram V4.5 (vertical spacing between the sloth and the title was awkward). On the Jerusalem bus-stop prompt, the Nano Banana 2.1 image got the two-row LED board right, with only local details off (the line number and destination sit above the windscreen on real buses).

With one to four annotations per model, on different prompts, these numbers don't compare the models with each other, and I'm not going to pretend they do. The chart shows the individual annotations for that reason.

What surprised me, and what I'd change

I built the rubric around text accuracy, because that was the known weakness, and the first data says the rubric is missing things: aspect ratio, physical plausibility of a scene, and whether the style suits a technical blog at all. I have a process-notes box in the annotation form for exactly this, and these three are now on the list for the next revision of the form.

The cost side is the other thing I'll be watching. The list prices in these rounds run from $0.009 to $0.044 per image. If the cheapest models keep spelling labels correctly, the price of a usable banner is a few cents, and the benchmark's cost-per-usable-image figure (price divided by the share of images passing every tick) will matter more than any single rating. I can't compute it from 12 images.

Caveats

This is a limited evaluation, and I'd treat it as a progress report on a dataset rather than a result. 12 annotations, one annotator (me), one to four images per model on different prompts, list prices rather than billed costs, and a rubric still being revised: ticks were added on 2026-10-28, so earlier annotations may lack some fields. Nothing here says how any of these models behaves in general, and it is not a verdict on any model. The remaining 83 images, and the image-to-image and Hebrew rounds, may well change the picture.

Data and code

The benchmark is deliberately lightweight: a few cheap models, a few prompts, one modality at a time, every few months, because models change too fast for an exhaustive comparison to stay useful. All three datasets are public under CC BY 4.0.

• Source repo, design and grading rules: Tech-Graphics-Benchmark on GitHub

• Test prompts: danielrosehill/Tech-Graphics-Benchmark

• Generations (images and manifests): danielrosehill/Tech-Graphics-Benchmark-Runs

• Raw human annotations: danielrosehill/Tech-Graphics-Benchmark-Ratings