Miikael Mändmets

Nobody checked whether the judge can see: auditing VLM-as-judge for text style editing

Thu, 30 Jul 2026 10:00:00 +0300

Main takeaways

What I found:

Judge reliability is judge-dependent, not just task-dependent: on identical stimuli, Gemini 3.1 Pro’s scores track error magnitude on 11 of 12 dimensions, GPT-4o’s on 6, and Qwen2.5-VL’s on 3.
Where judges do penalise errors, they start late: onsets sit at or beyond the magnitude that’s clearly visible to a human, and several error types are never reliably penalised at all.
Verifying an instructed value (is this the right hex colour, the right angle?) is a harder task for a judge than noticing an incidental anomaly. Colour verification fails outright for every judge tested, at every magnitude.
Holistic “overall quality” scores diverge sharply across judges on identical, unedited images (a 2.5-point spread on a 5-point scale) while decomposed rubric dimensions do not (0.41 points). Controlled-stimulus evidence that decomposed rubrics are the right design choice.
Judges can see errors they don’t penalise: shown the same images under a reference-based framing instead of the benchmark’s instruction-based one, GPT-4o registers significantly more errors than it penalises.

Contents

What building an evaluation taught me about reading them

Thu, 30 Jul 2026 09:00:00 +0300

I recently published an audit of whether VLM judges can perceive the errors that text-in-image editing benchmarks ask them to penalise. This was my first scientific work, written under a thesis deadline, with not much prior work to build on. Most design decisions were mine to make and I had little to check them against. Below is what the process taught me.

The thesis did not start as an audit

The plan was a synthetic training pipeline for quantitative text style editing. Render designed documents from HTML/CSS in a headless browser, apply edits programmatically (“scale the title by 20%”, “shift the byline 40 pixels right”), and train on the resulting code-to-image pairs, where the correct answer is known to the pixel. The capability gap was somewhat documented: TextEditBench reports that models handle content edits well but fail on spatial and stylistic transformations. Rendered supervision looked like an obvious way at it that nobody had taken.

About

Mon, 01 Jan 0001 00:00:00 +0000

About