Posts

Main takeaways What I found: Judge reliability is judge-dependent, not just task-dependent: on identical stimuli, Gemini 3.1 Pro’s scores track error magnitude on 11 of 12 dimensions, GPT-4o’s on 6, and Qwen2.5-VL’s on 3. Where judges do penalise errors, they start late: onsets sit at or beyond the magnitude that’s clearly visible to a human, and several error types are never reliably penalised at all. Verifying an instructed value (is this the right hex colour, the right angle?) is a harder task for a judge than noticing an incidental anomaly. Colour verification fails outright for every judge tested, at every magnitude. Holistic “overall quality” scores diverge sharply across judges on identical, unedited images (a 2.5-point spread on a 5-point scale) while decomposed rubric dimensions do not (0.41 points). Controlled-stimulus evidence that decomposed rubrics are the right design choice. Judges can see errors they don’t penalise: shown the same images under a reference-based framing instead of the benchmark’s instruction-based one, GPT-4o registers significantly more errors than it penalises. Contents ...

I recently published an audit of whether VLM judges can perceive the errors that text-in-image editing benchmarks ask them to penalise. This was my first scientific work, written under a thesis deadline, with not much prior work to build on. Most design decisions were mine to make and I had little to check them against. Below is what the process taught me. The thesis did not start as an audit The plan was a synthetic training pipeline for quantitative text style editing. Render designed documents from HTML/CSS in a headless browser, apply edits programmatically (“scale the title by 20%”, “shift the byline 40 pixels right”), and train on the resulting code-to-image pairs, where the correct answer is known to the pixel. The capability gap was somewhat documented: TextEditBench reports that models handle content edits well but fail on spatial and stylistic transformations. Rendered supervision looked like an obvious way at it that nobody had taken. ...