Nobody checked whether the judge can see: auditing VLM-as-judge for text style editing

Main takeaways

What I found:

• Judge reliability is judge-dependent, not just task-dependent: on identical stimuli, Gemini 3.1 Pro’s scores track error magnitude on 11 of 12 dimensions, GPT-4o’s on 6, and Qwen2.5-VL’s on 3.
• Where judges do penalise errors, they start late: onsets sit at or beyond the magnitude that’s clearly visible to a human, and several error types are never reliably penalised at all.
• Verifying an instructed value (is this the right hex colour, the right angle?) is a harder task for a judge than noticing an incidental anomaly. Colour verification fails outright for every judge tested, at every magnitude.
• Holistic “overall quality” scores diverge sharply across judges on identical, unedited images (a 2.5-point spread on a 5-point scale) while decomposed rubric dimensions do not (0.41 points). Controlled-stimulus evidence that decomposed rubrics are the right design choice.
• Judges can see errors they don’t penalise: shown the same images under a reference-based framing instead of the benchmark’s instruction-based one, GPT-4o registers significantly more errors than it penalises.

Contents

• Why this went unchecked
• Designing the audit
• What I found
• What this audit measures
• If you are putting a model in the grading loop
• Artifacts

Show GPT-4o an image edit with a clearly visible error, and ask it to score the edit on a five-point rubric. It usually gives the same near-ceiling score it gives a correct edit.

Text-in-image editing benchmarks like TextEditBench and WeEdit delegate their quality scoring to VLM judges. Neither cites evidence that the judges can perceive the errors they are asked to penalise, and I could not find that evidence elsewhere.

For my master’s thesis at the University of Turku I measured it. I generated 634 degraded stimuli where the edited style result deviates from the correct edit by a known amount: a fixed pixel displacement, a known ΔE of colour error, an exact scaling deviation. That covers 12 error dimensions. A further 255 stimuli are controls. Three judge models scored them under TextEditBench’s own rubric and prompt structure, giving 1,901 judgements on the degraded set. The judges are the ones these benchmarks actually deploy: GPT-4o for TextEditBench, Gemini 3.1 Pro for WeEdit, plus Qwen2.5-VL 7B as the open-weight option.

Reliability depends on which judge you use. On identical stimuli, Gemini 3.1 Pro’s scores track error magnitude on 11 of 12 dimensions, GPT-4o’s on 6, and Qwen2.5-VL 7B’s on 3. Scores produced under different judges are not comparable, and swapping the judge silently changes what a benchmark measures.

Where judges do penalise, they start late. Onsets sit at or beyond the magnitude I calibrated as clearly visible to a human, and several error types are never reliably penalised at all. Neither Gemini nor GPT-4o could verify a rendered colour against an instructed hex value at any magnitude I tested. These benchmarks exist to measure fine-grained verification of instructed values, which is the discrimination the judges failed at outright.

Figure 1. The anecdote above, made concrete. And it isn’t just GPT-4o: this is a real stimulus from the corpus, the colour is clearly not the one that was instructed, and all three judges scored it a perfect 5/5.

The rest of this post covers how the audit was built, what came out of it, and at length what it cannot claim. A companion post covers what building it taught me about the field.

Why this went unchecked

TextEditBench scores content editing, style editing, and “semantic expectation” under one aggregate benchmark. Semantic expectation covers cases where an edit is implicitly expected to produce side effects, like a total line updating after a price changes. A benchmark measuring that many things at once gives you an aggregate capability number. It does not tell you which of those capabilities is weak, and the quantitative style editing this audit is about is mixed into the aggregate rather than separable from it.

Underneath that aggregate sits the judge. WeEdit (whose release is what made me pivot from the original synthetic-data-and-training angle; more on that pivot in the companion post) scores its benchmark entirely with Gemini 3.1 Pro, and TextEditBench does mostly the same with GPT-4o. WeEdit produced state-of-the-art text editing results, but the question of whether its scores mean what they appear to mean sits untouched underneath its own results.

Designing the audit

The methodology chapter runs about twenty pages. Four decisions carry most of the logic, and each one is a trade with a cost.

Synthetic degradations instead of real model outputs. The obvious design is to collect real editing-model failures, have humans label them, and measure judge agreement against those labels. I rendered parametric errors instead. Part of the reason is practical: rendered stimuli are cheap and I could generate as many as I needed at any magnitude. The larger reason is attribution. Real failures fail on several axes at once and have ambiguous ground truth, so when the judge disagrees with the label you cannot tell whether the judge is wrong, the label is wrong, or the disagreement sits on a dimension nobody controlled. A rendered stimulus deviates from correctness by exactly one known quantity on exactly one axis, so every disagreement points at something specific.

The cost is scope as two things separate this corpus from real use: the errors are clean and single-axis, and the documents are simple. Both are in the limitations, and I come back to both at the end.

Magnitude grading instead of binary pairs. A corpus of correct/incorrect pairs tells you whether a judge can separate good edits from bad ones. The more useful question for anyone consuming benchmark scores is how wrong an edit has to be before the judge notices. Every dimension is therefore sampled across tiers from sub-perceptual to unmissable, with the exact parameter value recorded, and the analysis fits score against magnitude the way a psychophysics study fits a detection curve. Concretely: for each (judge, error type) pair, I first test whether scores respond to magnitude at all with a Spearman rank correlation, then locate where penalisation begins by fitting a floored logistic regression that predicts the probability of a below-ceiling score from log-magnitude. The onset is the magnitude where that curve crosses 75%, floored at the judge’s own false-positive rate on flawless edits, with bootstrapped confidence intervals throughout. That turns “is this judge accurate” into “here is the error magnitude below which this judge is blind”, which is a number anyone putting a model in the grading loop can act on.

Replicating the deployed protocol. The judges run under TextEditBench’s rubric, prompt structure, scoring scale, and input format, at temperature zero. Improving the prompt would be a reasonable study, and I list several such ablations as future work.

One deviation is worth naming: I replaced TextEditBench’s Semantic Expectation rubric dimension with an overall-quality score, since semantic expectation doesn’t apply to single-property style edits.¹

Two controls. An instruction-not-followed control presents the unedited source as the “result”. This is the easiest possible case for the instruction-following dimension, since the image the judge is asked to check is pixel-identical to the one it is comparing against, and all three models handle it: instruction following scores near the floor (1.12 to 1.71). The dimension works.

A correct-edit control presents a flawless edit as the “result” and measures each judge’s false-positive rate, the score it docks from work with nothing wrong. That floor turned out to be load-bearing as GPT-4o scores 19.5% of flawless edits below ceiling and Qwen 76%, so a low score on a degraded stimulus means nothing until it is read against what that judge does to a perfect one. Every threshold in the thesis is estimated relative to this floor rather than to the top of the scale, and Qwen’s floor is high enough that it has to be dropped from the threshold analysis entirely.

Figure 2. What each judge does to a flawless edit and to an unedited image presented as an edit. The floor everything else is measured against.

What I found

Reliability is determined by the judge. On identical stimuli, Gemini 3.1 Pro shows a significant score-magnitude correlation on 11 of 12 error dimensions, GPT-4o on 6, and Qwen2.5-VL on 3. The stimuli do not change across judges, so this is the judge and not the task. For anyone comparing numbers across benchmarks: scores produced under different judges are measuring different things.

Figure 3. Score sensitivity to degradation magnitude, by judge and error dimension. Reliability tracks the judge, not the error type.

Penalisation starts late or never. Where onsets are locatable at all, they sit at or beyond the magnitude tier calibrated to be clearly visible to a human. Several dimensions never cross the reliability criterion inside the tested range. An editing model can produce a plainly visible style error and still receive a score at or near ceiling, so benchmark scores built on these judges likely overstate progress on quantitative style editing.

Verifying an instructed value is harder than noticing an incidental anomaly. Stimuli where the degraded property is the one the instruction targets, such as a scale instruction judged for scale error, show consistently later and less certain penalisation than the same property degraded incidentally alongside a different edit. Rotation shows the contrast most sharply. Incidental rotations are penalised at near-minimal tiers. Verifying an instructed angle stretches to roughly twenty degrees for Gemini, with confidence intervals covering most of the axis, and produces no threshold at all for GPT-4o. Colour is the surprising case: neither Gemini nor GPT-4o produced a usable threshold for verifying a rendered colour against an instructed hex value, at any magnitude tested.

Figure 4. Where penalisation actually begins, against the magnitude that’s clearly visible to a human. Rotation shows the split most sharply: penalised early when incidental, late or never when it’s the value being verified.

Holistic scores diverge where decomposed ones do not. On identical unedited images, the overall quality score I added spreads 2.5 points across the three judges (Gemini 1.12, GPT-4o 2.94, Qwen 3.62) while the four decomposed rubric dimensions stay within 0.41. Meaning the judges resolve “overall quality” in different ways. I could not find much guidance on rubric design for this task when I was building it. Both TextEditBench and WeEdit use decomposed rubrics, and this is controlled-stimulus evidence that they are right to.

Judges can detect errors they do not penalise. A subordinate diagnostic showed each judge the ground-truth and degraded images side by side, with no instruction, and asked only whether they differ. Under this reference-based framing GPT-4o registers significantly more errors than it penalises under the benchmark framing: 96 stimuli detected but not penalised against 61 in the reverse direction. Gemini trends the same way (53 against 37) though, without reaching significance. The result points at a research question: where ground truth exists, does letting the judge compare against it work better than making it infer correctness from the instruction?

What this audit measures

The audit measures how far a rendered-text style edit has to deviate from correctness, on one clean axis at a time, before a VLM judge running TextEditBench’s rubric scores it below what that same judge gives a flawless edit. It answers that separately for 12 error types, at graded magnitudes, for three judges, using each judge’s own false-positive rate as the floor.

It does not measure agreement with human judgement on real model outputs. Real editing models produce font bleed, hallucinated strokes, and compound failures spanning several dimensions at once. Single-axis errors are what make attribution possible, and they are also the easiest version of the discrimination task, so what the audit establishes is an upper bound on judge reliability. How judges handle messier failures is open, and answering it needs the paired human-annotation study I avoided.

The documents are simple. Layouts span an easy tier and a medium tier, and I planned a hard tier that I dropped once it was clear the judges were already failing on the simpler ones. That reasoning assumed complexity only makes detection harder, which I no longer think is safe. Clutter can mask a subtle change, and it can also expose one. A 20-pixel shift that moves through empty space in a sparse layout has nothing to be measured against, while the same shift in a dense layout breaks alignment with a neighbouring element and becomes obvious. This also connects to a deeper problem with the magnitude tiers. A tier is a fixed physical quantity, a pixel window for position or a ΔE range for colour, applied identically across every layout, and physical units are not perceptual ones. The tier I call “clearly visible” was set by me, by eye, on these simple layouts, with no multi-rater study behind it. Judges that do not penalise until the largest tiers are late against any plausible placement of that boundary, but the corpus cannot tell you how large the gap is, and it cannot tell you whether that boundary would sit at the same pixel count in a busier document.

A full fix is expensive. Layout-dependent tier boundaries would need human annotation for every layout, which is most of a research project on its own. A cheaper route may be to calibrate a VLM annotator against human labels on a small subset and use it to extend. There is some irony in using a VLM here given what the rest of this post says about them, but the reference-based framing is the one they handle better, and the graded magnitudes give a natural validation signal.

The results are also conditional on one prompt, one rubric, three judge instances, and one input resolution. Three judges are enough to show that reliability varies by judge.

If you are putting a model in the grading loop

Five things I’d take from this if I were designing an evaluation with a model in the grading loop:

1. Measure the judge’s false-positive floor before reading any score. What a judge gives a flawless output determines what a low score on a bad one means.
2. Grade magnitudes rather than pairs. The result should be an onset. The error size below which the judge is blind, not an accuracy.
3. Decompose the rubric. Holistic quality is where judges diverge most on identical inputs.
4. Treat verification of an instructed value as a harder (and a separate) ask than detection of an anomaly.
5. Measure the judge’s sensitivity profile before delegating scoring to it. It costs API calls, but on these dimensions, it would have said no.

The findings of this work hold, but the audit is not finished. Making it publishable takes a few concrete additions, mainly more judge models and more stimuli per cell so the confidence intervals tighten; the harness and corpus already exist, so that part is mostly API cost. Making it a benchmark the field could rely on for a clear signal of progress takes considerably more: a multi-rater perceptual study in place of the single-rater anchor, magnitude tiers calibrated per layout, and stimuli closer to real documents.

Artifacts

Everything is released.

Dataset: huggingface.co/datasets/Memand/vlm-judge-audit: source, ground-truth, and degraded images, with per-stimulus metadata down to the jitter-resolved degradation parameters.

Code: github.com/miikaelm/vlm-judge-audit: the generation pipeline, evaluation harness, and analysis code.

Running the audit against a new judge is API spend against the existing corpus. Extending it to new dimensions, magnitude ranges, or layouts means regenerating, which the pipeline does deterministically.

A companion post covers what building this taught me about reading evaluations.

Miikael Mändmets, University of Turku. Thesis: “Auditing the VLM-as-Judge Protocol for Text-in-Image Style Editing: Sensitivity to Controlled Degradations”, 2026.