I recently published an audit of whether VLM judges can perceive the errors that text-in-image editing benchmarks ask them to penalise. This was my first scientific work, written under a thesis deadline, with not much prior work to build on. Most design decisions were mine to make and I had little to check them against. Below is what the process taught me.

The thesis did not start as an audit

The plan was a synthetic training pipeline for quantitative text style editing. Render designed documents from HTML/CSS in a headless browser, apply edits programmatically (“scale the title by 20%”, “shift the byline 40 pixels right”), and train on the resulting code-to-image pairs, where the correct answer is known to the pixel. The capability gap was somewhat documented: TextEditBench reports that models handle content edits well but fail on spatial and stylistic transformations. Rendered supervision looked like an obvious way at it that nobody had taken.

Scoping the project turned up a second problem. I had no good way to evaluate what I wanted to train. TextEditBench existed, but it scores content editing, style editing, and “semantic expectation” under one aggregate benchmark, and the style editing I cared about was in there, mixed in with the rest and not separable from it. So evaluation entered the plan as a supporting component: a programmatic checker that would pull bounding boxes, stroke thickness, colour, and similar measurable properties out of the rendered output. I got some way into building it and then stopped as the pivot below took the time it would have needed.

Midway through implementation, Tencent’s WeEdit came out and demonstrated the core of my proposal. HTML+CSS code-to-image pairs, glyph-guided, producing state-of-the-art text editing with far more compute than a master’s thesis commands. What it did not demonstrate was how anyone knew. WeEdit scores its benchmark entirely with a VLM judge, and TextEditBench does mostly the same. Neither validates the judge. That measurement question was untouched, including underneath WeEdit’s own results, and the rendering pipeline I had built for training data was most of an instrument for answering it.

Evaluation looked like the easy half

Train a model and you run into the “now how do I evaluate this” problem. Here I was building the evaluation itself, so I’d know what it measured. Generate degraded stimuli, run the judges, see what the data says.

That was naive. Only afterwards did I realise the evaluation needs evaluating too, and that this is the thing you should work out before you start, at a point where you might not yet know precisely enough what you’re measuring. In this work, grounding the degradation magnitudes in human-perceptible tiers ended up doing most of that job. Doing it properly needs human annotation, which is separate work and needs time I didn’t have.

The lesson I am taking for the future is about the final figure. It feels counterintuitive to picture your results before you have any, since deciding in advance what the data will show is exactly how you bias yourself. But there’s a difference between fixing the axes and fixing the curve. If you have the figure in mind, you can work backwards from it: what will this actually be able to say, what will it not be able to say, what would I have to build for it to say more. That surfaces limitations while you can still do something about them.

On reading evaluations before writing one

I landed on this topic because the evaluation in these papers didn’t look good enough, and for this specific work that held up. Benchmarks in this space use VLM judges and nobody had checked whether the judges work and that was the gap. I also had a broader version of that judgment that I’d built up from reading: that evaluation practice generally wasn’t handled well. This judgment did not hold long.

I noticed a pressure only after I was writing my own limitations section. Research gets communicated by making a case, and every scope condition you state plainly gives a reason to set the work aside. Narrowing a claim makes the finding harder to state cleanly and might feel like it rewards you nothing in return. So the incentive is to write the limitation in as few words as you can and move on. And part of it is just ego and attachment. It’s my work, I wanted the results to be strong, and writing the sentence that makes them weaker kind of hurts.

But the bigger thing I hadn’t understood is how limitations get diluted as they move through the literature, mostly without anyone necessarily doing anything wrong. If a paper uses a benchmark from earlier work, the limitations of that benchmark usually aren’t restated; it’s on the reader to go find the original and read them there, where they may already be understated for the reasons above. If a paper uses a method that’s standard in the field, the limitations of the method may not have been studied at all, or may have been studied on the side of some other paper, in which case the finding is buried somewhere in the literature and there’s no reasonable way for a reader to know to look for it. And when authors notice an evaluation gap and build an evaluation alongside their main contribution, that evaluation doesn’t get the attention the main work gets, so its limitations end up thin. None of these are failures by any individual author. Perfect research isn’t possible and there are always limitations. They just don’t propagate.

Still, as a reader, what I’d want is for papers to say what they evaluated and what they didn’t, and treat that as part of the result. Evaluations aren’t valid in general, only for particular claims. But having tried to write one, I understand why that part is often on the light side.

Miikael Mändmets, University of Turku.