Recently, I asked AI to help me polish a blog post.
It wrote quickly. The structure was complete, and the ideas were clear. When I said I wanted it to sound more professional, it added technical discussion. When I asked for academic support, it found recent papers.
The article grew more serious, but I began to hesitate: did it really need to be written this way?
There was nothing obviously wrong with the content. But what I had wanted to write was a piece in my own voice, one that would let readers see how I thought through a problem. After several rounds of revision, it was gradually becoming a research review that I would need time to digest myself.
My expectations were changing along the way, too. At the outset, I had not fully worked out what “professional” or “in-depth” should mean for this particular article.
That experience made me rethink what makes a good agent.
If I checked the requirements one by one, our collaboration had accomplished quite a lot: the article was written, the research was done, and the references were in place. Each individual request had a corresponding deliverable.
But when I put them together, I still had to ask: is this the article I want to publish?
It is easy to check what an agent has done. Explaining whether those actions have moved the user closer to their goal takes more work.
In this case, “add academic support” was an explicit request. Its purpose was to make the argument more credible. If adding research caused the article to lose its original voice, we needed to reconsider how to use that material.
A user’s new request is not always something to simply add on top. It may be a correction in direction, or even a reason to remove work that has already been completed.
For an agent that stays involved in a task, understanding these changes affects the quality of the entire collaboration. It needs to revise its understanding of the task as feedback comes in, rather than continually adding more to the result.
This also makes “done” more complicated. The article may have an ending, and the task may be marked complete, while the user still does not have something they can use.
So before designing an evaluation, I want to clarify something else: what judgment are we trying to make from the results?
Checking factual accuracy means verifying citations and supporting evidence. Assessing revision ability means observing whether the agent understands feedback and preserves content that still works. Judging whether it can take on a task from start to finish also means looking at how much work it leaves for the user.
These questions are related, but “the overall quality is good” does little to answer them all.
Only when we are clear about what we are evaluating can we know what evidence to collect, and how much a score can tell us.
An evaluation also needs to be fair to the conditions at the time.
In this writing process, some goals only became clear through discussion. Early responses should be judged against the needs that had already been expressed, rather than preferences that emerged later.
If the requirements were already clear and the result was still unsuitable, we need to identify the problem more precisely. Did the agent misunderstand? Did it forget an earlier constraint? Or did satisfying a new request undo something that was already working?
All of these might be described as “poor writing,” but they call for very different improvements. Providing more task information, adjusting the context, or changing the approach to execution may each help. First, though, we need to know where things went wrong.
Other requirements should not change with personal preference. Whether citations are real, facts are accurate, and conclusions are supported must be checked independently. A user liking a result is not enough to establish that these requirements have been met.
I think this distinction matters. It allows evaluation to account for specific needs while retaining necessary standards of judgment.
Even when the goal is clear and the agent gets it right this time, there is still more to observe.
Can it do the same task well a second time? If the user phrases the request differently, will it misread the intent? When information is missing, can it acknowledge uncertainty instead of filling the gaps with fluent prose?
One study of agent reliability examined consistency, robustness, predictability, and safety separately. Across the models and benchmarks it tested, gains in capability brought only limited improvements in reliability. It is a reminder that after learning what an agent can accomplish, we still need to understand the conditions under which we can rely on it. Related research
So who gets to define a good agent?
I believe the user’s actual purpose should be the starting point. Product teams need to be clear about which needs they intend to serve and how much responsibility they will take on. Engineering needs to understand the implementation and the limits of the system’s capabilities. Evaluation puts those expectations into concrete situations and tests whether the results bear them out.
Someone needs to be accountable for the final trade-offs, and there needs to be evidence on which to discuss them.
Take the request to make something “more professional.” One person thinks of terminology and references. Another thinks of more rigorous reasoning. Someone else expects the author to pinpoint the heart of the problem. When we apply the request to an actual article, it becomes easier to see that we are not all talking about the same thing.
I would rather build agreement through cases like these. Which facts must be correct? Which parts affect whether the result is usable? Where is there room for different choices? All of this can be discussed. The more specific our standards are, the better our chances of explaining disagreements later.
Those standards should also be open to revision. New situations may expose problems we had not considered, and users may begin asking for different things. What matters is recording why we made an adjustment, so the team can distinguish an improvement in the product from a change in how it is judged.
The value of evaluation also needs to show up in the choices we make afterward.
If an evaluation leaves us clearer about what to prioritize, able to explain why an improvement is worth investing in, and aware of what additional tests we need when the evidence is insufficient, then the scores have practical value.
Looking back, AI did help me make the article more complete. It was the later rounds of discussion that helped us work out what to keep and what we could remove.
That is also what I hope evaluation can help a team do: when a result already looks good, still have a way to judge whether it is ready to put in a user’s hands, and why we can be confident in that judgment.