From Perfect Traces to Broken Products: The New Paradigm for Evaluating AI Agents

The Mirage of the Perfect Trace

At the VB Transform 2026 conference, Harrison Chase (CEO of LangChain), Hui Zhang (CTO of Conviva), and Emmanuel Turlay (director of engineering at CoreWeave) agreed on a troubling diagnosis: a single conversation with an AI agent can appear flawless when evaluated in isolation yet hide systemic failures that make the product perceived as broken by users. This gap between individual evaluation and real-world experience is driving a paradigm shift in how companies measure their agents' performance.

According to Zhang, the fundamental dilemma is the tension between automated methods (such as LLM-as-judge or agent-as-judge) and human review. "You have the scalable but not grounded, whether it's agents as judges or LLMs as judges. You score the output, you score the work. It's still very hard to ground it, and then you use humans, but that's not scalable," he stated. "The whole industry is facing this: which poison do you want to choose."

Evaluation Criteria as the New PRD

Chase warned that many teams fall into "evaluation paralysis": trying to build a comprehensive test suite before launching anything, which delays real learning. "The best teams launch and then iterate," he said. For Chase, evaluation criteria should be understood as a living product requirements document (PRD): "Evaluations are the new PRD. They define what your agent should and should not do."

Turlay described a parallel experience: "I was trying to achieve 100% coverage in my tests, and I still had bugs in production." His solution was to prioritize broad, always-on monitoring, which captures more real failures than an exhaustive pre-launch test suite. He recommends first implementing broad online checks, identifying failure classes as they occur, and then building a specific offline evaluation set for the problems that arise.

The Mistake of Scoring Traces One by One

Zhang criticized the common practice of sampling traces (50 or the entire population) and scoring them in isolation. This approach misses a signal that only appears when comparing user cohorts against a baseline, a method he calls contrastive analysis. He gave an example from retail: a buyer asks for a half-marathon running shoe, the agent asks qualifying questions, and the user purchases. Individually, the interaction seems excellent. But when comparing cohorts, the clarification ratio (follow-up questions before task completion) turned out to be three times higher than the baseline for that footwear category. A second metric—how often buyers completed the purchase outside the conversation—was five times higher. Neither number is visible from a single trace. Both point to a debuggable, category-specific problem.

Zhang also noted that the industry lacks a second data source: what happens before, between, and after the conversation, not just the trace itself.

Matching the Judge to the Job

Once contrastive analysis identifies which category is truly broken, the next problem is which model will monitor it and at what cost. Turlay proposed a simple rule: use the smallest, cheapest model that can detect the specific failure class. "You don't need GPT-4 to detect a formatting error," he said. This trend toward cheaper, more specialized judge models runs parallel to the methodological shift toward cohorts.

Together, the three interventions draw a new map for agent evaluation: moving from individual traces to cohorts, from exhaustive pre-launch tests to continuous monitoring, and from large, generic judge models to small, specific ones. Companies that adopt this approach will be able to detect hidden failures that would otherwise go unnoticed, saving costs and improving user experience.