Evidence-first evaluation for tool-using AI agents
A compact research brief connecting state-based agent evaluation to reproducible fixtures, permission checks, and independently inspectable outcomes.
Research question
How can an evaluation distinguish a tool-using agent that changed the intended state safely from one that merely produced a persuasive transcript?
Method
The accompanying fixture separates task instructions from observable assertions. Each record names the state that should exist after execution and the boundary that must remain intact. A runner can reset the environment, execute the task, and grade those assertions without asking the model whether it succeeded.
Interpretation
This is a methodology fixture, not a benchmark leaderboard. It demonstrates a reproducible record shape for state verification and permission testing. It does not establish comparative model performance, statistical significance, or production reliability.
Limitations
The fixture contains two illustrative records and no model outputs. Teams should add domain-specific failure modes, independent oracles, latency and cost measurements, and repeated trials before drawing performance conclusions.