# Evidence-first evaluation for tool-using AI agents

> A compact research brief connecting state-based agent evaluation to reproducible fixtures, permission checks, and independently inspectable outcomes.

- Type: research-page
- Canonical URL: https://www.searchforagents.com/research/evidence-first-agent-evaluation
- Published: 2026-09-22
- Updated: 2026-09-22
- Topic: Search Infrastructure
- Reviewer: Search for Agents Editorial Desk
- Canonical Markdown SHA-256: 3bf53a275f2e759e20f8864c2b602746a55123525d519891e2520b3b490c3ca2
- Source bundle: https://www.searchforagents.com/api/v1/content/research_evidence_first_agent_evaluation/source-bundle

## Research question

How can an evaluation distinguish a tool-using agent that changed the intended state safely from one that merely produced a persuasive transcript?

## Method

The accompanying fixture separates task instructions from observable assertions. Each record names the state that should exist after execution and the boundary that must remain intact. A runner can reset the environment, execute the task, and grade those assertions without asking the model whether it succeeded.

## Interpretation

This is a methodology fixture, not a benchmark leaderboard. It demonstrates a reproducible record shape for state verification and permission testing. It does not establish comparative model performance, statistical significance, or production reliability.

## Limitations

The fixture contains two illustrative records and no model outputs. Teams should add domain-specific failure modes, independent oracles, latency and cost measurements, and repeated trials before drawing performance conclusions.

## Methodology references

- [methodology](https://www.searchforagents.com/methodology)

## Sources

1. [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://arxiv.org/abs/2310.06770) — arXiv (2023-10-10)
2. [AgentBench: Evaluating LLMs as Agents](https://arxiv.org/abs/2308.03688) — arXiv (2023-08-07)