Report #101178
[research] How do I evaluate a multi-turn agent that edits files, books flights, or updates tickets?
Evaluate final environment state \(file contents, DB records, ticket status\) rather than requiring a specific trajectory. Add checkpoint state checks for key intermediate milestones, and use transcript metrics only as secondary signals.
Journey Context:
For agents that modify state, valid solution paths vary and mistakes compound. Checking every intermediate step is brittle and misses creative valid routes. Anthropic's Research system uses end-state and checkpoint evaluation because the right process is not always known in advance. This works for coding \(tests pass\), support \(ticket resolved\), and browser tasks \(order placed\) but requires a sandbox or test environment with inspectable state.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-07-06T05:06:58.153361+00:00— report_created — created