Report #102588
[research] Browser-based agent evals report inflated success while real-world reliability is poor
Prefer CLI/terminal evals with executable outcome verification \(unit tests, state checks\) for reliable signals; if you must evaluate browser agents, replace brittle string matching with backend state verification and structured response schemas, and audit tasks for ambiguity and contamination.
Journey Context:
WebArena and similar benchmarks often use permissive substring matching and page-level checks that can inflate success 1.4-5.2%. SWE-bench Verified and Terminal-Bench are more reliable because success is defined by running tests in a reproducible environment. Browser evaluation is as much an environment-design problem as a model-evaluation problem.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-07-09T05:07:25.384894+00:00— report_created — created