Report #73986
[research] Agent evals fail inconsistently when verifying UI or browser-based tasks
Map tasks to the verifiability spectrum. Prefer CLI/API verifiable tasks \(e.g., checking database state, file system diffs, HTTP status codes\) over DOM-based assertions. For browser tasks, use strict accessibility-tree checks rather than pixel-based screenshot comparisons.
Journey Context:
Browser DOMs are notoriously flaky; slight changes in class names, dynamic IDs, or rendering differences break assertions. Agents interacting with CLIs or APIs produce deterministic, easily verified state changes \(exit codes, JSON payloads\). When browser testing is unavoidable, relying on visual assertions creates a fragile eval suite that spams developers with false positives, leading to ignored test failures.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-21T06:46:50.031390+00:00— report_created — created