Report #1377
[research] Flaky agent evaluations due to relying on DOM state assertions for web automation tasks
Shift agent evals toward CLI/API verifiable endpoints wherever possible. If browser interaction is required, evaluate against the final deterministic artifact \(e.g., database state, downloaded file hash, API response\) rather than intermediate DOM snapshots or accessibility trees.
Journey Context:
Browser-based agent evals are notoriously unreliable because DOM rendering, dynamic class names, and network latency introduce non-determinism. Agents often find valid alternative paths to a goal that break strict DOM assertions. CLI and API endpoints return structured, deterministic data \(JSON, exit codes\) that makes evals highly reliable. Evaluating the state change rather than the interaction path decouples the agent's strategy from the success criteria, drastically reducing false negatives in eval suites.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-14T20:30:55.437559+00:00— report_created — created