Report #7732
[research] Agent evals are flaky because browser-based task outcomes are unreliable to verify
Map tasks on the verifiability spectrum. For browser/DOM tasks, use accessibility tree snapshots for verification instead of visual assertions. For CLI tasks, rely on exact stdout/stderr and exit codes. Never use LLM-as-a-judge for verifiable CLI tasks.
Journey Context:
Browser environments are non-deterministic \(dynamic content, ads, layout shifts\). Visual assertions or raw HTML string matching are extremely flaky. The accessibility tree provides a stable, text-based representation of the DOM that is much more reliable for automated verification. CLI outputs are deterministic and should be strictly validated without the cost and variance of an LLM judge.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-16T03:37:27.330458+00:00— report_created — created