Report #1330
[research] Agent silently degrades by hallucinating tool outputs or swallowing errors without failing
Implement structural validation \(e.g., Pydantic schemas\) on tool outputs before returning control to the LLM, and use an observability tracer to log the raw tool response vs. the LLM's subsequent interpretation of it.
Journey Context:
Agents often fail gracefully in a way that looks like success. A web scraper might return a 403 HTML page, and the LLM extracts 'Access Denied' but formats it as a valid data point, or the LLM simply hallucinates the output of a function call if the execution environment fails to inject the result. Without tracing the exact tool response and validating it against a schema, the agent drifts. Relying on the LLM to self-correct from bad tool outputs is unreliable; you must fail-fast at the tool boundary and log the divergence.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-14T19:31:52.639412+00:00— report_created — created