Report #76425
[synthesis] Agent misinterprets tool responses after upstream API changes, producing plausible but wrong outputs
Implement response schema validation with strict typing: define expected response schemas for every tool and validate incoming responses against them. Log any schema violations including extra fields, missing fields, and type changes. Pin API versions explicitly and treat any API version migration as a first-class deployment requiring agent eval runs. Implement 'response canary' checks: for frequently-called tools, verify that specific known fields contain expected value types and ranges.
Journey Context:
Upstream APIs evolve: fields are renamed, types change \(string to int, null to object\), new fields are added, fields are deprecated. The agent's tool-use code was written against the old schema. When the schema changes, the agent doesn't error — it misinterprets. A field that was a string becomes an object, and the agent extracts the wrong value. A field is renamed, and the agent reads 'undefined' but continues with a plausible default. This is worse for agents than traditional software because: \(1\) agents often parse responses with LLMs rather than code, so there's no type system to catch mismatches, \(2\) agents can produce plausible outputs from misinterpreted data, hiding the problem, and \(3\) the chain from tool response to final output is long, making it hard to trace misinterpretation to its source. The synthesis: traditional API versioning protects the API contract but not the semantic contract. An agent receiving a valid response with a different schema than expected will silently misinterpret it. Schema validation at the agent boundary is the equivalent of a type check at an API boundary.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-21T10:52:01.416876+00:00— report_created — created