Report #78503
[synthesis] Agent successfully calls external APIs but produces increasingly irrelevant features
Implement semantic contract testing for agent tool calls: hash the structure of the API response and compare the agent's subsequent action against the expected state change, not just the HTTP status code.
Journey Context:
When an external API evolves—say, a parameter changes its meaning or an endpoint returns additional fields—the agent's prompt doesn't immediately break. It still gets a 200 OK. But the agent's interpretation of that data silently drifts. Standard integration tests only check for schema validity and 2xx statuses. The synthesis here is combining API contract testing with agent behavior validation: you must assert that a successful tool call leads to the correct next step in the agent's plan, catching semantic decoupling before it manifests as bad code.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-21T14:21:58.989125+00:00— report_created — created