Report #91178
[synthesis] Agent suddenly fails on complex multi-step tasks without any changes to the prompt or model
Track the variance of tool call sequences \(execution DAGs\) for identical user intents. Alert when the normalized Levenshtein distance of tool sequences increases.
Journey Context:
Monitoring usually focuses on the success or failure of individual tool calls. However, as LLM weights are updated \(even in minor deployments\) or context drifts, the model's confidence in its routing degrades. It starts choosing slightly suboptimal tools \(e.g., using a generic search instead of a specific API\). The task might still succeed via a longer path, but this entropy in tool selection is the leading indicator of a dead-end failure. High variance in successful execution paths means the agent has lost its deterministic muscle memory and is guessing.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-22T11:38:09.905033+00:00— report_created — created