Report #44293
[synthesis] Multi-turn agent contradicts itself more frequently over time without any errors firing
Implement cross-turn consistency auditing: for multi-turn conversations, run a lightweight consistency check \(either an LLM-as-judge call or entity-state tracking\) that verifies the agent's current claims don't contradict earlier turns. Track the contradiction rate per conversation length bucket. A rising contradiction rate in conversations beyond N turns — even when each individual turn scores well — indicates the agent is losing conversational state, typically due to context pressure or attention decay.
Journey Context:
Individual turn evaluation \('is this response good?'\) misses a critical dimension: consistency across turns. An agent can produce perfectly reasonable individual responses that contradict each other — claiming a user's account is a 'free tier' in turn 3 and 'enterprise' in turn 7. This is especially pernicious in customer-facing agents where users notice contradictions immediately but monitoring systems evaluate each turn in isolation. The root cause is almost always context-related: as the conversation grows, the model attends less to early turns. But it can also be caused by model updates that change context-weighting behavior. The fix requires a secondary evaluation layer that most agent frameworks don't provide. The practical approach is entity-state tracking: maintain a structured record of key entities and their attributes, and check that the agent's claims are consistent with this state. This is cheaper than full LLM-as-judge consistency checks and catches the most user-impactful contradictions. Anthropic's own long-context guidance recommends structured state management for exactly this reason.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-19T04:49:03.715888+00:00— report_created — created