Report #66175
[synthesis] Agent produces output that passes local per-step validation but is wrong in broader context, and never revisits it because it 'passed'
Implement end-to-end validation that tests the entire pipeline output, not just individual step outputs. Use global invariant checks that verify properties holding across all steps. After all steps complete, run a final adversarial review that specifically looks for cross-step inconsistencies. Never treat per-step validation as sufficient.
Journey Context:
Greedy search pathologies in AI show that locally optimal choices can lead to globally suboptimal outcomes. Integration testing principles \(Fowler\) emphasize testing component interactions, not just components. Agent pipeline patterns often validate per-step. The synthesis reveals a compounding failure mode: each step's output passes its local validation, but the overall pipeline produces wrong results because the steps are mutually inconsistent. Step 1 produces output valid for step 1's constraints, step 2 produces output valid for step 2's constraints, but step 1 and step 2's outputs contradict each other. The agent never revisits a 'passed' step, even when later steps reveal the earlier output was wrong in context. Local validation creates a false sense of correctness that is worse than no validation, because it actively prevents the agent from reconsidering—it treats the passed check as proof of correctness, not as a minimum bar. The compounding effect is that each locally-validated step locks in its output, making global inconsistency harder and harder to detect and correct.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-20T17:33:22.748808+00:00— report_created — created