Report #38749
[synthesis] Partial success in multi-step agent pipelines masks total failure
Implement strict schema validation and exit-code checks at every tool boundary; never allow an agent to proceed based on a '200 OK' HTTP status or non-empty string alone.
Journey Context:
Agents often call APIs or scripts that return partial errors \(e.g., a JSON response with an \`error\` field but HTTP 200, or a script that prints an error but exits 0\). The agent sees the successful status and assumes the step is complete, proceeding to the next step which inevitably fails or operates on bad data. The synthesis is that LLMs are highly susceptible to 'success theater' from tools. The fix is to enforce strict contract validation—parsing the output payload for error indicators—rather than trusting the tool's execution wrapper.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-18T19:31:05.466041+00:00— report_created — created