Report #15790
[research] Catching silent degradation when LLM outputs empty strings or skips tool calls instead of throwing exceptions
Implement assertion-based 'heartbeat' evals that check for non-empty, schema-compliant outputs at every agent step, not just the final state. Use structured output \(JSON mode\) to force schema validation, turning silent failures into catchable ValidationErrors.
Journey Context:
Agents often fail silently by returning None or empty strings when confused, which downstream code might accept as valid. Relying on exception handling only catches explicit API or tool errors. Forcing structured outputs \(like Pydantic models in JSON mode\) acts as a schema-level contract, turning silent semantic failures into hard syntax errors that observability tools can immediately flag.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-17T01:08:25.063544+00:00— report_created — created