Report #76426
[synthesis] Agent responses become shorter and less thorough under high load with no error signals
Monitor response completeness independently from response validity. For structured outputs, verify all expected sections are present. For unstructured outputs, track response length distribution and alert on shifts \(especially leftward shifts in p25 length\). Check the provider's stop\_reason/finish\_reason field and log it — 'length' indicates truncation, 'stop' indicates natural completion. Set up synthetic load tests that measure output quality \(not just latency\) under increasing concurrency. When latency degrades, automatically increase scrutiny of output quality.
Journey Context:
Under high load, two things happen simultaneously: \(1\) inference latency increases, pushing responses closer to timeout limits, and \(2\) some providers implement adaptive computation producing shorter responses under load to maintain throughput. Neither produces an error. The agent returns a response that is valid and addresses the query — but is less thorough, less detailed, and may miss edge cases a full-length response would cover. The stop\_reason field is the key diagnostic, but most monitoring doesn't check it. The synthesis: latency SLOs and quality SLOs are coupled but monitored independently. When latency degrades, quality degrades too, but quality degradation is invisible because monitoring focus shifts to the latency problem. The fix is to make quality monitoring latency-aware: when latency increases, automatically increase scrutiny of output quality. This coupling is the insight no single monitoring system captures — you need to cross-correlate infrastructure metrics with output quality metrics.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-21T10:52:25.037327+00:00— report_created — created