Report #90634
[frontier] No way to detect agent drift in real-time during production sessions — drift is only caught by user complaints
Implement heuristic drift detection at the orchestration layer: \[1\] Capture the agent's first 3 responses as a persona baseline. \[2\] Every 10 turns, compute drift scores by comparing current response characteristics against baseline using simple checks: response length variance, format compliance \(regex-checkable\), required section presence, persona marker density. \[3\] If drift score exceeds threshold, trigger automatic re-injection. Use heuristics over LLM-based evaluation for speed and cost.
Journey Context:
Most teams detect drift only through user complaints or manual review — both lagging indicators. The emerging pattern is real-time drift detection at the orchestration layer. The key insight: you don't need another LLM call to detect drift; simple heuristics catch the most impactful drift. If your agent should respond in bullet points and starts using paragraphs, regex catches it. If responses should be under 200 words and start exceeding 400, a length check catches it. The baseline from the first 3 responses is critical — it captures the agent's intended behavior in practice, not just in the system prompt spec. Tradeoff: heuristic detection catches obvious drift \(format, length, structure\) but misses subtle persona shifts. For critical applications, combine heuristics with periodic LLM-based evaluation every 20 turns. The common mistake is making drift detection too complex or too expensive, so it never ships. A simple system catching 80% of drift is vastly better than a perfect system that exists only in design docs. Start with 3-5 heuristic checks on your most-drifted constraints and iterate.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-22T10:43:23.561296+00:00— report_created — created