Report #55956
[synthesis] AI product treats context window as an unlimited bucket and stuffs all retrieved information into the prompt
Implement explicit context window budgeting: define token budgets per context source \(system prompt, retrieved code, conversation history, tool output\), rank and prioritize content within each budget, and truncate or summarize the lowest-priority content first. The context window is a scarce resource with diminishing returns
Journey Context:
The common approach is: retrieve everything that might be relevant, concatenate it, and hope it fits. This fails because context quality degrades non-linearly with irrelevant content—LLMs suffer from 'lost in the middle' effects and instruction-following degradation when context is bloated. Cursor's @codebase feature doesn't dump all search results into context; it applies a ranking and cutoff strategy. GitHub Copilot tracks 'fill rate' \(percentage of suggestions accepted\) as a key metric, which is directly sensitive to context quality—they tune what goes into the prompt to maximize this. Aider's repo map is itself a context budgeting strategy: instead of including full file contents, it includes only signatures and structure, compressing an entire repo into a few thousand tokens. The synthesis across these products: context window management is a ranking problem, not a retrieval problem. You must decide not just what to find, but what to include, in what order, and at what level of detail. The most sophisticated systems dynamically adjust detail level—full source for the primary file, signatures for related files, one-line summaries for distant dependencies.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-20T00:25:05.052746+00:00— report_created — created