Working with AI agents comes with the inevitable challenges of a context system setup. When agents need to understand a task thoroughly, the first intuition is usually to provide as much context as possible for them to consider.
We have come a long way from a context window of ~4k tokens to 1M+. GPT-3 started with 2k — about 1,500 words. Vibe coding would devour that in one prompt completion.
And every year since, we saw the context window increase. The assumption: more context equals better performance. A larger context window means more context passed to an agent, and therefore a better understanding. Or so we thought.
Hitting the big numbers for context windows did serve as an important milestone. It enabled:
- Full codebase ingestion for deeper, more complete understanding
- Large document reviews without breaking files into chunks
- Better large-scale research without fragmenting a study
But as we look closer in the agentic coding space, it becomes apparent where the 1M+ token context window doesn't serve us in the way we thought.
The gaps became clearer:
- Memory across sessions
- Collaboration across agents
- Relevance selection of context
- Soaring token costs
Looking back at the focal narratives across the years:
- 2023 — prompt engineering. Knowing how to talk to AI and designing prompts to get desired outputs was the center of AI implementations.
- 2024 — RAG architecture, solving for scale when retrieving knowledge from large amounts of external data.
- 2025 — the year of agents. Architecture built over the years became sufficient for agents capable of taking more sophisticated actions.
With all the scale and accessibility that the preceding developments gave rise to, the next set of challenges were intuitive. Given all the code knowledge needed with the scale of code generation, the challenge was no longer whether there is enough space to fit all the context in — but rather, what knowledge needs to be fit in.
The difficulty with coding agents and why they need more than just 'passing all the context'
A significant amount of technical reasons already made the need for a better context system inevitable. For coding agents, we explored the reasons contributing towards why the context window didn't solve for the current needs.
When we looked at coding tasks more specifically, the structure of the code directories plays a crucial role in the way we think of passing context to an agent. Repositories being the logical separation for code made context retrieval also siloed. An agent indexing is primarily on a repo level. While referencing has become possible across repos, the primary search for context still remains an individual repository.
As the scale of code generation grows, the amount of information that is not relevant also grows — test files, deprecated APIs, old implementations, and more. These don't just fail to help the agent; they make it more confused and produce worse output. Depending on the model, studies have observed a noticeable degradation of output accuracy as more context was fed to the model.
In addition, it doesn't help that a fraction of the context fed to the model is truly used or needed for the task. A research paper, ContextBench: A Benchmark for Context Retrieval in Coding Agents, cites how coding agents highly favor recall over precision — indicating a wide gap between the information explored and the information utilized. Agents favor fetching more data in an attempt to not miss anything, rather than attempting precise retrieval.
Choices made in the code about architecture mention the WHAT but not the WHY. Agents are only aware of what the code includes, not the reason for it — which is why agents struggle to understand the context of a service, framework, or technology in your code.
This became more than a needle-in-a-haystack problem. We aren't just looking for the lines of code relevant to a request. We are looking for the logic, reasoning, and understanding behind it.
Think of a system that understands your architecture decisions, preferences, and way of working — and maintains this understanding when you take actions relating to it. A form of derived intelligence context beyond the lines of code itself.
And without a doubt, the crippling costs that token usage comes with. Surely needs no explanation.
Why just passing all the knowledge didn't work and why a context management system is rising in prominence
While the reasons mentioned above are true, those are symptoms of the underlying limitation. They don't truly answer why a large enough context window isn't solving the context problem.
The real reason sits behind how transformer models really operate.
When we talk about coding agents and the nature of codebases, as code scales, relationships scale exponentially. The model is now asking which tokens matter, how they relate, which should influence the answer, and which should be ignored — all over 10,000 files instead of 100.
Software is highly interconnected. Files are connected in different ways for different uses.
Establishing these relationships becomes increasingly harder for a model when the number of input tokens is large.
Each token from the input is compared against every other token for a relationship. The result is a softmax applied over all tokens, for which the weights must sum to 1.0 across the entire context. The model has to discriminate among a substantially larger set of candidate tokens — assigning high attention scores to the small subset that matters compared to all the others that don't.
Research from Adobe's NoLiMa: Long-Context Evaluation Beyond Literal Matching emphasizes the increasing challenge of retrieval and prioritization as context increases, especially when relevance depends on semantic relationships rather than exact matches.
The advertised context window vs practical wisdom
The 1M context window implies the model is capable of ingesting 1M tokens. Ingesting — often confused with reasoning.
We don't need the model to simply read through 1M tokens. We need it to establish knowledge we can derive from them: imports, dependencies, calls, architectural relationships, decisions made in the past, and interactions across services.
The real problem: context length increases the number of possible relationships the model could consider far faster than it increases the model's reasoning capacity.
The NoLiMa: Long-Context Evaluation Beyond Literal Matching study shows the scores across models with the increase in input length.
We see similar results in the study done with increasing input context for multiple models — all show the same trend of scores reducing with increasing context length.
What becomes clear is that the real limitation is no longer about how much context a model can consume, but whether it can effectively derive meaning from it.
While the context window may possibly continue to grow, the real development is in the context engineering space. The solution is going to look less like large, ever-growing prompts and more like multi-layered architecture that includes the nuances needed to make the right decisions to write code.
Understanding of code is beyond just what the code is or where it resides — but relationships across repositories, why that code exists, beyond just the current state, and how that state evolved through prior decisions and interactions.
The real win is going to be driven by the relevance of the context passed to the models. The shift of focus from model capability to context design — being able to precisely input what the model should know at the exact moment it needs to know it.