What Is Context Engineering? Why More Context Makes AI Worse

Here's a finding that should bother you more than it probably does.

Researchers at Chroma gave 18 leading models a task with no reasoning in it at all: copy this list of words back to me exactly. A shell script does this perfectly at any length. Every single model degraded as the list got longer, and past roughly 500 to 750 words they started under-generating and inventing words that were never there (Chroma Research, July 2025).

No ambiguity. No judgment required. Just copying. And they all got worse.

That result kills the assumption most teams are quietly building on — that a million-token context window means you can put a million tokens in it. What you get instead is a window that technically accepts the input and a model whose reliability fell apart somewhere well before the limit.

Context engineering is the discipline that grew out of that gap. And its core move is the opposite of what people expect.

Key Takeaways

• All 18 models tested by Chroma degraded as input length grew, including on a trivial copy task.

• Accuracy drops 30%+ when the needed information sits in the middle rather than the ends (Liu et al., TACL).

• The guiding principle is subtraction: the smallest set of high-signal tokens (Anthropic, 2025).

• Contexts fail in four distinct ways — poisoning, distraction, confusion and clash — and each has a different fix.

What Is Context Engineering?

Context engineering is the practice of deciding what goes into a model's context window, and what stays out, so the model behaves reliably.

Prompt engineering was about wording — how do I phrase this request? Context engineering is about the whole payload: system instructions, tool definitions, retrieved documents, conversation history, prior tool outputs, notes from earlier steps. In an agent, that payload is assembled fresh on every turn, and most of it isn't written by you.

Anthropic's engineering team frames the discipline around one question: "What configuration of context is most likely to generate our model's desired behavior?" Their stated guiding principle is worth memorising, because it inverts the intuition — find "the smallest set of high-signal tokens that maximize the likelihood of your desired outcome."

Smallest. Not most complete.

The mental shift is treating context as a scarce resource with a spending limit, rather than a container to fill. Every token you add competes for the model's attention with every token already there. That framing sounds like a metaphor. The research says it's closer to a hardware constraint.

Doesn't a Million-Token Window Fix This?

No, and this is the most misunderstood point in the entire field. The window is a capacity limit, not a performance guarantee.

The Chroma study tested 18 models — across Claude, GPT, Gemini and Qwen families — on needle-in-a-haystack variations, a 113,000-token conversational benchmark, and the repeated-words task. Four findings stand out:

That third row deserves a second look. A randomly shuffled pile of text produced better retrieval than the same content arranged coherently. Nobody predicted that, and it's a clear signal that our intuitions about how models read long inputs are unreliable.

Add the older, peer-reviewed result to this. Liu and colleagues found models attend well to the beginning and end of their context and poorly to the middle — a "lost in the middle" effect producing accuracy drops above 30% purely from where information sits, not whether it's present.

So position matters as much as presence. The same document, same model, same question — different placement, different answer. The authors' conclusion is blunt: models "do not maintain consistent performance across input lengths," even on trivial tasks.

The Four Ways Context Fails

Drew Breunig's taxonomy is the most useful diagnostic tool in this space, because "the agent got confused" isn't actionable and these four are (Breunig, June 2025):

Poisoning is the nastiest in practice because it compounds. Once a fabricated detail is in the history, every subsequent turn treats it as established fact. The model isn't hallucinating repeatedly — it's faithfully reading its own earlier mistake.

Confusion explains a mistake almost everyone makes early: attaching every tool you have "just in case." Each tool definition is context, and its mere presence raises the chance the model reaches for it. A focused loadout beats a comprehensive one.

Our read: these four failures explain why "just give it more information" backfires so consistently. Each addition carries a specific risk — a wrong fact can poison, a similar fact can confuse, a stale fact can clash, and sheer volume distracts. Meanwhile the Chroma data shows a single distractor doing measurable damage. The uncomfortable implication is that the most dangerous thing you can put in a context window is something almost relevant. Obvious junk gets ignored. Near-misses get used.

How Do You Actually Fix It?

Four techniques, three of them from Anthropic's engineering guidance and all four in production use:

1. Compaction. Summarise the conversation history and replace the raw version. Instead of carrying forty turns, carry a distilled account of what was established and what's still open. This directly attacks distraction, and it's what keeps long agent sessions coherent.

2. Structured note-taking. Have the agent write findings to an external file, then read back only what it needs. Memory lives outside the window rather than inside it, so the working context stays small while the knowledge accumulates.

3. Sub-agents with clean windows. Spin up a specialised agent for a focused task and let it burn context freely in isolation. The detail matters here: a sub-agent might consume tens of thousands of tokens exploring, then return only a distilled summary of roughly 1,000 to 2,000 tokens. The parent agent gets the conclusion without inheriting the mess.

4. Retrieval on demand. Fetch information when it's needed rather than pre-loading it. This is where the naive "long context replaces RAG" argument breaks down — if degradation begins well before the window fills, then selective retrieval is a performance decision, not just a cost one.

The unifying principle across all four: information should enter the context window as late as possible and leave as early as possible.

What This Means for How You Build

Six practical rules that follow directly from the research above:

1. Set a working context budget well below the advertised limit, then measure where your own reliability actually falls off. Don't trust the spec sheet as a performance figure.

2. Put critical instructions at the start or end, never buried mid-context. The lost-in-the-middle effect is a free 30% you can avoid losing.

3. Prune tools per task. If the agent doesn't need it for this job, don't declare it.

4. Filter retrieval hard. Five precise chunks beat fifty plausible ones, because near-misses cause confusion.

5. Compact before you hit trouble, not after. Summarise history on a schedule rather than waiting for degradation.

6. Test at realistic lengths. An agent evaluated on 2,000-token cases tells you nothing about its behaviour at 100,000.

That last one is where most teams get surprised in production. The prototype worked. The prototype was also short.

[INTERNAL-LINK: instrumenting AI features before you scale them → practical guide to measuring agent cost and reliability]

Why This Became a Job

Because most agent failures aren't model failures. They're context failures, and they're fixable by whoever understands the difference.

That's why "context engineering" moved from a blog-post phrase to something teams staff for. The work sits between infrastructure and product: deciding what an agent should see, in what order, from which sources, and what to throw away. It's closer to information architecture than to prompting, and it isn't automated by a better model — a smarter model with a poisoned context still fails.

If you want to build the skill, the fastest route is to break something deliberately. Take an agent that works on short inputs, grow the context until it degrades, then find which of the four failure modes you triggered. That exercise teaches more than any framework tutorial, and it's directly demonstrable in an interview.

Frequently Asked Questions

Is context engineering just prompt engineering renamed?

No. Prompt engineering optimises the wording of a request. Context engineering decides the entire payload — instructions, tools, retrieved documents, history, notes — most of which is assembled automatically rather than written by you, and reassembled on every agent turn.

Does long context make RAG obsolete?

The research says no. Chroma found all 18 tested models degrading as input grew, and accuracy drops over 30% occur purely from mid-context placement. Selective retrieval keeps the high-signal material where models attend best, making it a quality decision as well as a cost one.

What is context rot?

The observed degradation of model performance as input length increases. Chroma's report documented it across 18 models on retrieval, long-conversation and pure-replication tasks, finding the decline non-uniform rather than gradual and predictable.

Where should I put the most important instructions?

At the beginning or the end. Liu and colleagues found models attend well to both ends of their context and poorly to the middle, with accuracy penalties above 30% from placement alone. Never bury a critical constraint mid-payload.

The Skill Is Deciding What to Leave Out

For two years the industry treated context length as a scoreboard. Bigger window, better model, problem solved.

The measurements say otherwise. Reliability falls off well before the limit, position changes answers, a single near-miss document does real damage, and a shuffled pile can beat a coherent one. None of that gets fixed by another zero on the spec sheet.

Which makes context engineering a subtraction discipline in an industry built on addition. The engineers who get good at this won't be the ones who can stuff the most into a window. They'll be the ones who can look at a working agent, find the four tokens' worth of near-relevant garbage quietly wrecking it, and take them out.

Sources