Context Engineering: The Discipline That Replaced Prompt Engineering in Production Agents
Two years ago, when a client's LLM application underperformed, the fix was usually a better prompt. We would iterate on instructions, add few-shot examples, tune the tone — and quality would climb. That era is over for the systems we build now. In 2026, the applications that matter are agents: systems that make dozens to hundreds of model calls per task, carry state across those calls, call tools whose outputs come back messy and huge, and drift toward context windows that are simultaneously enormous and disappointingly finite.
For those systems, "prompt engineering" is the wrong unit of work. You cannot prompt-engineer your way out of a context window containing forty tool results, a stale conversation summary, and a system prompt that has accreted fourteen exceptions over six months of patches. The discipline that actually governs quality in production agents is context engineering: the deliberate design of everything that enters the model's context, and everything that gets kept out of it.
Anthropic popularized the term in 2025, framing it as "the art and science of filling the context window with the right information, for the right reasons." In practice, at Haal Lab, it has become the core design activity on every agent engagement — more consequential than model choice, and second only to retrieval architecture in determining whether a system works.
Why prompts stopped being the unit of design
Consider a representative agent we shipped last year: a research assistant over a client's internal document store. One user task fans out into roughly 40-80 model calls. At each call, the model sees:
1. The system prompt (2,000 tokens of role, constraints, tool guidance) 2. Tool schemas for 12 tools (another 3,000-4,000 tokens) 3. Task state and plan (grows over the run) 4. Retrieved chunks and tool results (the bulk — often 20,000-100,000+ tokens by mid-run) 5. Conversation history or its compressed remnant
The single most important fact about this list: items 3-5 are produced at runtime by the system, not authored by a human. No amount of prompt wording fixes a context that contains the wrong documents, a summary that lost the key constraint, or a tool result that buried the one relevant line in 8,000 tokens of JSON. Quality is determined by what your plumbing puts in the window — and in what order, at what length, with what structure.
There is also a hard economics argument. Every token of context is paid for on every call, and agents multiply calls. Carrying 80k tokens of stale history through 60 calls is 4.8M tokens of pure overhead for one task. Context engineering is not just a quality discipline; it is a cost and latency discipline.
The attention budget: the core mental model
The mental model we use on every project: the context window is an attention budget, not a container. Large-context models will technically accept hundreds of thousands of tokens, but effective reasoning degrades as irrelevant content grows — the model must distribute attention across everything present, and distractors measurably hurt task performance. This is well documented in the long-context literature (the "lost in the middle" findings of Liu et al. remain the canonical demonstration), and every practitioner we know has reproduced it operationally: stuffing more context helps until it silently starts hurting.
So we budget the window explicitly, in rough token allocations:
| Context category | Typical share | Managed by |
|---|---|---|
| System prompt and tool schemas | 10-20% | Authored once; pruned aggressively at design time |
| Task state, plan, decisions | 10-15% | Structured and rewritten by the system, not accumulated |
| Retrieved knowledge / tool results | 40-60% | Just-in-time retrieval; summarized or dropped after use |
| Scratch / working space | 10-20% | Explicitly marked as scratch so it can be truncated safely |
The core techniques, and their failure modes
Context engineering is not one technique; it is a small set of interlocking ones. Each solves a problem and introduces a new one. We list them with the honesty of having been bitten by every failure mode below.
1. Just-in-time retrieval
Do not preload "everything relevant" at the start of the run. Retrieve when the agent needs a fact, and retrieve narrowly. Combined with agentic retrieval (query grading and rewriting — see our RAG decision guide), this keeps the knowledge share of the window small and current.
Failure mode: retrieval variability. Two runs fetch different chunks and diverge downstream. Gate with evals; cache canonical retrievals for deterministic subtasks.
2. Tool result curation and compaction
Raw tool output is the worst context pollutant in production. An API can return 15,000 tokens when the agent needed three fields. Every serious agent stack needs a tool-result shaping layer: project to the needed fields, truncate lists with counts, summarize large blobs, and put the important content at the beginning of the result (position matters — models attend less reliably to middles).
Failure mode: a projection drops a field the model actually needed for a later decision, and the agent confidently invents it. Mitigate by keeping projections configurable per task and logging the raw result outside the context for audit.
3. History compaction and structured state
Conversation history should almost never accumulate raw past a few turns. Replace it with structured state: a running task object with verified facts, decisions made with their reasons, open questions, and constraints — updated by the system at each step, rendered fresh into context each call. This is more reliable than asking the model to compress a transcript, because the state has a schema.
Failure mode: the state update itself is a model call that can drop or distort a constraint. Keep critical facts (identifiers, file paths, numeric constraints) verbatim in the schema, never paraphrased.
4. Sub-agent isolation
When a subtask is noisy — browse 30 pages, grep a codebase, read a long thread — spawn a sub-agent with a fresh window containing only the scoped task, and return a distilled result to the parent. This is the single most effective context hygiene technique in complex systems, and it is how modern coding agents keep their main loops coherent.
Failure mode: the distillation boundary. The sub-agent's summary is the parent's entire world; if it omits a relevant negative result ("the config file does not exist"), the parent reasons from incomplete ground truth. Design return schemas to force the negatives out.
5. System prompt hygiene
The system prompt is the one region you fully control, so keep it worthy of the attention it commands. We audit client system prompts that have grown into 5,000-token archives of every incident ever patched. Every instruction is attention tax paid on every call. Prune, merge, and move task-specific guidance into task state where it can be scoped.
Failure mode: instructions added under time pressure conflict silently ("be concise" plus a template demanding exhaustive detail), and the model resolves the conflict unpredictably. Diff prompts in review like code.
A practical context audit method
Here is the audit we run when we inherit a struggling agent. You can do a version of it in a day:
1. Dump real traces. Capture the full context sent to the model at every call for 10-20 real tasks — not synthetic ones. Most teams have never actually looked at this. 2. Categorize every token. Label context content by the budget categories above. The usual finding: retrieved knowledge and tool results occupy 70-90% of the window, most of it no longer needed at the moment of the call. 3. Find the dead weight. For each call, ask: would the answer change if this block were removed? Anything that survives only "just in case" is a candidate for just-in-time retrieval. 4. Find the position problems. Check whether critical instructions or key facts sit deep in the middle of huge blocks. Reorder: instructions first, key evidence near the front, bulk last. 5. Measure the cliff. Run your eval set at different context sizes for the same task (trim aggressively, then re-add). If quality does not drop when you halve the context, you have found free cost savings. If it drops sharply, you have found what actually matters — protect that. 6. Re-budget and enforce. Set per-category caps in code, add compaction triggers at 60-70% window use, and put token counts per call into your observability so context bloat shows up on a dashboard instead of a bill.
The consistent result across the audits we have run: 40-70% of context at any given call is removable without quality loss, and removing it usually improves answer quality while cutting cost by a comparable fraction.
What context engineering does not fix
Honesty section. Context engineering cannot rescue a bad retrieval architecture — if the right information never enters the window at any point, better budgeting just organizes the ignorance. It cannot fix an incoherent tool design (twelve overlapping tools with ambiguous schemas will produce wrong selections no matter how clean the context is). And it adds its own maintenance burden: compaction logic, state schemas, and projection layers are code that can have bugs, and every summarization step is a lossy model call that must itself be evaluated. This is the actual trade: you accept a more complex, stateful system in exchange for quality and cost characteristics that raw prompting cannot reach. In our experience the trade is worth it the moment an agent exceeds roughly ten model calls per task — which is to say, for essentially every agent in production.
References and further reading
1. Anthropic (2025). "Effective context engineering for AI agents." Anthropic Engineering Blog. 2. Liu, N. et al. (2023). "Lost in the Middle: How Language Models Use Long Contexts." arXiv 2307.03172. 3. Zhang, T. et al. (2024). "In-Context Retrieval Augmented Language Models." arXiv 2402.00037. (TACO-RL work on context relevance, qH and qV metrics.) 4. Asai, A. et al. (2023). "Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection." arXiv 2310.11511. 5. Lewis, P. et al. (2020). "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." arXiv 2005.11401. 6. OWASP Foundation (2025). "OWASP Top 10 for LLM Applications." (LLM04: unauthorized disclosure of sensitive data via context.) 7. Kim, H. J. et al. (2024). "Memory in LLM Agents: A Survey." arXiv 2404.13501. 8. Anthropic (2024). "Building effective agents." Anthropic Engineering Blog.
The shift from prompt engineering to context engineering is really the shift from designing a call to designing a system. Want a context audit on your agent, or help designing one that stays coherent across long tasks? Contact us — this is one of our core studio engagements. More of our engineering notes live on the blog.