A prompt isn’t a magic spell. It’s the runtime environment you hand the model — everything it knows about the task, the user, the rules and the world, assembled fresh for every request. Once you see it that way, most “prompt engineering” problems turn into ordinary architecture problems: what goes in, in what order, how big each part is allowed to get, and how you know a change made things better.

This post is how I design that environment when building on Claude. None of it is about clever wording. It’s about layout, budgets, interfaces and feedback loops.

The context window is a system with a layout

Every request I send is assembled from the same parts, in the same order:

  1. Tools — the actions the model can take, with their schemas.
  2. Instructions — role, rules, output contract. The part that changes when you ship a new version, not per request.
  3. Reference material — long, stable documents: policies, product docs, a codebase map.
  4. Retrieved context — the facts fetched for this request.
  5. State and the current task — the conversation so far, and what’s being asked right now.

The order isn’t aesthetic. It follows how volatile each layer is, from content that changes once a release to content that changes every turn — and that ordering pays off twice.

STABLEVOLATILEtools — small, typed, in a fixed orderinstructions — role, rules, output contractreference docs — long and stablecache breakpointretrieved context — per requeststate + the current taskeverything above the breakpoint is byte-identical between requests
Order the context from most stable to most volatile — it helps the model, and it makes the prefix cacheable.

It makes the prefix cacheable. Prompt caching works on an exact prefix match: tools, then system, then messages. If the first 30,000 tokens of every request are byte-identical, you can mark a cache breakpoint after them and pay a fraction of the input price to re-read them. A single volatile value near the top — a timestamp in the system prompt, a tool list whose order depends on a hash map — silently breaks the match for everything after it.

It helps the model. Long reference material works best near the top, with the specific question at the end, clearly separated. I wrap each layer in its own XML tags — <policy>, <retrieved_documents>, <task> — so instructions can refer to them by name (“answer only from <retrieved_documents>”) and the model never confuses a quoted document with an instruction.

Here’s what that looks like as an actual request:

import Anthropic from '@anthropic-ai/sdk';

const client = new Anthropic();

const response = await client.messages.create({
  model: 'claude-opus-5',
  max_tokens: 16000,
  tools: TOOLS, // same tools, same order, every request
  system: [
    { type: 'text', text: INSTRUCTIONS },
    // Stable reference docs end the cached prefix.
    { type: 'text', text: `<policy>${POLICY_DOCS}</policy>`, cache_control: { type: 'ephemeral' } },
  ],
  messages: [
    ...history,
    {
      role: 'user',
      content: `<retrieved_documents>${retrieved}</retrieved_documents>\n\n<task>${question}</task>`,
    },
  ],
});

// Check the cache is actually being hit.
console.log(response.usage.cache_read_input_tokens);

That last line is the one people skip. If cache_read_input_tokens stays at zero across repeated requests, something in your “stable” prefix isn’t.

Give every layer a budget

A large context window is a capacity, not a target. Filling it has costs: every token is paid for on every request, latency grows with input, and the more irrelevant material the window holds, the easier it is for the model to latch onto the wrong part of it.

So each layer gets a budget, and each budget has an owner:

  • Instructions: as short as they can be while still being complete. When a new rule gets added after an incident, check whether an older rule already covers it. Instructions accrete; prune them every release.
  • Reference material: include what the task actually uses. If only three sections of a 200-page handbook matter for this feature, send those three.
  • Retrieved context: a fixed number of chunks, ranked, with the lowest-scoring dropped. Retrieving ten mediocre passages is worse than three good ones — the model has to read the noise too.
  • Conversation state: bounded. Old turns get summarised or dropped, not carried forever.

The test for whether something belongs in the window is simple: can you say what the model would get wrong without it? If not, it’s costing you tokens and attention for nothing.

Context is the API surface between your system and the model. Design it like one.

Keep state outside the window

Long-running conversations and agents tend to treat the context window as their database. It’s a poor one: it’s expensive to read, it can’t be queried, and it fills up.

I keep durable state in the application — a structured object of what’s been decided, what’s pending, what the user has told us — and render a compact version of it into each request. The transcript becomes a log, not the source of truth. When a conversation gets long, earlier turns can be summarised or trimmed without losing anything important, because the important things were never only in the transcript.

For agents, the same idea applies to tool results. A tool that returns a 40,000-token API response has handed the model a problem; one that returns the five fields that matter, with an ID for fetching more, has handed it an answer.

Tools are the other half of the interface

If context is what the model knows, tools are what it can do. The design principles are the same ones you’d apply to any API meant for a consumer you don’t fully control.

  • Few and sharp. A small set of tools with clearly different purposes beats a large set with overlapping ones. If you struggle to explain when to use tool A instead of tool B, so will the model.
  • Descriptions are documentation. The description is the only manual the model gets. Say what the tool does, when to use it, when not to, and what it returns.
  • Typed, strict inputs. Enums instead of free strings, required fields actually required, no unexpected properties. Strict schema validation turns a class of malformed calls into impossibilities.
  • Results sized for reading. Return what the model needs to take its next step, in a predictable shape. Paginate or summarise large results.
  • Errors that teach. When a call fails, return an error result that says what went wrong and how to fix it — “order_id must look like ORD-12345; got 12345” — rather than a stack trace or an empty string. The model can recover from a good error message.
const lookupOrder: Anthropic.Tool = {
  name: 'lookup_order',
  description:
    'Fetch one order by its ID. Returns status, line items and refund eligibility. ' +
    'Use when the user refers to a specific order. Do not use to search by customer — use find_orders.',
  input_schema: {
    type: 'object',
    properties: {
      order_id: { type: 'string', description: 'Order ID, e.g. ORD-10293' },
    },
    required: ['order_id'],
    additionalProperties: false,
  },
  strict: true,
};

And validate every call on your side anyway. The model choosing a tool isn’t authorisation to run it: permissions, rate limits and confirmation for irreversible actions belong in your code, not in the prompt.

Close the loop with evaluation

A context layout is a design decision, and design decisions need feedback. Every change — a reworded instruction, a new retrieval strategy, a trimmed reference doc — should run against a fixed set of real examples before it ships, with results broken down by the kinds of failure you care about. I’ve written about how I build that loop in Evaluating LLM features without fooling yourself; the short version is that without it, you’re choosing between prompts by vibes.

It also changes how you treat prompts in the codebase. Mine live in version control next to the code that assembles them, are reviewed like code, and carry a version that’s logged with every request. When behaviour changes in production, the first question — “what changed in the context?” — has an answer.

The checklist I use when reviewing a prompt-based feature:

  • Context is assembled in a fixed order, most stable to most volatile, with layers clearly delimited.
  • The stable prefix really is stable, and cache hits are verified in the usage numbers.
  • Each layer has a token budget, and everything in the window earns its place.
  • Durable state lives in the application; the transcript is a log.
  • Tools are few, sharply described, strictly typed, and return compact results and useful errors.
  • Every change to the context runs against an eval set before it ships.
  • Prompts are versioned, reviewed, and logged with each request.

Treat the context window like any other system you own — designed, budgeted, versioned and measured — and prompting stops being guesswork.