Paul SerbanSoftware Engineer

Prompt Engineering, Context Engineering, Loop Engineering: The Three Layers of Reliable AI AgentsWhy writing a better prompt stopped being the hard part, and what to engineer instead

Introduction

Two years ago, "prompt engineering" was treated almost as a job title. Today, most engineers who spent that time wrestling with wording, delimiters, and few-shot examples will tell you that phrasing was rarely the reason their system failed in production. It failed because the model was missing a document it needed, because a tool result got truncated, because a conversation history grew until the instructions at the top of the context were effectively ignored, or because an agent kept retrying the same broken action in an infinite loop until someone's API bill made the problem impossible to ignore. None of those are prompt problems. They are problems with what the model sees and how the system around the model behaves over time.

This article is about the resulting shift in vocabulary and practice: from prompt engineering (shaping a single instruction), to context engineering (curating everything the model sees at inference time), to loop engineering (designing the control flow that decides what happens across many inference calls). These are not competing terms for the same skill. They are three distinct engineering disciplines that sit on top of each other, each addressing a different class of failure, and each becoming relevant only once the layer below it is reasonably well handled. Understanding where one ends and the next begins is, in practice, the difference between a demo that works once and a system that works reliably in front of paying customers.

The Problem: Why Prompting Alone Stopped Being Enough

In the single-turn era of LLM applications - a chatbot answering one question, a summarizer condensing one document, a classifier labeling one input - prompt wording genuinely was the main lever available to a developer. Small changes in instruction phrasing, the order of examples, or the presence of a role description could measurably change output quality. This is why early guides from OpenAI and Anthropic focused heavily on technique: clear and direct instructions, few-shot examples, chain-of-thought elicitation, and structured output formats. Chain-of-thought prompting itself was formalized in a widely cited 2022 paper by Wei et al., which showed that asking a model to reason step by step before answering measurably improved performance on multi-step problems. That paper, and the broader prompt engineering literature that followed it, is not wrong - it is simply describing one layer of a larger system.

The trouble starts when an application needs more than one inference call to do its job: a customer support agent that has to look up an order, check a policy document, and decide whether to escalate; a coding assistant that has to read a repository, run tests, and revise its own output. Once retrieval, tool results, prior turns, and system instructions are all competing for space in a single context window, the dominant failure mode changes. In mid-2025, Shopify CEO Tobi Lütke argued publicly that "prompt engineering" was the wrong frame for this problem, proposing "context engineering" instead as a better description of the underlying skill: assembling everything a model needs to plausibly solve a task. Andrej Karpathy endorsed the reframing shortly after, describing the work as filling the context window with precisely the right information for each step of a task, not simply writing a short instruction. The term spread quickly through engineering blogs at LangChain, Anthropic, and elsewhere because it named something practitioners were already doing without a shared vocabulary for it.

A third failure mode appears once a system is allowed to act over multiple steps without a human approving each one - the defining feature of an agent rather than a single workflow. An agent, at its simplest, is often described as a model that runs tools in a loop until a task is complete or a limit is reached, a framing popularized in agent-building circles by Simon Willison and echoed across recent write-ups from LangChain and independent practitioners under the heading of "loop engineering." Dex Horthy's widely circulated 12-Factor Agents project, built after interviewing engineering teams building production agents, makes a related point directly: many systems marketed as autonomous agents work well not because the loop is magical, but because the surrounding control flow is disciplined, ordinary software. Once a system can call itself again, the question stops being "what do I say to the model" and becomes "when does this process stop, and what happens when a step fails."

The Three Layers: A Deep Technical Explanation

It helps to define each layer precisely, because the boundaries between them are exactly where most production bugs live. Each layer operates at a different scope: a single call, a single context window, and a sequence of calls over time, respectively. Treating them as one undifferentiated skill - "prompting" - is what leads teams to try to fix a context problem by rewording an instruction, or to fix a loop problem by adding yet another paragraph to an already-overloaded system prompt. The three definitions below are deliberately narrow, because narrowness is what makes each one testable in isolation: you can evaluate a prompt against a fixed context, evaluate a context-assembly function against a fixed set of retrieved documents, and evaluate a loop's stopping behavior against a fixed sequence of simulated tool results, without the other two layers ever changing underneath you.

Prompt Engineering: Shaping a Single Inference Call

Prompt engineering is the discipline of designing the text (and increasingly the structure) of a single request to a model: the system instructions, the task description, output format constraints, and any few-shot examples included inline. Its techniques are well documented - role framing, explicit output schemas, chain-of-thought elicitation, and self-consistency checks - and its scope is deliberately narrow. A well-engineered prompt answers the question "given everything this model can see right now, how do I phrase the request so the model produces the response I need." It does not answer the question of what that "everything" should contain in the first place, which is where the next layer begins.

Context Engineering: Managing the Working Memory of the System

Context engineering operates one level up: it decides what goes into the context window at all, in what form, and in what order, before any prompt wording is applied on top of it. Anthropic's engineering team has described this as the practice of curating and maintaining the optimal set of tokens during inference - system instructions, retrieved documents, tool definitions, conversation history, and intermediate results - treating the context window as a scarce, finite resource rather than an inbox to be filled. This involves concrete sub-practices: selection (deciding which documents or tools are relevant enough to include), compression (summarizing or truncating history so it fits), and ordering (placing the most important instructions where model attention is strongest, since research on long-context behavior, notably Liu et al.'s "Lost in the Middle" study, has shown that models retrieve information less reliably from the middle of a long context than from its beginning or end).

Loop Engineering: Designing the Control Flow Around the Model

Loop engineering sits above both. It governs what happens across repeated model calls: how a tool call is parsed and executed, how the resulting observation is fed back in, how many iterations are permitted, what counts as "done," and what happens when a step produces an error or an ambiguous result. This is classic software engineering - state machines, retries, timeouts, idempotency - applied to a component whose output is probabilistic rather than deterministic. The ReAct pattern described by Yao et al. in 2022, which interleaves reasoning traces with actions and observations, is one of the earliest formal descriptions of this loop, and it remains the conceptual basis for most agent frameworks in use today, including LangGraph and the agent loops built into tools like Claude Code.

Implementation: Building the Three Layers in Code

The cleanest way to see the boundary between these layers is to build a small agent and keep each concern in its own function. Below is a simplified but realistic pattern for a research-and-answer agent: a prompt builder that only concerns itself with instruction wording and output format, a context assembler that decides what evidence and history to include, and a loop controller that owns iteration, stopping, and error handling. None of these three pieces knows how to do the others' job, which is precisely the point - it is what makes each one independently testable. The prompt layer shown first is intentionally the simplest of the three: it takes already-assembled context as a plain string input and is responsible only for instruction clarity, role framing, and enforcing a structured response format, so that the loop controller further down can parse the model's output deterministically instead of guessing at free-text intent.

// prompt.ts - prompt engineering layer: wording and output structure only
interface AgentTurnInput {
  task: string;
  contextBlock: string; // produced by the context engineering layer
  availableTools: { name: string; description: string }[];
}

export function buildSystemPrompt(input: AgentTurnInput): string {
  const toolList = input.availableTools
    .map((t) => `- ${t.name}: ${t.description}`)
    .join("\n");

  return [
    "You are a research assistant that answers questions using the",
    "provided context and, when necessary, the tools listed below.",
    "",
    "Rules:",
    "1. Use only facts present in the context or returned by a tool call.",
    "2. If the context is insufficient, call a tool rather than guessing.",
    "3. Respond with a single JSON object matching this schema:",
    '   { "action": "final_answer" | "tool_call"',,
    '     "tool_name"?: string, "tool_input"?: object',,
    '     "answer"?: string, "confidence": "low" | "medium" | "high" }',
    "",
    `Available tools:\n${toolList}`,
    "",
    `Context:\n${input.contextBlock}`,
  ].join("\n");
}

The context layer is where most of the real engineering effort belongs, and it is deliberately the largest piece of code in this example. Its job is to decide, on every turn, which pieces of retrieved evidence, tool output, and prior conversation are worth the tokens they cost, and to compress or drop the rest before the prompt layer ever sees them. The two functions below split that job in half: one ranks and prunes candidate context items against a fixed token budget, and the other orders whatever survives so the highest-value material sits at the edges of the block rather than buried in its middle. Once this block is assembled, it is handed to the prompt layer unchanged - the prompt builder never re-evaluates relevance, it only wraps whatever context it is given in clear instructions.

# context_assembler.py - context engineering layer: selection, compression, ordering
from dataclasses import dataclass
from typing import List

TOKEN_BUDGET = 6000  # reserve room for system prompt, output, and margin

@dataclass
class ContextItem:
    source: str
    text: str
    relevance_score: float
    token_estimate: int

def select_and_compress(items: List[ContextItem], summarizer) -> List[ContextItem]:
    """Rank by relevance, drop low-value items first, and summarize
    anything that survives selection but is still too long."""
    ranked = sorted(items, key=lambda i: i.relevance_score, reverse=True)
    kept: List[ContextItem] = []
    running_total = 0

    for item in ranked:
        if running_total + item.token_estimate <= TOKEN_BUDGET:
            kept.append(item)
            running_total += item.token_estimate
            continue
        # Item didn't fit as-is; try compressing it before discarding it.
        compressed_text = summarizer(item.text, max_tokens=300)
        compressed_estimate = len(compressed_text) // 4
        if running_total + compressed_estimate <= TOKEN_BUDGET:
            kept.append(ContextItem(
                item.source, compressed_text, item.relevance_score, compressed_estimate
            ))
            running_total += compressed_estimate

    return kept

def assemble_context_block(items: List[ContextItem]) -> str:
    """Order matters: put the most load-bearing instructions and the most
    recent, most relevant evidence at the start and end of the block,
    since mid-context information is retrieved less reliably."""
    ordered = sorted(items, key=lambda i: i.relevance_score, reverse=True)
    lines = [f"[{i.source}]\n{i.text}" for i in ordered]
    return "\n\n".join(lines)

// agent_loop.ts - loop engineering layer: control flow, stopping, error handling
// This owns everything the prompt and context layers do not: how many turns
// are allowed, how tool failures are handled, and what condition ends the
// loop. This is the part most naive agent implementations get wrong, because
// it is tempting to let the model itself decide when it is "done."
import { callModel, executeTool } from "./runtime";
import { buildSystemPrompt } from "./prompt";

const MAX_ITERATIONS = 6;
const MAX_CONSECUTIVE_ERRORS = 2;

export async function runAgentLoop(task: string, contextBlock: string, tools: any[]) {
  let consecutiveErrors = 0;

  for (let step = 0; step < MAX_ITERATIONS; step++) {
    const systemPrompt = buildSystemPrompt({ task, contextBlock, availableTools: tools });
    const response = await callModel(systemPrompt, task);

    let parsed;
    try {
      parsed = JSON.parse(response);
    } catch {
      consecutiveErrors++;
      if (consecutiveErrors >= MAX_CONSECUTIVE_ERRORS) {
        return { status: "failed", reason: "unparseable_output", step };
      }
      continue; // ask again rather than crash the whole run
    }

    if (parsed.action === "final_answer") {
      return { status: "done", answer: parsed.answer, confidence: parsed.confidence, step };
    }

    if (parsed.action === "tool_call") {
      const result = await executeTool(parsed.tool_name, parsed.tool_input);
      contextBlock += `\n\n[tool_result:${parsed.tool_name}]\n${result}`;
      consecutiveErrors = 0; // reset on a successful, well-formed step
      continue;
    }

    return { status: "failed", reason: "unknown_action", step };
  }

  return { status: "failed", reason: "max_iterations_reached" };
}

Notice what this loop deliberately does not delegate to the model: it does not ask the model whether it should keep going, and it does not let a single malformed response crash the process or spin forever. A parsing failure gets a bounded number of retries rather than an immediate crash, a successful tool call resets the error counter so a single transient failure doesn't end the run early, and the iteration cap guarantees the process terminates even if the model never produces a valid final_answer. Those decisions all live in ordinary, deterministic code, which is the central argument behind Dex Horthy's 12-Factor Agents principles and Anthropic's own guidance on agentic systems - that reliable agents are mostly conventional software with LLM calls inserted at well-chosen points, not one large, unstructured loop that hopes the model self-regulates.

Trade-offs and Common Pitfalls

Context engineering has an obvious failure mode in each direction. Too little context and the model lacks the facts it needs, producing confident-sounding but wrong answers. Too much context, and two separate costs compound: a direct cost in latency and token spend, and a subtler accuracy cost from the "lost in the middle" effect, where relevant information buried in the center of a long context window is retrieved less reliably than information near the edges. Teams that respond to a quality problem by simply appending more instructions or more retrieved documents to the prompt often make the underlying retrieval problem worse rather than better, because the model's effective attention budget does not grow with the size of the context window; it just gets diluted further.

Loop engineering carries a different and, in production, often more expensive set of risks. A loop without an explicit, code-level stop condition can retry the same failing action indefinitely, multiply API costs by the number of iterations it runs, or take irreversible actions - sending an email, executing a database write, charging a card - before a human has any visibility into what happened. The 12-Factor Agents project frames this as a control-flow ownership problem: the fix is not a cleverer prompt asking the model to "be careful," but explicit code that caps iteration counts, distinguishes recoverable errors from hard failures, and routes side-effecting actions through human approval or a sandboxed dry run before they touch production systems. Because these two layers interact - a badly compressed context can cause a loop to misjudge whether a task is finished - debugging an agent failure usually means checking both independently rather than assuming the fault sits wherever the most recent prompt change was made.

Best Practices

At the prompt layer, the highest-leverage habits are the least glamorous ones: state the task and constraints explicitly rather than implying them, specify the exact output format the downstream code expects (ideally a schema, not free text), and keep the system prompt stable across turns so that prompt caching - supported by both the Anthropic and OpenAI APIs - can reduce latency and cost on repeated calls. Iterating on prompt wording still matters, but it should happen after the context and loop layers are stable, using a small held-out set of representative inputs rather than a handful of manual spot checks, so that a wording change which helps one example does not silently regress another.

At the context layer, the discipline that pays off fastest is treating the context window as a budget to be allocated, not a container to be filled. That means retrieving narrowly rather than broadly (passing a model five relevant tools instead of fifty available ones tends to improve tool-selection accuracy), summarizing conversation history instead of carrying full transcripts forward indefinitely, and deliberately placing the instructions the system depends on most at the start or end of the context rather than burying them in the middle. Layered memory - a short-term window for the current task and a separate, retrievable long-term store such as a vector database for facts that need to persist across sessions - keeps this budget from growing unbounded as an application matures.

At the loop layer, the practices that separate demos from production systems are almost entirely about explicit control: define a maximum number of iterations and a clear, code-checked definition of "done" rather than trusting the model to self-report completion; distinguish transient errors worth retrying from structural ones that should halt the run and escalate to a human; and log every step - prompt, context snapshot, tool call, and result - so that a failure can be replayed and diagnosed rather than reproduced by guesswork. Human-in-the-loop checkpoints before irreversible actions are not a concession to caution; they are a standard control-flow pattern for any system, human or automated, that can take real-world side effects.

Mental Models for Thinking in Layers

A useful analogy, one Karpathy has used in discussing this shift, is to think of the model as a CPU and the context window as its RAM: a CPU's raw compute (the model's weights and training) matters less, in a given task, than what has been loaded into the limited memory it can actually operate on in that moment. Prompt engineering is choosing the exact instruction you hand the CPU for this cycle. Context engineering is deciding what data gets loaded into RAM before that instruction runs, and how it is laid out so the most important bytes are cheap to access. Loop engineering is the operating system around both - the scheduler deciding how many cycles a process gets, what happens when it hangs, and when to kill it and report an error rather than let it run forever.

A second, more physical analogy is a job interview conducted over several rounds rather than one conversation. The prompt is the specific question asked in a given round. The context is the candidate's resume, the interviewer's notes from earlier rounds, and any reference material on the desk - everything available to answer well, curated so the interviewer isn't drowning in irrelevant paperwork. The loop is the interview process itself: how many rounds there are, what triggers moving to the next round versus rejecting the candidate, and what happens if a round produces an inconclusive result. No single well-phrased question fixes a process where the interviewer has the wrong file on the desk, and no amount of file organization fixes a process with no defined endpoint.

The 80/20 Insight

If a team can only invest engineering time in one of these three layers, the evidence from practitioner reports and Anthropic's own agent-building guidance points consistently toward context engineering as the highest-leverage layer once an application has moved past a single-turn demo. Most failures attributed informally to "the model isn't smart enough" turn out, on inspection, to be cases where the model was never shown the fact it needed, or was shown it in a form buried under irrelevant text. Auditing what actually enters the context window on a failing turn - not rewriting the prompt around it - resolves a disproportionate share of quality issues, which is the practical reason the term displaced "prompt engineering" in production-focused conversations in 2025.

The second highest-leverage investment, for any system that runs more than one model call per task, is owning the control flow explicitly rather than letting an unstructured loop run until it happens to stop. A hard iteration cap, a code-level definition of task completion, and a clear separation between recoverable and unrecoverable errors will prevent the large majority of runaway-cost and irreversible-action incidents that make agentic systems risky to deploy. Neither of these two practices requires a new framework or a research breakthrough; both are achievable with the kind of state-machine and budget-management thinking that has existed in distributed systems engineering for decades, simply pointed at a new, probabilistic component.

Prompt wording, by contrast, has diminishing returns once the model is a modern frontier model and the output format is enforced structurally rather than requested politely. This does not make prompt engineering worthless - a poorly framed instruction can still sink an otherwise well-built system, and getting the schema and role framing right is a real prerequisite - but it does mean the marginal hour is rarely well spent there once that baseline is in place. In practice, teams that keep a running log of production failures tagged by layer tend to find the same pattern: the context and loop layers account for the large majority of incidents worth engineering time, while prompt wording accounts for a small, mostly front-loaded share of the total effort.

Key Takeaways

The three layers are cumulative, not competitive: a good loop cannot fix bad context, and a good prompt cannot fix a starved context window or a runaway loop. Treat them as separate concerns with separate code, separate tests, and separate failure signatures, and resist the instinct to reach for prompt wording as the universal fix whenever an output looks wrong - it is frequently the layer least responsible for the failure. The five practices below are not exhaustive, but they cover the changes that most reliably move a system from "works in the demo" to "survives contact with real users and real edge cases."

  • Diagnose failures by layer before touching wording: ask whether the model had the right information available (context), whether the process ever should have stopped (loop), or whether the instruction itself was ambiguous (prompt) - in that order.
  • Budget the context window explicitly. Rank and compress what goes in; do not let retrieval, tool output, and history accumulate unchecked.
  • Put load-bearing instructions and the most relevant evidence at the start or end of the context, not the middle, given known long-context attention degradation.
  • Give every agent loop a hard iteration cap and a code-checked definition of "done" - never rely on the model to self-report completion.
  • Route irreversible or side-effecting actions through an explicit approval or dry-run step rather than trusting a single model call to gate them.

None of these five practices requires a new framework, a bigger model, or a research breakthrough to implement. They are ordinary engineering discipline - budgets, schemas, caps, and approval gates - applied to a new kind of component, and a team that adopts even the first two will typically resolve more production incidents than a team that spends the same time rewriting system prompts. Treat this list as a starting checklist for a post-incident review: when an agent misbehaves, walk it top to bottom before assuming the fix has to involve a rewritten instruction.

Conclusion

The move from prompt engineering to context engineering to loop engineering is best understood as a maturing of what "building with LLMs" actually means, not a rebranding of the same skill. Each term names a real layer of engineering effort with its own failure modes, its own best practices, and - as the code above shows - its own testable, independent piece of a system. A team that only ever tunes prompt wording will hit a ceiling well before a team that also disciplines what enters the context window and owns the control flow around repeated model calls.

As frontier models continue to improve at following instructions reliably, the prompt layer will keep shrinking in relative importance, exactly as several practitioners already argue is happening: phrasing variance is increasingly something models handle gracefully on their own. What does not shrink is the engineering work of deciding what a model should see and how a multi-step process should behave when things go wrong - because both of those remain problems of system design, not problems of language. That is also why the skills involved transfer cleanly from traditional software engineering: budget management, state machines, retries, and explicit control flow were solved problems long before language models existed. Applying them to a new kind of unreliable component is, in the end, just software engineering with an unusually chatty dependency.

The practical implication for any team building LLM-powered systems today is to stop asking "how do we prompt this better" as the default question when something goes wrong, and start asking which of the three layers actually owns the failure. That single habit - diagnosing by layer before reaching for the prompt - is the closest thing this field currently has to a reliable debugging method, and it scales far better than intuition as a system grows from a single-turn assistant into a multi-step, tool-using agent operating with real consequences.

References