AI Agent Development: Architecture, Tools and Guardrails for Production

AI agent development is mostly software engineering around a language model: deciding which steps the model may choose for itself, giving it a small set of well-designed tools, keeping state outside the prompt, and putting hard limits on what it can do without a person. The model call is the easy part. Most production problems come from tools that do too much, context that grows without control, missing approval steps for irreversible actions, and no way to measure whether a change made the agent better or worse.

This article covers the architecture, ending with a reference design in pseudocode and a table of failure modes. The project side (use case selection, pilots, rollout) is in our companion piece on the AI agent development process. Dates and product status are as of October 2026.

What counts as an agent, and what is just a workflow?

The most useful working definition comes from Anthropic’s engineering article “Building effective agents” by Erik Schluntz and Barry Zhang, published in December 2024. It separates two kinds of system that are both often sold as “agents”. In a workflow, LLM calls and tools are arranged along code paths that a developer defined in advance. In an agent, the model directs its own process: it decides which tool to call next, reads the result, and decides again, until it judges the task finished or hits a limit.

The article describes five workflow patterns, and they are worth knowing by name because most business automation fits one of them:

An autonomous agent is the remaining case: an open-ended loop in which the number and order of steps cannot be predicted. Anthropic’s advice is to find the simplest solution and add complexity only when it improves results. We agree, for a plain reason: a workflow has a known set of paths, so you can test each of them, estimate its cost per run and explain to an auditor what happened.

So most business cases should start as workflows. Invoice extraction, ticket triage and report drafting have a known shape. They become agent candidates only when the path depends on what the system finds along the way, such as a research question across sources of unknown relevance. Even then, the agent is usually one node inside a larger workflow, with deterministic steps before and after it.

How does the agent loop work?

Almost every agent framework implements some version of ReAct, from Yao et al. (2022), the Princeton and Google paper “ReAct: Synergizing Reasoning and Acting in Language Models”, published at ICLR 2023. The idea is to interleave reasoning traces with actions: the model writes down what it thinks, takes an action such as a search or an API call, reads the observation, and continues. The paper showed that this combination reduced hallucination compared with reasoning alone on knowledge tasks such as HotpotQA, because the model could check facts against an external source instead of relying on memory.

Modern APIs turned this into native function calling: the model returns a structured tool call, the application runs it and appends the result. The loop in its minimal form:

messages = [system_prompt, user_task]
for step in range(MAX_STEPS):
    response = model(messages, tools=TOOL_SCHEMAS)
    if response.is_final_answer:
        return response.text
    for call in response.tool_calls:
        check_policy(call)               # permissions, approval, budget
        result = run_tool(call)          # with timeout
        messages.append(tool_result(call.id, truncate(result)))
raise StepLimitExceeded

Three things in that sketch separate production agents from demos. The step limit is enforced in code. Every tool call passes a policy check that the model cannot talk its way around. And tool results are truncated or summarised before they go back into the context, because one database query or web page can otherwise fill the window and push the original instructions out of focus.

The loop also needs a stop condition beyond “the model says it is done”, for example “ticket updated and reply drafted”. When a run ends on a limit instead, it should leave a defined state that a person can pick up.

How should tools be designed?

Tools are the agent’s interface to your systems, and their design decides more about reliability than the choice of model. A tool definition has a name, a description the model reads, and a JSON Schema for its arguments. Everything in it is effectively prompt text.

Some rules that hold across providers:

• Keep the tool set small and distinct. Two tools with overlapping purposes (“search_customers” and “find_customer”) make the model pick the wrong one.
• Design tools around tasks, not around your API surface. Wrapping every REST endpoint one to one turns one job into a dozen calls, each a chance to go wrong.
• Constrain arguments in the schema. Enums for fixed choices, formats for dates, minimum and maximum for numbers, required fields marked as such. Validate again on the server side; schema-constrained generation reduces malformed calls but does not replace validation.
• Return errors the model can act on (“Customer ID must be 8 digits; you sent 7”), not stack traces.
• Return compact results: paginate, filter fields, return identifiers.

The more important split is between read and write tools. Read tools (search, look up, list) can usually run without friction. Write tools change something: they send an email, update a CRM record, issue a refund, merge a pull request. They need three properties that read tools do not.

First, idempotency. Agents and frameworks retry, and calls time out after the server has already done the work. A write tool should take an idempotency key derived from the run ID and step, so a repeated call does not create a second refund.

Second, a dry-run mode that returns what the tool would do (“would update 3 records”) without doing it. The dry-run result goes to a person or a rule, and only then is the real call made with the same arguments.

Third, a risk class (read, reversible write, irreversible write), with orchestration code, not the model, deciding what each class requires. A typical policy: reads run freely, reversible writes are logged, irreversible writes need human approval.

tools:
  - name: get_invoice
    class: read
  - name: draft_reply
    class: reversible_write      # creates a draft, nothing is sent
  - name: send_email
    class: irreversible_write
    requires: human_approval
    idempotency_key: run_id + step
    dry_run: true
    rate_limit: 20/hour

What does the Model Context Protocol change?

Anthropic released the Model Context Protocol (MCP) as an open standard in November 2024. It defines how an application connects to servers that expose tools, resources and prompts, locally over standard input and output or remotely over HTTP with OAuth-based authorisation. A tool integration written once as an MCP server works with any MCP-capable client.

During 2025 OpenAI, Google and Microsoft added MCP support to their agent tooling and products, and in December 2025 Anthropic donated MCP to the Agentic AI Foundation, a directed fund under the Linux Foundation co-founded with Block and OpenAI. As of October 2026 MCP is maintained there under open governance and is the default way most agent frameworks connect to external tools. Google’s Agent2Agent protocol (A2A), for communication between agents, is also hosted by the Linux Foundation but so far less widely adopted.

For an enterprise, the benefit is that internal systems (ERP, document management, ticketing) can be wrapped once as MCP servers with their own access control and reused across agents. The risk is that every MCP server is code running with credentials, and its tool descriptions are text that enters the model’s context, so a third-party server with a manipulated description is a prompt injection channel. Pin versions, review what servers expose, give each its own narrowly scoped service account, and do not let agents connect to arbitrary servers at runtime.

How do agents handle memory and state?

“Memory” in agent systems covers three different things, and mixing them up leads to bad designs.

Short-term context is the message history of the current run. It is limited by the context window and, in practice, by quality well before the limit: models attend less reliably to material in the middle of a long context, and long runs accumulate stale tool output. Truncate tool results, summarise old steps, and keep the task and constraints at a stable position.

Run state is what the application needs to resume or audit a run: current step, tool calls made, approvals pending, budget used. It belongs in a database, not in the prompt, so a run can resume after a crash or a two-day wait for approval. LangGraph’s checkpointers, which persist graph state after each step, are built on this idea.

Long-term memory persists across runs: user preferences, facts about a customer, outcomes of earlier tasks, kept in a table or vector index the agent reaches through tools. Two decisions matter. Who may write: if the model writes freely, one injected instruction can persist into future runs, so writes should be structured and validated. And what gets deleted: under GDPR, memory about identifiable people is personal data, so the store needs a retention policy and a way to erase a person’s data on request.

Summarisation bridges the three: old turns are replaced by a model-written summary, and at the end of a run a structured summary goes into long-term memory. Summaries lose detail and their errors compound, so keep raw traces in your logs.

When should an agent decide what to retrieve?

Classic retrieval-augmented generation retrieves once, before the model answers: embed the question, fetch the top passages, generate. That works well for direct questions against a document set, and it is what most internal knowledge assistants do (we describe that set-up in our article on a RAG chatbot for internal documents).

Agentic RAG gives the retrieval decision to the model. Retrieval becomes one or more tools (“search_policies”, “search_tickets”, “get_document”), and the agent decides whether to search at all, which source to search, how to rephrase the query, and whether the results are good enough or another search is needed. The academic background is Self-RAG by Asai et al. (2023) (University of Washington, Allen Institute for AI and IBM Research; ICLR 2024), which trained a model to emit reflection tokens deciding when to retrieve and whether passages are relevant and support the answer. Production agents get a similar effect with tool calls and a grading step.

Agentic retrieval pays off when questions need several hops (“which of our suppliers in Poland have contracts expiring before the new tariff applies?”) or when the right source depends on the question. It costs more calls and latency, and adds failure modes: stopping too early, looping on near-identical queries, or answering from model knowledge without retrieving. Require at least one cited passage, cap search calls, and evaluate retrieval separately from answer quality.

The search tool must filter by the permissions of the user on whose behalf the agent acts, not those of the agent’s service account. Otherwise the agent becomes a way around your document permissions.

Single agent or multi-agent?

Multi-agent systems split work across several model instances with different prompts, tools or models, such as a planner with executors or a supervisor routing to specialists. The appeal is separation of concerns and parallel work. The costs are well documented.

Anthropic’s engineering write-up “How we built our multi-agent research system” (June 2025) is candid about both sides. Its lead agent, which delegates to parallel subagents, outperformed a single agent by about 90 percent on Anthropic’s internal research evaluation. The same article reports that agents use about four times the tokens of a chat interaction and multi-agent systems about fifteen times,, and says multi-agent set-ups fit tasks that parallelise well and are valuable enough to pay for that, and fit poorly where agents must share much context, which it says applies to most coding tasks.

On the failure side, Cemri et al., “Why Do Multi-Agent LLM Systems Fail?” from UC Berkeley and collaborators (2025, NeurIPS 2025 Datasets and Benchmarks track), analysed execution traces from seven open-source multi-agent frameworks and built a taxonomy (MAST) of 14 failure modes in three groups: system design and specification issues, inter-agent misalignment, and task verification. Their central finding is that many failures come from system design (unclear role specifications, agents ignoring each other’s input, ending early, weak verification) and not from the limits of the underlying model, so better models alone do not fix a badly designed system.

Our rule of thumb follows from these two sources. Use a single agent, or a workflow with one agent step, by default. Move to multiple agents when at least one of these holds:

• The task splits into parts that run in parallel with little shared context, such as searching several independent sources.
• The parts need different permissions, and separating them enforces least privilege (a read-only research agent, a review step, then an agent that may write).
• A single agent’s context fills with material from different subtasks and its quality drops measurably.

When you do build multi-agent, define hand-offs between agents as structured data; give one component the job of verifying the final result against the original task; and trace every agent’s steps under one run ID, or debugging becomes guesswork.

How do you choose and route models?

Model choice in an agent is per step, not per system. Planning and tool selection on ambiguous tasks benefit from the strongest model you can use; classification, extraction, summarising tool output and simple routing often work as well on a smaller, faster model. A common pattern is a strong model for the orchestrator and smaller models for subagents or for well-defined steps, which is also what Anthropic’s research system did.

Constraints usually narrow the field before quality does. Data residency may require EU-hosted endpoints or models you run yourself (see our notes on LLM hosting). Open-weight models vary more between versions in how reliably they follow tool schemas, so test function calling on its own. Latency matters more than in chat because it multiplies with the number of steps.

Practical rules:

• Pin model versions; “latest” aliases change agent behaviour under you.
• Keep the model behind an interface so switching providers is a configuration change.
• Route by measured results: use a smaller model for a step type only if task success on your evaluation set holds.
• Have a tested fallback for provider outages.

Which frameworks are worth using, and when is plain code better?

As of October 2026 the main options have settled somewhat after two years of churn. A short overview (a longer comparison is in our article on AI agent development frameworks):

Plain code is often the better choice. A workflow with one or two model calls and a handful of tools is easier to read, test and debug as a hundred lines around the provider’s SDK, with no dependency that changes its API every quarter. Anthropic’s article makes the same point: frameworks add layers that hide prompts and responses.

Frameworks earn their place for durable state across long approval waits, complex branching graphs, standard tracing, or several teams on one platform. Keep business logic, tools and policies in your own modules so the framework stays replaceable.

Guardrails: what keeps an agent safe in production?

The OWASP Top 10 for LLM Applications, 2025 edition, puts prompt injection first (LLM01) and lists excessive agency (LLM06) separately: a system given more functionality, permissions or autonomy than its task needs. In agents the two combine. An agent that reads untrusted content and holds broad write permissions can be steered by that content into actions nobody intended.

Indirect prompt injection is the variant that matters most for agents. Greshake et al. (2023), “Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection”, from CISPA Helmholtz Center and sequire technology, presented at the ACM AISec workshop, showed that instructions planted in content the model retrieves (a web page, an email, a document) can take control of LLM-integrated applications without the attacker ever talking to them directly. Their demonstrations included data theft and manipulation of Bing Chat. Every tool result is a potential carrier, from customer emails to MCP tool descriptions.

No prompt-level defence reliably prevents injection. The defences that work are architectural:

• Least privilege. Each agent gets only the tools its task needs, each tool runs with credentials scoped to it, and where possible the agent acts with the end user’s permissions, not a broad service account.
• Separate reading untrusted content from acting. An agent that has processed untrusted input should not, in the same context, be able to trigger high-impact writes. Either pass a structured, validated result to a separate step with write rights, or require approval.
• Human approval for irreversible actions. Payments, external emails, deletions, contract changes and production deployments go through an approval step showing the exact dry-run output. Approvers need enough context to judge, or approval becomes a reflex click.
• Output validation. Check tool arguments against allow-lists (recipient domains, account numbers, maximum amounts) in code before execution. Treat model output that goes into SQL, shell commands or HTML as untrusted, which OWASP lists separately as improper output handling.
• Sandboxing. Agents that execute code run in isolated containers or microVMs without default network access or production credentials, and with resource limits.
• Budgets, step limits and timeouts. Cap steps per run, tokens per run, tool calls per tool, wall-clock time per run and per tool call. OWASP’s unbounded consumption entry (LLM10) covers the cost and denial-of-service side; in practice the more frequent cause is a loop where the agent retries the same failing call.
• Egress control. Limit which hosts tools can reach; exfiltration usually needs an outbound channel such as a URL with data in its parameters.

Classifier-based input filters (“detect prompt injection”) are useful as one signal, but treat them as a tripwire, not a wall. Design so that a successful injection can do little damage.

How do you evaluate an agent?

Agent evaluation has two levels, and you need both.

Task-level evaluation asks whether the agent reached the right end state. Build a set of realistic tasks from your domain (start with a few dozen, grow it from production failures) with checkable outcomes. Check end states in code where possible, use rubric-based model grading where not, and spot-check the grader against human judgement.

Trajectory evaluation looks at how it got there: right tools, sensible order, no unnecessary steps, nothing touched that should not be. Two runs can both succeed, one in four steps and one in twenty with a near-miss on a write tool.

Because agents are non-deterministic, run each task several times. The tau-bench paper by Yao, Shinn, Razavi and Narasimhan (2024), from Sierra, introduced the pass^k metric for this: the probability that an agent succeeds on all k attempts at the same task. In its simulated retail and airline customer-service domains, where a model plays the user and the agent must follow domain policies while calling APIs, success fell noticeably as k grew. For a business process, consistency is often what matters most.

Public benchmarks are reference points, not acceptance tests. SWE-bench, from Jimenez et al. (2023) at Princeton, published at ICLR 2024, tests whether a system can resolve real GitHub issues in Python repositories, verified by the projects’ own tests. Its history shows the limits of public benchmarks: in February 2026 OpenAI announced it would stop reporting SWE-bench Verified scores after an audit found flawed tests in a large share of the hard tasks it reviewed, and evidence that frontier models had seen some of the solutions during training. A benchmark says roughly how capable a model is at a type of task, and nothing about your tools, data or policies.

Run the evaluation set in CI on every change to prompts, tools, models or framework versions; agents regress in odd places when a tool description is reworded.

What does observability look like for agents?

Every run should produce a trace: a root span for the run, child spans for each model and tool call, with inputs, outputs, tokens, latency, model version and policy decisions. Without it, “why did the agent do that?” has no answer, and cost control is guesswork.

The OpenTelemetry project has GenAI semantic conventions that define standard attribute names and span types for model calls, tool execution and agent invocation (spans such as “invoke_agent” and “execute_tool”), plus conventions for MCP. As of October 2026 they are still marked as in development, not stable, and in June 2026 they moved to a dedicated repository so they can be versioned on their own. Several agent frameworks and observability vendors emit these attributes already. Use them, pin the convention version you build against, and expect some renames.

Traces contain prompts and tool outputs, often with personal data, so they need the same access control and retention rules as the source systems. Put the run ID into the ticket, CRM record or approval log, so a complaint leads straight to the trajectory.

Where does the EU AI Act apply to agents?

The AI Act, Regulation (EU) 2024/1689, classifies AI systems by use, not by architecture. Whether something is an agent or a workflow does not decide its risk class.

Most business agents (document processing, internal support, IT operations) fall outside the high-risk categories. An agent becomes high-risk when used in an Annex III area, such as evaluating job applicants or credit scoring. The Digital Omnibus amendment to the AI Act, in force since July 2026, moved the application date for Annex III high-risk obligations to 2 December 2027, and for high-risk systems under Annex I product legislation to 2 August 2028.

The transparency obligations in Article 50 were not postponed and have applied since 2 August 2026. Systems intended to interact directly with people must inform them that they are dealing with an AI system, unless that is obvious from the context, so a customer-facing agent that answers emails or chats must disclose it. Providers of systems generating synthetic content must also mark outputs in a machine-readable way; for systems on the market before 2 August 2026, the Omnibus allows until 2 December 2026 for that marking duty.

In Germany, an agent that monitors or evaluates employees usually involves the works council under the Works Constitution Act, independent of the AI Act. Plan for that early. For a wider view on governance, see our overview of AI governance consulting companies.

A reference architecture in pseudocode

The components above fit together like this. It is deliberately framework-neutral.

Gateway
  - authenticate user, attach user identity and permissions
  - AI disclosure for external users (AI Act Art. 50)

Orchestrator (workflow engine, owns the run)
  run = RunStore.create(task, user, budget={steps: 25, tokens: N, minutes: 10})
  pre_steps:  classify(task) -> route to fixed workflow OR agent node
  agent node:
    loop until done or budget exhausted:
      ctx   = ContextBuilder(run.state, memory.read(user, scope), task)
      step  = Model[router.pick(step_type)].call(ctx, tools=Registry.for(task_type))
      for call in step.tool_calls:
        decision = Policy.check(call, user, run)     # least privilege, allow-lists
        if decision == NEEDS_APPROVAL:
            preview = Tools.dry_run(call)
            ApprovalQueue.submit(run, call, preview); RunStore.suspend(run); exit
        if decision == DENY: append error for model; continue
        result = Tools.execute(call, idempotency_key=run.id + step.n,
                               timeout=call.tool.timeout, sandbox=call.tool.sandbox)
        run.state.append(call, summarise_if_large(result))
      RunStore.checkpoint(run)
  post_steps: validate output schema, citation check, Verifier(task, output)
  memory.write(user, structured_summary(run))        # validated, retention-tagged

Tool registry (MCP servers or in-process)
  - each tool: schema, risk class, scoped credentials, timeout, rate limit, dry-run

Cross-cutting
  - Tracing (OpenTelemetry GenAI conventions), cost metering per run
  - Eval suite in CI: task success, trajectories, pass^k on critical tasks
  - Kill switch per agent and per tool

The model appears in two lines. The rest is ordinary engineering.

Failure modes and mitigations

FAQ

What is the difference between an AI agent and an AI workflow?

In a workflow, the developer defines the sequence of LLM calls and tools in code; the model fills in steps but does not choose the path. In an agent, the model decides which tool to call next based on what it has seen so far, until it finishes or hits a limit. Anthropic’s “Building effective agents” draws the line this way.

Do we need a framework for AI agent development?

No. For a workflow or a single agent with a few tools, plain code around the provider’s SDK is often easier to test. Frameworks pay off for durable state, long approval waits, complex graphs or one platform for several teams.

Is MCP required to build an agent?

No, but as of October 2026 it is the common standard for connecting agents to tools, governed under the Linux Foundation’s Agentic AI Foundation. Each MCP server still needs its own access control, version pinning and review.

How do you protect an agent against prompt injection?

Assume it will happen and limit the damage: least privilege, no high-impact writes in the same context as untrusted content, human approval for irreversible actions, argument allow-lists in code and restricted outbound access.

When is a multi-agent system worth it?

When the work splits into parallel parts with little shared context, or when separate agents enforce different permissions. Anthropic reported about fifteen times the tokens of a chat for its multi-agent system, and Cemri et al. (2025) traced many failures to system design, so the extra cost needs measured gains to justify it.

Does the EU AI Act classify agents as high-risk?

Not as such. Risk depends on the use: an agent screening job applicants is high-risk, one sorting invoices is not. Customer-facing agents must disclose that people are interacting with AI under Article 50, which has applied since 2 August 2026.

Building agents with AI Superior

AI Superior is an AI engineering company in Darmstadt, founded in 2019 by AI scientists. We design and build LLM applications, RAG systems and AI agents for B2B clients, with the parts this article describes: tool design, permission and approval layers, evaluation suites and tracing. If you are deciding whether a process needs an agent or a well-built workflow, our AI automation services and LLM development and consulting teams can help scope and build it. For budgeting, see our breakdown of AI agent development cost, and for the step-by-step project plan, the AI agent development process.

Sources and further reading

• Schluntz, E. and Zhang, B. (Anthropic), “Building effective agents”, December 2024.
• Anthropic, “How we built our multi-agent research system”, June 2025.
• Anthropic, “Introducing the Model Context Protocol”, November 2024; Model Context Protocol specification.
• Yao, S. et al., “ReAct: Synergizing Reasoning and Acting in Language Models”, 2022 (ICLR 2023).
• Asai, A. et al., “Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection”, 2023 (ICLR 2024).
• Cemri, M. et al., “Why Do Multi-Agent LLM Systems Fail?”, 2025 (NeurIPS 2025 Datasets and Benchmarks).
• Greshake, K. et al., “Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection”, 2023 (ACM AISec).
• OWASP, “Top 10 for LLM Applications”, 2025 edition.
• Yao, S., Shinn, N., Razavi, P. and Narasimhan, K., “τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains”, 2024.
• Jimenez, C. E. et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?”, 2023 (ICLR 2024).
• OpenTelemetry, Semantic Conventions for Generative AI systems.
• Regulation (EU) 2024/1689 (AI Act), as amended by the Digital Omnibus on AI, 2026.