AI Agent Architecture: Components, Memory and Orchestration

AI agent architecture is the set of components around a language model that decide what it sees, what it may do and what happens when it is wrong. A workable design has eight parts: an entry point that establishes identity, a control loop, a model adapter, a context builder, a tool layer, a policy gate, a state store and a trace. Memory and additional agents are optional, and each one brings failure modes you will have to pay for. The dividing line that shapes everything else: the model proposes, and deterministic code decides what actually runs.

This post is a reference for that structure, to read one component at a time. To walk through a build in order, with running Python for the loop, see how to build an AI agent. Production practice such as tool design rules, guardrails and evaluation is in our article on AI agent development. Product and protocol status is as of November 2026.

The architecture diagram

                 user or calling system
                           |
                           v
               +-----------------------+
               | ENTRY POINT           |   who is asking, on whose
               | identity, disclosure  |   permissions, with what budget
               +-----------+-----------+
                           |
                           v
 +-------------------------+----------------------------+
 | CONTROL LOOP (your code; owns the run)               |
 |                                                      |
 |   +-----------------+        +-------------------+   |      +--------------+
 |   | CONTEXT BUILDER |------->| MODEL ADAPTER     |---+----->| model        |
 |   | what the model  |        | one call, one     |<--+------| (hosted or   |
 |   | sees this step  |        | neutral format    |   |      | self-run)    |
 |   +--+-----------+--+        +---------+---------+   |      +--------------+
 |      ^           ^                     |             |
 |      |           |                     | proposal:   |
 |      |           |                     | text or     |
 |      |           |                     | tool call   |
 |      |           |                     v             |
 |      |           |           +-------------------+   |      +--------------+
 |      |           |           | POLICY GATE       |---+----->| person       |
 |      |           |           | allow, deny, or   |<--+------| (approval    |
 |      |           |           | ask a person      |   |      |  queue)      |
 |      |           |           +---------+---------+   |      +--------------+
 |      |           |                     | allowed     |
 |      |           |                     v             |
 |      |           |           +-------------------+   |      +--------------+
 |      |           +-----------| TOOL LAYER        |---+----->| your systems |
 |      |        tool results   | registry, schemas,|<--+------| and data     |
 |      |        (UNTRUSTED)    | scoped credentials|   |      | (UNTRUSTED   |
 |      |                       +-------------------+   |      |  content)    |
 +------+------------------+----------------------+-----+      +--------------+
        ^                  |                      |
        v                  v                      v
 +--------------+   +--------------+   +----------------------------+
 | MEMORY       |   | STATE STORE  |   | TRACE                      |
 | across runs  |   | this run,    |   | every proposal, decision,  |
 | (optional)   |   | checkpoints  |   | call; feeds evaluation     |
 +--------------+   +--------------+   +----------------------------+

The same diagram in words

A request arrives at the entry point, which attaches an identity, a set of permissions and a budget, then starts a run. The control loop owns that run from here on.

On each step, the context builder assembles what the model will see: instructions, the task, recent tool results, anything read from memory. The model adapter sends that to the model and returns one of two things, a final answer or a proposed tool call. A proposal is only a proposal. It goes to the policy gate, which allows it, denies it with an error the model can read, or suspends the run and asks a person. An allowed call goes to the tool layer, which runs it with credentials scoped to that tool and passes the result back to the context builder for the next step.

Two facts about the picture matter more than its boxes. Content coming back from tools is untrusted, because a customer email or web page can contain instructions, and it re-enters the model’s context on the next step. And the policy gate sits after the model on the way out, where nothing the model reads or writes can remove it. The state store, memory and trace sit underneath the loop: the state store holds this run, memory carries information across runs, and the trace records all of it.

The components, one at a time

Entry point

The entry point answers three questions before any model call: who is asking, on whose permissions the agent will act, and how much budget this run has. If it is skipped, the agent falls back on one broad service account, and every user can reach whatever that account can reach. OWASP’s Top 10 for Agentic Applications (December 2025) lists this class of problem as ASI03, Identity & Privilege Abuse. NIST’s CAISI launched an AI Agent Standards Initiative in February 2026 with identity among its research pillars, but its page listed no final agent guidance as of the August 2026 update.

For customer-facing agents, the AI Act’s transparency duty in Article 50, which has applied since 2 August 2026, is also handled most easily here.

Control loop

The loop is ordinary code that alternates model calls and tool calls until a stop condition fires. It owns the step count, the deadline and the decision that a run is over. It should hold no business logic. What belongs elsewhere: rules go to the policy gate, prompts to the context builder, integrations to the tool layer.

A sensible default is to embed the loop as one node of a larger workflow, with fixed steps before it (classify, route) and after it (validate the output). The how-to post has a runnable version with each stop condition listed.

Model adapter

One function, with one neutral format for messages and tool calls, translating to whichever provider you use. It carries the timeout, rate-limit retries, pinned model version and fallback. Per-step model routing also lives here, with a smaller model for summarising tool output if your evaluation set shows no loss.

Context builder

The context builder decides what the model sees on this step. Without it as a named module, messages simply accumulate. Its jobs are to keep the instructions and task in a stable position, trim or summarise old tool output, load only the tools relevant to the task, and label which content came from outside.

OWASP’s 2026 LLM list has an entry called Hidden Context Exposure (LLM08). Our rule for this component: never put a secret in context on the assumption that the model will keep it.

Tool layer

The tool layer is a registry: for each tool, a schema, a risk class, its credentials, a timeout and a rate limit. The risk class (read, reversible write, irreversible write) is what connects it to the policy gate. Tools can be in-process functions or remote MCP servers.

For remote tools, the MCP revision of 28 July 2026 changes the deployment picture. Protocol sessions and the initialize handshake are gone, version and capabilities travel with each request, and server/discover is mandatory. Our reading is that this makes remote MCP servers easier to run behind ordinary load balancers, since the protocol no longer holds a session, but any state your tool needs is now your problem. HTTP+SSE, Sampling, Roots and Logging are among the deprecated parts, with a twelve-month minimum window, so check what your existing servers rely on.

Supply chain is the structural risk here, since every remote server is code holding credentials whose descriptions enter the model’s context. OWASP’s LLM04 and ASI04 cover it, and MITRE ATLAS has a technique for it, AI Agent Tool Poisoning (AML.T0110).

Policy gate

The gate is deterministic code that inspects each proposed call against the user’s permissions, allow-lists, budgets and the tool’s risk class, then returns allow, deny or ask. The model cannot edit it, and it does not depend on the model’s cooperation. Of all the components, this one most limits damage when something else fails. OpenAI’s practical guide names two triggers for human intervention: exceeding failure thresholds, and actions that are “sensitive, irreversible, or have high stakes”. Both are implemented here.

An architecture without a gate relies on the prompt for safety, and that is a poor bet. “The Attacker Moves Second” (Nasr, Carlini et al., October 2025) reports that adaptive attacks bypassed 12 published defences against prompt injection, most at above 90% success, though most of those defences had reported near-zero attack rates. Our post on AI agent security covers the threat model and controls.

State store

Run state is everything needed to resume or audit one run: step number, messages or a pointer to them, calls made, approvals pending, budget used. It belongs in a database. Keep it out of the prompt. The need becomes obvious with approvals, since a run suspended for two days on someone’s desk cannot live in process memory. LangGraph’s documentation calls this short-term memory, thread-scoped state kept through checkpoints, and the OpenAI Agents SDK lists sessions. Whatever the word, write a checkpoint after each step, and treat write tools as retryable after a crash, which means giving them idempotency keys.

Trace and evaluation hooks

Every run should leave a record of each context sent, proposal returned, gate decision and tool result under one run ID. The evaluation harness reads the same record: what Anthropic’s “Demystifying evals for AI agents” (January 2026) calls the transcript is, in a production system, this trace. OpenTelemetry’s GenAI semantic conventions list agent spans but are still marked Development, so adopt them behind your own thin wrapper and a rename touches one file.

Traces hold prompts and tool output, often personal data. In Germany, traces that show who used the agent and how are also a likely works council topic, as our process article explains.

AI agent memory

“Memory” names several things, and the design questions differ for each. LangChain’s LangGraph documentation splits it into short-term memory (the state store above) and long-term memory across sessions, the latter divided into three types: semantic, episodic and procedural. That is one vendor’s vocabulary, applied here to storage decisions.

We have one strong opinion on this table: start with no long-term memory at all. Most first agents can read what they need from the source system through a tool, and the source system stays correct and has an owner. Memory adds a second copy that can go stale or be poisoned. One injected instruction written to memory reaches later runs that never saw the attacker’s content. OWASP lists Memory & Context Poisoning as ASI06, and MITRE ATLAS has a Memory sub-technique under AI Agent Context Poisoning (AML.T0080.001).

Vector indexes bring their own entry in the OWASP 2026 list, LLM09, Vector and Embedding Weaknesses. If memory holds documents, retrieval has to filter by the permissions of the user being served, as in our RAG chatbot for internal documents design.

Memory about identifiable people is personal data under the GDPR, so every long-term store needs a retention period and a way to erase one person’s entries. That is much easier if entries are structured and tagged by data subject from the start, and painful to add to prose summaries later.

Types of AI agents

The textbook classification comes from Stuart Russell and Peter Norvig’s Artificial Intelligence: A Modern Approach. It predates LLMs and sorts agents by how they decide:

These types describe decision logic, so they say little about what you have to build around an LLM agent. Most LLM agents behave as goal-based, whatever else they do. The word “agent” itself also varies among vendors: Google Cloud’s overview uses a loose definition of applications that respond to natural-language requests, while OpenAI and Anthropic require the model to control execution. Our post on AI agents vs agentic AI compares those definitions. For architecture, three properties you can check tell you more, because each one changes what you must build.

A read-only, single-request agent choosing among fixed routes needs little beyond the loop and tool layer. A multi-day agent with irreversible writes needs all eight components, whatever its marketing category. Google’s ADK, for example, separates LLM agents, workflow agents and custom agents, which corresponds roughly to the first axis.

AI agent orchestration patterns

Orchestration decides how work moves when more than one model instance is involved. Microsoft’s “AI Agent Orchestration Patterns” (February 2026) names five: sequential, concurrent, group chat, handoff and magentic. OpenAI’s guide names two: a manager pattern, where a central LLM orchestrates, and a decentralised one, where agents hand off to each other. Microsoft’s advice, paraphrased, is to use the lowest level of complexity that reliably meets your requirements. We agree: each step across the table below adds new ways to fail.

Single agent

Stay here until a measured problem forces a move. Too many tools and an overfull context are the usual pressures, and both can often be fixed by a narrower task or by loading tools per task type. The structural weakness is that the agent marks its own homework and holds every permission in one place. A deterministic verification step fixes the first without any multi-agent machinery.

Orchestrator-workers

This pattern is OpenAI’s manager pattern in other words. Two reasons justify it. Independent parts can run concurrently, as when searching several unrelated sources. Or different parts need different permissions, so a read-only research worker that processes untrusted content never holds the write credentials.

The weak point is the boundary. A worker sees only what the orchestrator chose to tell it, and the orchestrator sees only what the worker chose to return. Prompt injection also crosses it: a worker that reads a poisoned document can pass the attacker’s instruction upwards inside its result. The relevant OWASP agentic entries are Agent Goal Hijack (ASI01) and Insecure Inter-Agent Communication (ASI07). Treat worker output as you treat tool output, untrusted and validated against a schema. A hand-off contract might look like this:

{
  "run_id": "r-20261105-0412",
  "parent_step": 3,
  "worker": "invoice_research",
  "permissions": ["erp.invoices.read"],
  "budget": {"steps": 8, "seconds": 120},
  "goal": "Find the purchase order that matches invoice INV-1001.",
  "must_return": {
    "status": "found | not_found | gave_up",
    "purchase_order_id": "string or null",
    "evidence": ["source record ids"]
  }
}

Note what the contract does not include: the orchestrator’s full conversation, and any credential. The worker gets a budget carved out of the run’s total, so fan-out cannot exceed the cap set at the entry point. Permissions can only narrow on the way down.

An attributable public example is Allianz’s Project Nemo in Australia. Allianz reported seven agents handling food spoilage claims under AUD 500, live since July 2025, with an 80% cut in processing and settlement time and a person making every payout decision. These are Allianz’s own figures, and we found no independent audit. Allianz’s article does not settle which pattern this is, so we draw only one structural point from it: many agents, one narrow task, and the irreversible step kept with a person.

Hierarchical

A hierarchy is orchestrator-workers applied recursively, and we would want a strong argument before using it in a company’s first few systems. Each level summarises for the level above, so the goal drifts going down and detail is lost coming up. A wrong result can be picked up by sibling branches before anyone notices, which is OWASP’s ASI08, Cascading Failures. Delegation also tends to launder authority: unless permissions only narrow at each level, a low-level agent can ask a higher one to do what it may not, and that again falls under ASI03. Delegation loops need a depth limit in code.

If separate teams own separate domains, consider instead a flat orchestrator calling each team’s agent through a stable interface. A2A is the protocol built for agent-to-agent calls. Its v1.0 is its first stable specification, and the Linux Foundation reports more than 150 supporting organisations, though it does not define “supporting” and names no production customers. We found no independent data comparing its adoption with MCP’s.

Handoffs

In a handoff, control passes between agents with no central planner, the pattern OpenAI calls decentralised. It suits triage, where one agent routes to a specialist and leaves. The failure modes are agents bouncing a case between them and unclear ownership of the final result, so assign one owner per run and cap the number of handoffs.

Build or take from a framework

Frameworks differ mainly in which of these components they supply. The descriptions below are the vendors’ own.

Whatever you adopt, keep the policy gate and tool registry in your own code. They encode your rules, and they are what you least want to rewrite when a framework changes or, like AutoGen, is archived. The state store is a fair thing to take from a framework, and so is the loop itself.

Where each component runs, in EU terms

The diagram doubles as a data map for a data protection review. The model adapter decides whether prompts leave your region, so if residency matters, an EU-hosted endpoint or a self-run model goes in that box. The context, state store, memory and trace all hold copies of whatever the agent reads, and each copy needs its own access control and retention rule.

The AI Act classifies by use, so no architecture choice removes an agent from the high-risk category. An agent assessing the creditworthiness of natural persons falls under Annex III 5(b) however it is built, with obligations for Annex III systems applying from 2 December 2027 after the Digital Omnibus, as the European Commission’s page states. Architecture can make the obligations practical to meet: the trace gives you records, and the policy gate a documented point of human oversight.

Frequently asked questions

What is the difference between AI agent memory and state?

State belongs to one run and exists so it can resume and be audited. Long-term memory carries information across runs. LangGraph’s documentation calls the first short-term memory and divides the second into semantic, episodic and procedural.

What should an AI agent architecture diagram include?

At minimum, the path from model proposal through a policy gate to tools, the return path for untrusted tool results, where a person approves, and where state and traces are stored. A diagram showing only “LLM, tools, memory” leaves out the parts that limit damage.

When is orchestrator-workers better than a single agent?

When parts of a task can run in parallel with little shared context, or need different permissions. For a cost view, see our article on multi-agent system development cost. Otherwise, a single agent is easier to test and debug.

Does an agent architecture need MCP or A2A?

Neither is required. MCP is useful when several agents or clients share tools, and A2A when agents owned by separate teams or companies call one another. In-process functions and plain function calls are fine inside one system.

Designing an agent architecture with AI Superior

AI Superior is an AI engineering company in Darmstadt, Germany, founded in 2019 by AI scientists, and we build LLM applications, RAG systems and AI agents for B2B clients. Our AI agent development services cover design and build. Where identity, audit and existing systems drive the design, see enterprise AI agent development.

Sources and further reading

• OpenAI, “A practical guide to building agents”, 2025.
• Anthropic, “Building effective agents”, December 2024.
• Russell, S. and Norvig, P., Artificial Intelligence: A Modern Approach.
• Grace, Hadfield, Olivares, De Jonghe (Anthropic), “Demystifying evals for AI agents”, January 2026.
• Google Cloud, AI Agent Ecosystem overview, undated.
• Microsoft, “AI Agent Orchestration Patterns”, February 2026.
• LangChain, LangGraph memory documentation, read November 2026.
• Project documentation: OpenAI Agents SDK, Claude Agent SDK, Microsoft Agent Framework, Google ADK, CrewAI, read November 2026.
• Model Context Protocol, “Key Changes” for revision 2026-07-28.
• Linux Foundation, press release on A2A, April 2026.
• OWASP, GenAI LLM Top 10 2026; Top 10 for Agentic Applications for 2026, December 2025.
• MITRE ATLAS, content version 2026.10, October 2026.
• NIST CAISI, AI Agent Standards Initiative, February 2026, page updated August 2026.
• OpenTelemetry, semantic conventions for generative AI.
• Nasr, Carlini et al., “The Attacker Moves Second”, October 2025.
• Allianz, Project Nemo article, November 2025.
• European Commission, AI Act regulatory framework page; AI Act Annex III.