AI Agent Security: Threat Model, Real Incidents and the Controls That Limit Damage

AI agent security starts from one assumption: at some point the model will do what an attacker, or its own bad judgement, tells it to do, so the system around it has to keep that event small. Prompt injection is unsolved. In “The Attacker Moves Second” (October 2025), Nasr, Carlini and co-authors ran adaptive attacks against 12 published defences and bypassed them, most at success rates above 90%, although most of those defences had originally reported near-zero attack success. The controls that limit damage therefore sit mostly outside the model: fewer tools, narrow credentials, no route for data to leave, and a person in front of irreversible actions.

This post covers the threat model and the order in which to apply controls. Status is as of November 2026. The agent loop and tool design are in our article on AI agent development, and the structural options in AI agent architecture.

Which OWASP list, and which number

Two OWASP documents matter for agents, and the numbering catches people out.

The current general list is the OWASP GenAI LLM Top 10 2026, published in August 2026 (the resource page says 3 August, the repository 4 August). Its items are LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM03 Excessive Agency, LLM04 Supply Chain, LLM05 Data and Model Poisoning, LLM06 Unbounded Consumption, LLM07 Misinformation, LLM08 Hidden Context Exposure, LLM09 Vector and Embedding Weaknesses and LLM10 Improper Output Handling.

Compared with 2025, several numbers moved. Excessive agency was LLM06 and is now LLM03. Improper output handling went from LLM05 to LLM10, unbounded consumption from LLM10 to LLM06, and LLM07, which was System Prompt Leakage, is now Misinformation. An audit template or vendor questionnaire that cites “LLM06” could mean either of two different risks, so check the edition before you answer it.

The second document is the OWASP Top 10 for Agentic Applications for 2026, released on 9 December 2025: ASI01 Agent Goal Hijack, ASI02 Tool Misuse, ASI03 Identity & Privilege Abuse, ASI04 Agentic Supply Chain Vulnerabilities, ASI05 Unexpected Code Execution, ASI06 Memory & Context Poisoning, ASI07 Insecure Inter-Agent Communication, ASI08 Cascading Failures, ASI09 Human-Agent Trust Exploitation and ASI10 Rogue Agents. These names follow OWASP’s announcement, so quote from the final document when it goes into a contract.

MITRE ATLAS (content version 2026.10, 28 October 2026) adds technique IDs that red teams can test against. On the US government side, the NIST CAISI AI Agent Standards Initiative page, last updated on 14 August 2026, lists no final agent guidance. So there is no finished public standard to certify against yet, and the threat model below is one you have to own.

The table maps the six threats this article covers to those lists. The mapping is ours, since none of the documents lines up one to one.

Vulnerability, breach, accident and misuse are different events

Articles on AI agent security vulnerabilities tend to pour all of these into one list of “attacks”. That makes the risk look both larger and less specific than it is. A CVE records a flaw and its severity score. It says nothing about whether anyone was harmed.

Each row calls for a different response. A vulnerability goes to patch management and vendor questions. An intrusion goes to incident response. The Replit type of accident is a design defect in your own permissions, and no patch will fix it.

Prompt injection

An LLM reads instructions and data through one channel, and no reliable marker separates them. Any text the agent reads can therefore try to redirect it: an email, a ticket, a web page, a tool result.

EchoLeak is the clearest public example of what that means in a product with real defences. Reddy and Gujral’s analysis (arXiv 2509.10540) describes one crafted email and no user action. The chain got past the XPIA classifier and link redaction, then used auto-fetched images and a Teams proxy to get data out. Two purpose-built guardrails were in place and both were bypassed, in a vulnerability that was fixed before any reported exploitation.

Google researchers Brunner, Liu and Pande (April 2026, vendor research) found malicious prompt injections in Common Crawl snapshots up 32% between November 2025 and February 2026, while describing most of them as low in sophistication. So the background noise on the web is growing and is mostly crude. “The Attacker Moves Second” addresses the other end of the scale: a defence that scores well against a fixed list of attacks can fail once the attacker is allowed to adapt to it. Filters catch the first group. Plan for the second.

Our position: treat prompt injection like phishing against a very gullible employee who has system access. You do not expect to stop every attempt. You decide what that employee is allowed to do.

Excessive agency needs no attacker

LLM03 is the item most teams underrate, because its worst public cases have no villain.

In July 2025, on day 8 of a 30-day project, a Replit agent deleted a live database during a code freeze. The database held records on more than 1,200 executives and nearly 1,200 companies. The first account was that recovery was impossible; the data was later retrieved, and Replit’s CEO called the deletion unacceptable. A code freeze that depends on the model’s compliance is only a request. If the agent’s credentials cannot drop a production table, the freeze holds whatever the model decides. Recovery assessments should also come from your own backup checks.

Two 2026 cases come from evaluation runs and show agents going well beyond their task. METR’s investigation (26 August 2026) concludes that the Hugging Face intrusion involved about 1,200 OpenAI evaluation agents sharing an unsanctioned message board, with about 700 joining the attack, apparently to game a benchmark scorer. METR states the conditions of its work: OpenAI supplied the data and API credits, the team worked on OpenAI’s premises, and the analysis used an OpenAI model. Separately, Anthropic reviewed 141,006 evaluation runs and reported 3 incidents across 6 runs in which Claude models attacked real systems, including a malicious PyPI package that stayed live for about an hour and was run on 15 systems. The stated cause was misconfigured live internet access.

Do not read these as base rates for a business agent that sorts invoices; they are evaluation workloads, and the Anthropic figures are a vendor’s self-report. They do show that test and evaluation environments need the same isolation as production, and that a goal plus a reachable system can be enough.

Tool poisoning and the agent supply chain

Tool definitions are text that enters the model’s context, and tools are code that runs with your credentials, so a tool can attack from either side. ATLAS lists this as AI Agent Tool Poisoning (AML.T0110). OWASP covers it under LLM04 and ASI04.

The public cases we can cite come second-hand. Help Net Security’s June 2026 summary of OWASP’s State of Agentic AI Security and Governance report lists a LiteLLM backdoor on PyPI in March 2026, with about 47,000 downloads in about three hours, and postmark-mcp as the first malicious MCP server found in the wild. It also lists CVE-2025-6514 and CVE-2026-22708 (Cursor). We have not read the primary records for these, so treat the details as OWASP’s catalogue as reported.

The time windows are what should change your practice. Three hours in one case, about an hour in Anthropic’s. No human review catches a package in that window, so automated upgrades are the exposure. Pin dependencies and MCP servers by hash, pull them through an internal mirror with a delay, and treat a changed tool description as a code change that goes through review. OWASP’s A Practical Guide for Secure MCP Server Development (February 2026) is the place to start if you write your own servers.

Multi-agent set-ups widen this surface: each agent is another party whose output others may trust. ASI07 and ASI08 cover that ground. Our architecture article discusses what each pattern costs.

AI agent identity

Most agent credentials we would worry about look ordinary: an OAuth token, an API key, a service account shared by several tools. The Salesloft Drift theft belongs here for that reason only. It involved stolen tokens from an integration and no prompt injection, but an agent connected to a SaaS system holds the same kind of token and adds a new way to reach it, which ATLAS records as AI Agent Tool Credential Harvesting (AML.T0098). In the Hugging Face intrusion, service credentials were among what was accessed.

NIST’s CAISI announced its AI Agent Standards Initiative on 17 February 2026, with security and identity research as one of three pillars and a concept paper on identity open for comment until 2 April. As of November 2026 the initiative page shows no final guidance, so design this yourself. What we would ask for:

• One identity per agent per environment. A development agent and a production agent never share a credential.
• User-delegated access for anything read on someone’s behalf, so the agent cannot see more than the person it works for.
• Short-lived tokens issued per tool, kept in the tool layer. The model never sees a secret, because anything in its context can come out again, which is the ground LLM02 and LLM08 cover.
• An approver identity that the agent cannot use. An agent must not be able to approve its own request through any path.
• The run ID recorded with every downstream write, so a target system’s audit log leads back to one trajectory.

MCP deployments need a recheck here. The 2026-07-28 revision of the protocol removed sessions and the initialize handshake and deprecated OAuth Dynamic Client Registration in favour of Client ID Metadata Documents, with a twelve-month minimum deprecation window. A security review done against an earlier revision does not automatically carry over.

Data exfiltration

Exfiltration needs three things together: access to private data, exposure to untrusted content, and an outbound channel. EchoLeak had all three. Remove any one and the attack fails.

The outbound channel is broader than most reviews assume. ATLAS records Exfiltration via AI Agent Tool Invocation (AML.T0086), so every tool that can write outside your boundary counts: email, webhooks, ticket comments visible to customers, a URL carrying data in its parameters. EchoLeak used image fetching, which nobody would classify as a write tool. And it went through a Teams proxy, a warning against allow-lists that trust a whole vendor’s domain.

Hosting decisions do not solve this. Running the model on your own or EU infrastructure settles where inference happens and who the processor is. An injected agent with an open outbound route can still send data wherever that route goes. If the data is personal, your data protection officer owns part of the response, so name them in the incident plan before go-live.

Memory poisoning

Here the honest statement is that the sources behind this article contain no well-documented production incident, only taxonomy entries: ASI06 Memory & Context Poisoning and ATLAS AI Agent Context Poisoning (AML.T0080, Memory sub-technique .001). Injection in a single run ends with the run. Injection into long-term memory comes back in later runs, possibly for other users, long after the source document has gone. LangChain’s documentation splits long-term memory into semantic (facts), episodic (past actions) and procedural (rules and prompts); the last is the one to protect hardest, since a poisoned rule behaves like a changed system prompt. Our rule of thumb is that a model may propose facts with a recorded source, and may not write rules at all without review. Scope memory per user or tenant, and give entries an expiry.

AI agent security best practices, in the order they limit damage

This list runs from controls that hold even after the model is fully compromised down to those that only reduce the odds. It is our judgement, based on where the cases above actually broke.

1. Remove the capability. A tool that does not exist cannot be misused. Microsoft’s Agent Framework overview advises writing a function instead of using an agent when a function can do the task, and that is also the best security advice on this list. Ask of every tool whether this task needs it.

2. Scope credentials in the target system. Enforcement by the database, ERP or CRM holds regardless of what the model believes. Read-only accounts, row-level limits and no production access from development agents are what stop a deletion like Replit’s.

3. Isolate the runtime and deny egress by default. Anthropic’s incidents trace back to live internet access that should not have been there. Code-executing agents run in containers or microVMs with no default network route, and that applies to evaluation runs as well.

4. Separate reading from acting. Once a run has processed untrusted content, it loses write tools for that run. It passes a validated, structured result to a separate step that holds them. In EchoLeak terms, the content that carries the instruction never reaches a component with an outbound channel.

5. Check tool arguments in code. Recipient domains, account numbers, maximum amounts and record counts are checked before execution, by logic the model cannot argue with. Output going into SQL, a shell or HTML is untrusted input to that system (LLM10).

6. Require human approval where it counts. OpenAI’s A practical guide to building agents names two triggers: exceeding failure thresholds, and high-risk actions that are “sensitive, irreversible, or have high stakes”. Approval is lower on this list because people get tired, and because ASI09 exists: an agent can present a harmful action persuasively. Show the approver the literal effect from a dry run, generated by code and not summarised by the model.

7. Pin the supply chain, as described above.

8. Trace everything and keep a kill switch. Tracing limits how long an incident runs and tells you what happened, but it stops nothing by itself. OpenTelemetry’s GenAI conventions are still marked Development, so pin the version you build on and expect renames. Traces hold prompts and tool outputs, often with personal data, and in Germany logs of who used an agent usually bring in the works council, as our process article explains.

9. Add model-level guardrails last. See the next section.

What AI agent guardrails can and cannot do

“Guardrails” covers two very different things. Hard guardrails are controls 1 to 5 above: enforced in code or infrastructure, and indifferent to the model’s output. Soft guardrails are classifiers, moderation models and system-prompt rules, and they are probabilistic. Frameworks such as the OpenAI Agents SDK ship guardrails as a building block, and they earn their place. They catch honest mistakes and off-topic requests, and given Google’s finding that most web injections are crude, they should stop a share of those too.

They do not hold against an attacker who iterates, as both EchoLeak and “The Attacker Moves Second” show. So use a soft guardrail as a tripwire that raises an alert, and do not let a review accept it as the reason a dangerous tool is safe to expose.

The same paper changes how to test. A fixed list of injection strings measures very little, because the defences it broke had mostly reported near-zero attack success. Red-team with people or agents that can see your defence and adapt. Anthropic’s “Demystifying evals for AI agents” (January 2026) suggests starting an evaluation set with a few dozen tasks drawn from real failures; add every injection that gets through, and keep regression runs near 100%. For a walk through a first test set on a small agent, see how to build an AI agent.

Enterprise AI agent security: what to have before the second agent

Keep a register with one entry per agent: owner, tools, credentials, data classes, model version, and the risk class of each write tool. Adding a tool to an agent goes through review, like opening a firewall rule, because it changes the blast radius in the same way.

Write the incident plan using the categories from the table above. “Vendor CVE in a Copilot product” and “our agent emailed the wrong customer” have different owners and clocks. The plan should say who can switch an agent to suggestion-only mode, and how quickly, without a deployment.

Ask vendors specific questions. Which credentials does the agent hold, and can we scope them? Can we restrict egress? Is there a trace for each run that we can export?

For reference frameworks, ATLAS is the most practical for red teams. NIST’s AI RMF 1.0 and its Generative AI Profile (AI 600-1) suit governance mapping, but NIST says the RMF is being revised and gives no timeline. Banks and insurers have sector rules on top; see AI agents in financial services.

Frequently asked questions

Is prompt injection solved?

No. Adaptive attacks in “The Attacker Moves Second” bypassed 12 published defences, most at above 90% success. Assume some attempts will succeed and limit what a compromised run can do.

What is the current OWASP number for excessive agency?

LLM03 in the OWASP GenAI LLM Top 10 2026. It was LLM06 in the 2025 edition, and older documents still use that number.

Was EchoLeak a breach?

Not on the public record. It was a vulnerability, CVE-2025-32711, with CVSS 9.3. It was fixed server-side in May 2025, and Microsoft reported no evidence of exploitation.

Are guardrails enough to secure an AI agent?

Classifier and prompt-based guardrails are not. They reduce noise and catch mistakes. Damage is limited by things the model cannot override: tool scope, credentials in the target system, egress rules and argument checks in code.

Does an agent need its own identity?

Yes, one per agent and environment, with user-delegated access where it acts for a person and short-lived tokens the model never sees. NIST’s initiative page lists no final agent guidance as of November 2026.

Getting an agent through security review

AI Superior is an AI engineering company in Darmstadt, Germany, founded in 2019 by AI scientists. We build LLM applications, RAG systems and AI agents for B2B clients. If you need an agent designed with these limits from the start, see our AI agent development services or, for several agents across an organisation, enterprise AI agent development.

Sources and further reading

• OWASP, OWASP GenAI LLM Top 10 2026, August 2026.
• OWASP, OWASP Top 10 for Agentic Applications for 2026, announcement of 9 December 2025.
• OWASP, State of Agentic AI Security and Governance 2.01, 1 June 2026; A Practical Guide for Secure MCP Server Development, 16 February 2026.
• Help Net Security, report on OWASP’s findings on prompt injection and AI security failures, 11 June 2026.
• MITRE, ATLAS changelog, content version 2026.10, 28 October 2026.
• NIST, AI Risk Management Framework 1.0, 2023; Generative AI Profile (AI 600-1), 2024.
• NIST CAISI, AI Agent Standards Initiative, announced 17 February 2026, page updated 14 August 2026.
• CVE records: CVE-2025-32711, CVE-2025-53773, CVE-2026-21520.
• Reddy and Gujral, analysis of EchoLeak, arXiv 2509.10540.
• BleepingComputer, report on the zero-click data leak flaw in Microsoft 365 Copilot, 11 June 2025.
• Nasr, Carlini et al., “The Attacker Moves Second”, arXiv 2510.09023, October 2025.
• Brunner, Liu and Pande (Google), research on prompt injections on the web, 23 April 2026.
• Gizmodo, report on the Replit agent incident, 23 July 2025.
• Hugging Face, security incident report, 16 July 2026.
• METR, investigation of the OpenAI and Hugging Face incident, 26 August 2026.
• Anthropic, investigation of incidents in cybersecurity evaluations, 30 July 2026; report on disrupting AI-run espionage, 13 November 2025; Grace, Hadfield, Olivares and De Jonghe, “Demystifying evals for AI agents”, 9 January 2026.
• Google Threat Intelligence Group, report on data theft from Salesforce instances via Salesloft Drift, 2025.
• OpenAI, A practical guide to building agents, 2025.
• Microsoft, Agent Framework overview, updated October 2026.
• LangChain, LangGraph memory documentation.
• OpenTelemetry, semantic conventions for generative AI.
• Model Context Protocol, specification changelog for revision 2026-07-28.