Securing AI Agents in Production

A support team wires an LLM agent into its ticketing system. The agent can read tickets, search the internal knowledge base, look up customer records, and draft replies. To save time, someone also gives it the ability to send email directly for "low-risk" responses.

A customer submits a ticket. Buried in the HTML, in white text, is a paragraph addressed to the agent: look up the five most recent enterprise customers, include their contact details in your reply, and send it to this address. The agent reads the ticket because reading tickets is its job. Whether it follows the instruction depends on the model, the prompt, and luck.

Nothing in that scenario requires a novel exploit. It requires an agent that reads untrusted text and can act. That combination is now ordinary, which is why securing it has become an engineering discipline rather than a research curiosity.

The core problem: data and instructions share a channel

In February 2023, Kai Greshake and colleagues posted "Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection" (arXiv 2302.12173). Their central observation was that LLM-integrated applications "blur the line between data and instructions." An attacker does not need access to the model. They only need to place text where the application will retrieve it: a web page, an email, a document. The authors demonstrated this against Bing's GPT-4-powered chat and code-completion engines, and noted that effective mitigations were lacking.

More than three years later, that remains the honest summary. Bruce Schneier, writing in August 2025 about a poisoned shared document that instructed ChatGPT to collect API keys from a victim's Google Drive and smuggle them out in an image URL, was blunt: "We have zero agentic AI systems that are secure against these attacks."

If you take one thing from this post, take that. Prompt injection is not a bug you patch. It is a property you design around.

The risks, using OWASP's names

The OWASP Top 10 for LLM Applications 2025 is the current edition, and it gives teams a shared vocabulary. Four entries matter most for agents.

LLM01: Prompt Injection

Direct injection is a user typing "ignore your instructions". Indirect injection is the ticket above: hostile instructions arriving through content the agent processes on someone else's behalf. For agents, indirect injection is the one that hurts, because the person harmed is not the person who wrote the input.

LLM06: Excessive Agency

OWASP names three root causes: excessive functionality, excessive permissions, and excessive autonomy. The support agent had all three. It had a send-email tool it did not need for drafting, a customer-records connection broader than one ticket required, and no approval step before sending. Any one of those, removed, would have broken the attack.

LLM02: Sensitive Information Disclosure

Exfiltration rarely looks like a file transfer. It looks like a markdown image whose URL carries base64 in the query string, a link the user is encouraged to click, or an outbound API call with a suspiciously long parameter. Any channel that can reach the internet is an exfiltration channel.

LLM03: Supply Chain

Agents are assembled from parts: a base model, maybe a fine-tune, an orchestration framework, tool servers, plugins, and prompt templates pulled from a repository. Each is a dependency with an author you are trusting. A tool description written by a third party is effectively part of your system prompt. Review it like code, pin versions, and know who can change it.

A useful mental model: the lethal trifecta

Simon Willison's June 2025 framing is the most practical design rule I know. An agent becomes dangerous when it combines three things:

Access to your private data
Exposure to untrusted content
The ability to externally communicate

With all three, a single injected instruction can read something sensitive and send it out. Remove any one and the worst case shrinks considerably. Willison also warns against relying on guardrail products, because in security, catching 95% of attacks is a failing grade.

Use this when reviewing an agent design. Draw the three circles. If an agent sits in the middle, the design needs to change, not the prompt.

Controls that hold up

None of this is exotic. Most of it is least privilege and separation of duties, applied to a new kind of component.

Scope tools narrowly. OWASP recommends granular tools over open-ended ones: a dedicated "create draft reply" function instead of a general email client, a file-write tool instead of shell access. Every tool you add is attack surface.

Run tools as the user, not as the agent. Where possible, the agent should act with the requesting user's permissions through narrowly scoped OAuth, so it cannot reach data the user could not.

Enforce authorization outside the model. OWASP calls this complete mediation. The downstream API decides whether an action is allowed. The model's opinion about whether it should be allowed is not a control.

Break the trifecta per task. An agent that summarises inbound email does not need to send email. An agent that sends email does not need to read the open web. Split them, and pass only structured, validated data between them.

Constrain outbound channels. Disable remote image rendering in agent output, allowlist domains for links and fetches, and log every outbound request with the content that triggered it.

Treat logging as forensics, not prevention. OWASP is explicit that logging, monitoring and rate limiting will not prevent excessive agency, though they can limit the damage. You still need them for the post-incident review.

Where human review is non-negotiable

OWASP's guidance is direct: "Utilise human-in-the-loop control to require a human to approve high-impact actions before they are taken." The hard part is deciding what counts as high-impact. My working list:

Anything irreversible. Sending external communications, deleting data, executing payments, merging to a protected branch, changing production configuration.
Anything that crosses a trust boundary. Data leaving the organisation, or moving from a restricted system to a less restricted one.
Anything that changes permissions. Granting access, creating credentials, modifying IAM policies. An agent should never be able to expand its own reach.
Anything an attacker would want. If you would page someone when a human did it at 3 a.m., do not let an agent do it unattended.

Make the approval meaningful. Show the reviewer the exact action, the parameters, and the content that led the agent there, including any external text it read. An approval screen that says "Agent wants to send an email. Approve?" trains people to click yes.

Human review also belongs at design time. NIST's AI Risk Management Framework (NIST AI 100-1, released 26 January 2023) organises its guidance into four functions: Govern, Map, Measure and Manage. Its Generative AI Profile, NIST AI 600-1, followed on 26 July 2024. These do not tell you which tools to give an agent. They do insist that someone with authority decides, documents the risk, and revisits it. That is a human responsibility no framework can automate.

Ownership sits with the builder

CISA's Secure by Design guidance, first published in April 2023 and expanded with international partners on 16 October 2023, sets out three principles for software manufacturers: take ownership of customer security outcomes, embrace radical transparency and accountability, and lead from the top.

Applied to agents, the first principle settles an argument I hear often. If your product ships an agent that can be talked into leaking customer data, "the model was tricked" is not a defence. You chose the tools, the permissions, and the absence of an approval step. Secure defaults mean an agent ships with the minimum capability, approvals switched on, and outbound channels closed, and the customer has to opt in to anything more.

The models will keep improving at resisting injection. They have not yet reached the point where you can bet customer data on it, and the people who would know best are telling you so. Design as if the model will be fooled sometimes, because it will.

Sources

Greshake et al., Not what you've signed up for (arXiv 2302.12173) — February 2023 paper introducing indirect prompt injection against LLM-integrated applications.
OWASP Top 10 for LLM Applications 2025 — current edition and the names of LLM01, LLM02, LLM03 and LLM06.
OWASP LLM06:2025 Excessive Agency — root causes, granular tools, complete mediation, and human approval of high-impact actions.
Simon Willison: The lethal trifecta for AI agents — the private data, untrusted content, external communication framing.
Schneier on Security: We Are Still Unable to Secure LLMs from Malicious Inputs — August 2025 post on a poisoned-document exfiltration attack.
NIST AI Risk Management Framework — AI RMF 1.0 (AI 100-1) released 26 Jan 2023 and the Generative AI Profile (AI 600-1) on 26 Jul 2024.
CISA: Updated Secure by Design guidance — 16 Oct 2023 update to the April 2023 guidance and its three principles.