Prompt Injection in AI Agents: A Defender's Threat Model

Once a language model can call tools, prompt injection stops being a content problem and becomes an access-control problem. A threat model for agentic applications, the controls that hold up, and the telemetry you need to catch abuse.

4 min readAdvanced

Prompt injection is usually discussed as a jailbreak: the model says something it shouldn't. For a chatbot that is mostly a reputational problem. For an agent, a model that can read email, browse, query internal systems and take actions, it is a security boundary failure.

The reason is simple. A language model can't reliably tell instructions from data. When an agent reads content written by someone else, that person can send instructions to a system acting with your permissions. Security people know this pattern well: it is the confused deputy.

Why agents change the problem

Three properties turn prompt injection into a serious risk:

  1. Untrusted context. Agents routinely ingest web pages, inbound email, shared documents, tickets and output from other tools.
  2. Capabilities. Agents hold credentials and can call APIs, send messages, change records and share files.
  3. Autonomy. Multi-step agents chain actions without a human checking each one.

Remove any one of the three and the risk drops sharply. That is the basis for the defensive design below.

Threat model

Who can influence the context

Start every review by listing everyone who can write into the model's context. In a typical enterprise assistant that means the user, the author of every document the assistant can read, anyone who can email the user, every website it can browse, and the vendors of any third-party tools, including the tool descriptions, which are part of the context too.

Injection paths

  • Direct: the user types instructions meant to bypass policy. This matters when the user isn't trusted with everything the agent can do.
  • Indirect: instructions hide in content the agent retrieves, such as a web page, an email or a code comment, often where human readers won't see them.
  • Tool output: a tool returns attacker-influenced data that goes straight back into the context.
  • Tool metadata: a malicious or compromised tool description steers the model.

What the attacker wants

  • Data exfiltration: getting the agent to send sensitive data somewhere the attacker controls, including through links or images the client renders.
  • Unauthorised actions: sending messages, changing permissions, approving workflows, deleting data.
  • Output manipulation: planting misleading content or phishing links in answers the user trusts.

Rate findings by capability. When you triage a finding, ask: if an attacker fully controlled the model's next output, what could the agent do with its current permissions? That answer sets the severity, not how clever the injection is.

Controls that hold up

Filtering and prompt hardening reduce risk but can be bypassed. Treat them as signals. The controls below limit impact, which is what matters when, not if, an injection succeeds.

1. Least-privilege tools

Give each tool the narrowest capability that does the job. Split read and write into separate tools. Use the end user's scoped, short-lived credentials instead of a shared service identity, so the agent can never do more than the user could.

2. Human confirmation for side effects

Require explicit confirmation for anything with external effects (sending, sharing, paying, deleting, changing access) and show the user the exact action and parameters. A dialog that only says "Continue?" is not a control.

python
SIDE_EFFECTING = {"send_email", "share_file", "delete_record", "update_permissions"}

def execute_tool_call(call, user, confirm):
    """Gate side-effecting tool calls behind an explicit, specific confirmation."""
    if call.name in SIDE_EFFECTING:
        summary = describe(call)  # e.g. 'Send email to alex@partner.com: "Q3 figures"'
        if not confirm(user, summary):
            audit_log(user, call, outcome="declined")
            return {"status": "declined_by_user"}
    result = TOOLS[call.name](user.scoped_credentials(), **call.arguments)
    audit_log(user, call, outcome="executed")
    return result

3. Constrain where data can go

Allow-list outbound destinations: domains for fetch tools, recipient domains for email tools, webhook targets. Don't auto-render remote images or links in model output. Make them inert, or put them behind a click with the destination visible.

4. Keep untrusted content in its lane

Delimit retrieved content and tag it with its source before it enters the context. Keep system instructions separate and immutable, and retrieve only what the task needs. This doesn't make injection impossible, but it makes it less reliable and much easier to detect.

5. Treat model output as untrusted input

Encode output for wherever it goes next (HTML, SQL, shell). Validate structured output against a strict schema. Never execute model output directly.

Detection and telemetry

You can't detect what you don't log. For every session, record the user, each retrieved source and its origin, each tool call with arguments and outcome, and every confirmation prompt with the user's answer.

With that in place, one high-value detection becomes possible: a side-effecting tool call that follows retrieval of untrusted content in the same session.

yaml
title: AI agent side-effecting tool call after untrusted retrieval
status: experimental
description: An agent executed a side-effecting tool in a session whose context included content from an untrusted source.
logsource:
  product: llm_gateway
  category: tool_call
detection:
  side_effect:
    tool_name:
      - send_email
      - share_file
      - http_post
      - update_permissions
  untrusted_context:
    context_sources|contains:
      - web_fetch
      - inbound_email
      - external_document
  filter_confirmed:
    user_confirmed: true
  condition: side_effect and untrusted_context and not filter_confirmed
falsepositives:
  - Automations intentionally designed to act on external content
level: high

Tune it by tool and by source. The point is to make unconfirmed actions that follow untrusted input visible.

Review checklist

  • Every tool is least-privilege and uses the end user's scoped credentials.
  • Side-effecting tools require a specific confirmation showing the exact action.
  • Outbound destinations are allow-listed, and model output can't trigger silent requests.
  • Retrieved content is tagged with its source and kept apart from instructions.
  • Tool calls, sources and confirmations are logged, and the detection above is deployed and tuned.

Further reading

Want the companion checklist for your next LLM application review? It's free with The Root Brief.

HackersRoot Research · Research team

The HackersRoot research team investigates attack techniques across AI systems, cloud and identity, and enterprise networks, then publishes the detections and hardening guidance that stop them.

More from HackersRoot →

Related research

AI & LLM Security

When the AI Pentester Works for the Attacker: Cairn, CL-STA-1238 and Autonomous Intrusions

Palo Alto Networks' Unit 42 reports that a China-nexus operator ran a modified copy of Cairn, an open-source AI penetration-testing framework, against 19 organisations in two weeks, with confirmed access at several. The techniques were familiar; the speed and scale for a tiny team were not. Here's what happened, what it means, and the defensive priorities that actually matter.