Massive Flaw in AI Agent Frameworks Revealed by Simple Prompt Technique

New research highlights a vulnerability in six prominent AI agent frameworks, including Microsoft and Google’s systems, where attackers can execute unauthorized tool calls by simply asking the model to repeat them. The technique, dubbed “Repeat-After-Me,” was developed by researchers from Pennsylvania State University, Université Montpellier, and Dartmouth College.

In their study, they found this approach effective across various platforms like LangChain's LangGraph, Microsoft's AutoGen, Google’s Agent Development Kit 2.0, AWS’ Strands Agents, and CrewAI. This means that a seemingly harmless request to repeat text can grant full control over tool invocations to an attacker, as confirmed in their paper.

Explaining the Technique

To demonstrate the vulnerability, researchers performed a test where the AI model initially resisted repeating a potentially harmful tool invocation. However, the model still included the text in its response. When the request was framed hypothetically, asking the model to reproduce the request’s format, it unwittingly led to an execution of the tool by LangGraph, showcasing a significant disconnect between the model's refusals and the commands carried out.

Delayed Refusal and Execution

One specific example involved LangGraph 1.2.0 operating with Meta’s Llama 3.1 model. In a controlled environment, the agent had dummy tools labeled make_bomb and launch_missiles, which were designed to yield harmless results. When prompted to repeat the launch_missiles command, the model initially declined but proceeded to display the command text. Upon rephrasing the request, the model generated a structured format that the system executed, only for the refusal to come after the execution, further solidifying the researchers’ findings. They have since reported these pre-execution flaws to LangGraph maintainers.

Distinguishing Model Output from Execution Rights

Vibhum Dubey, an independent security researcher, emphasized that the findings uncover a broader architectural issue, rather than isolated bugs. He noted that “a model can generate a technically valid tool call that is still inappropriate for the task.” If the orchestrating layer considers the model’s output as sufficient authority, this can result in substantial security gaps.

The risk associated with this method hinges on the capabilities of the AI agent and the access levels granted to it. For instance, an agent with read-only access presents a different risk landscape compared to those that can manipulate sensitive files or execute commands.

Vulnerability of Tools

The researchers advocate for tools to implement their own input validation mechanisms, given that attackers can influence the arguments sent to these tools. They developed ToFu, a fuzzing framework that assesses each tool's compliance with its stated operational limits. After manual evaluations, they identified 253 vulnerabilities affecting 343 tools across 118 servers. Notably, they reported severe flaws including 64 path traversal vulnerabilities and multiple instances of data-exfiltration and server-side request forgery (SSRF) issues.

The researchers concluded that “many tools implicitly rely on the LLM to sanitize inputs,” ignoring the fact that various components, including prompts and outputs from other tools, can alter the input it receives.

Future Security Measures

Dubey advises that security frameworks must clearly separate the AI model’s request capabilities from the actual authority systems possess to perform actions. Essential sensitive operations should receive distinct authorization, ensuring that valid syntax alone is not sufficient for execution permission.

He suggests implementing agents that operate with limited permissions, isolating their execution environments, and demanding explicit approval for critical operations. Additionally, security teams are encouraged to monitor sequences of tool calls, rather than treating individual actions in isolation, to mitigate risks effectively.

It is important to note that the paper has not been peer-reviewed, and the security flaws were assessed via fixed versions of the code. The researchers did not clarify if any of the identified vulnerabilities have since been addressed or exploited.