This data persistence introduces additional data protection requirements and can add new risk vectors such as memory poisoning attacks where adversaries inject false information that corrupts decision-making across multiple interactions and users. This creates risks of unauthorized actions, runaway processes, and decisions that exceed intended boundaries when agents misinterpret objectives or operate on compromised instructions. Security evaluation often requires testing unsafe prompts or tool sequences; executing these against live systems is risky and irreproducible 210, 232, 239, 240, 241. Alongside insights based on current advances and progress, we discuss how new benchmarks can further augment evaluations by incorporating additional information or adopting relevant strategies. ST-WebAgentBench is an online, enterprise-focused benchmark for testing whether web agents avoid unsafe actions (e.g., destructive operations in business systems) while pursuing goals . On the long-standing MiniWoB/MiniWoB++ benchmarks, proposed methods have continued to make progress 224, 225, 226.
For instance, in the WebArena benchmark, the authors provide Unachievable (UA) Hints in the agent prompts for tasks that are impossible to achieve given the lack of evidence. Each scope introduces new capabilities—and corresponding security requirements—that organizations must prioritize when addressing agentic AI risk. This self-direction introduces risks of uncontrolled operations, explainability, and auditability, and makes it difficult to maintain predictable security boundaries.
Common approaches include adversarial training of the underlying LLM 176, 179, 194, or the use of dedicated detection https://influencemarketingnews.com/privacy-laws-and-influencer-marketing/ models that flag injected inputs before execution 187, 195. Prompt augmentation offers a lightweight and easily deployable defense against prompt injection, relying on carefully crafted system prompts or input modifications rather than model retraining or architectural changes. By checking for invalid or unexpected responses, such as forced tokens (e.g., “HACKED”), these approaches can identify compromised generations 97, 189. These approaches fine-tune smaller LLMs to distinguish legitimate instructions from injected ones 187, 188. Other system-focused defenses, such as Melon, employ constrained execution sandboxes and verification loops to limit the impact of potentially malicious instructions on downstream systems, serving as a defense against IPI attacks .
1.2 Intentional and Non-Intentional Prompt Injection
For instance, GPT-4 often reissues an already entered search phrase (“DMV area”) until it reaches the step limit, demonstrating a failure to incorporate short-term state and past actions into decision-making. These risks are not rooted in the agent’s internal reasoning or learning capabilities, but rather in the mismatch, fragility, or variability of https://madeintexas.net/accounting-services-in-poland.html the interfaces and environments through which the agent perceives and acts 162, 163, 164. While this capability improves system adaptability and efficiency, it also weakens clear trust boundaries.
Communication and coordination
It marks a move from models that respond to prompts to systems that operate as autonomous workers. That capability is now showing up in real products because organizations want AI that can complete multi-step work instead of producing one output at a time. Agentic AI refers to systems that can plan tasks, make decisions, and take actions without continuous human direction. Agentic AI security is the protection of AI agents that can plan, act, and make decisions autonomously.
This lets organizations preserve productivity without giving agents unrestricted autonomy. Security teams should consider message integrity, agent identity, data provenance, delegation rules, cross-agent permissions, cascade detection, and blast-radius controls. The attacker doesn’t necessarily need to compromise every agent in a chain — they may only need to influence one, and let trust relationships do the rest. Traditional penetration testing focuses on applications, networks, APIs, endpoints, and credentials. Static configuration isn’t enough — agent behavior can change because its context changes. For high-risk tools — deleting 20,000 customer records, for instance — an explicit approval gate belongs between the agent’s intent and execution.
Meanwhile, ToolEmu evaluates agents within an LLM-emulated tool sandbox, enabling scalable probing of risky behaviors and potential negative side-effects . Compared to pure web benchmarks, OS-Harm stresses desktop-level side-effects (e.g., unintentional data exfiltration or copyright infringement edits) and probes how trace format (screenshots and accessibility trees) affects automatic judging reliability . OS-Harm builds on OSWorld’s full desktop environment to evaluate agent safety across office applications and file operations, scoring both accuracy and adherence to safety guidelines via LLM-based judges with validated agreement levels to human annotations . Several benchmarks now also aim to evaluate agentic safety, i.e., risks arising from autonomous action, tool use, and long-horizon interaction, rather than from static chat completion. While these capability-oriented benchmarks are not framed as security tests per se, they expose control and reliability deficits that strongly interact with overall model safety. Table 1 summarizes existing benchmarks spanning both capability and security-specific focuses, highlighting their domains, safety focus, and evaluation methodologies.
- Isolation defense methods restrict the possible impact of harmful instructions by limiting an agent’s capabilities while engaging with untrusted input .
- Creating additional friction, CAPTCHA and other robot-detection mechanisms often create significant issues for agentic systems.
- ST-WebAgentBench is an online, enterprise-focused benchmark for testing whether web agents avoid unsafe actions (e.g., destructive operations in business systems) while pursuing goals .
- Known-answer detection introduces control questions to identify compromised executions, although this method can only be applied post-hoc.
- Static configuration isn’t enough — agent behavior can change because its context changes.
Taxonomy of Security Threats
Adversaries can leverage agents’ black box nature to human eyes, training methods, and interpretations of their inner logic, to conceal and inject instructions within images, sounds, or videos . Beyond traditional text-based prompt injection, AI agents are increasingly vulnerable to attacks that exploit the growing capabilities of modern models, including code generation/execution, and multimodal understanding 79, 80, 61. On the other hand, harmful instructions can also be created by adversaries, specifically with the objective of manipulating an LLM agent in intentional prompt injection. For example, unclear or badly worded user inquiries may unintentionally overrule system directives or result in dangerous actions. They also demonstrate how strategies like CSS obfuscation and hidden HTML elements can further improve stealth, making attacks invisible to users.
When AI agents are given instructions or permissions to act based on the data, parameters, instructions, and responses given to them, the boundaries of independence or autonomy they are permitted to act within are important to define. Unlike traditional FMs that operate in stateless request-response patterns, agentic AI systems introduce autonomous capabilities, persistent memory, tool orchestration, identity and agency challenges, and external system integration, expanding the risks that organizations must address. The Agentic Security Research Initiative explores the emerging security implications of agentic systems, particularly those utilizing advanced frameworks (e.g., LangGraph, AutoGPT, CrewAI) and novel capabilities like Llama 3’s agentic features. Tools are often the highest-risk surface in agentic systems because tools turn decisions into actions. As discussed, traditional LLM risks focus on prompts, data exposure, and output handling.
