The Evolution of Prompt Injection Attacks
Early prompt injection attacks were relatively simple—malicious instructions embedded in external content that models would follow without question. As language models have become more capable, attackers have adapted their tactics, shifting from direct instruction injection to sophisticated social engineering techniques that manipulate agents through context and persuasion rather than raw command overrides.
Social Engineering Model for AI Security
OpenAI frames AI agent security through the lens of social engineering risk management, drawing parallels to human customer service environments. The key insight is that perfect detection of malicious inputs is unrealistic; instead, systems should be designed so that the impact of manipulation is constrained even when some attacks succeed.
This three-actor model acknowledges that:
- AI agents need capabilities to be useful (accessing tools, retrieving information, taking actions)
- Agents are exposed to untrusted external content that may attempt to mislead them
- Systems require defensive boundaries to limit downside risk
Technical Defense Mechanisms
OpenAI combines social engineering principles with traditional security engineering approaches:
Source-Sink Analysis: Attackers need both a source (way to influence the system) and a sink (dangerous capability). For agentic systems, this typically means combining untrusted external content with sensitive actions like transmitting information to third parties or following links.
Safe Url Mitigation: When the assistant is convinced to transmit sensitive information learned in a conversation to external parties, OpenAI's "Safe Url" mechanism detects this behavior. The system either asks the user to confirm before transmission or blocks the action and suggests alternative approaches.
Safety Training: ChatGPT's safety training causes the agent to refuse most attacks before they reach dangerous actions.
User-Facing Security Expectation
The core principle: potentially dangerous actions or transmission of sensitive information should never happen silently or without appropriate user safeguards. This design philosophy applies across ChatGPT and other OpenAI products like Atlas, protecting navigation, bookmarks, searches, and information retrieval operations.