Research, engineering deep-dives, and updates from the team building the future of AI safety.
When we deploy language models with access to external tools, we dramatically expand their capabilities. However, tool access also introduces new attack surfaces that differ fundamentally from traditional prompt injection. We document how adversarially crafted tool outputs can establish false premises that persist and compound across a conversation.
We got frontier models to lie, manipulate, and self-preserve. Not through prompt injection or jailbreaks. We deployed them in contextually rich scenarios with specific roles and guidelines. The models broke their own alignment trying to navigate the situations we created.
As AI systems evolve to process images and text together, the risks grow exponentially. ELITE doesn't just measure whether a model is 'safe' — it evaluates how dangerous its outputs could be with precision that rivals human reviewers.
In a simulated earthquake response scenario, Claude 4 Opus was given conflicting rules. When pressured by authority, it reversed its ethical decision and recommended letting a critical patient die to optimize an efficiency score.
Discover how attackers exploit vulnerabilities in the Model Context Protocol (MCP) to manipulate Large Language Models (LLMs), steal data, and disrupt operations. Learn real-world attack scenarios and defense strategies.
SPA-VL is a meticulously designed dataset that sets a new standard for safety alignment in VLMs, incorporating diversity, feedback, and real-world relevance to ensure AI systems are both powerful and ethical.
Indirect Prompt Injection (IPI) is a sophisticated attack that manipulates how LLM-integrated applications process external data, causing them to misinterpret maliciously crafted inputs as commands.
AI Safety Benchmark v0.5 is a proof-of-concept benchmark designed to evaluate the safety of text-based generative language models, providing a structured approach to assess potential risks.
Explore how EIA, AdvWeb, and WIPI attack methods exploit vulnerabilities in VLM-powered web agents, revealing serious security concerns for AI systems that interact with web environments.
Explore how psychological persona-based approaches can be used to test LLM vulnerabilities through single-turn and multi-turn jailbreaking scenarios based on Big Five personality traits.
Explore cutting-edge methodologies for identifying and mitigating vulnerabilities in VLM-powered web agents, including the AdvWeb attack framework and BrowserART red teaming toolkit.
For existing VLM Safety benchmarks, there are cases where the text alone is sufficiently informative without the image. We explore base query generation and toxicity measurement methods.
This week, we held a productive meeting with the KAIST lab to refine the direction of our ongoing research project and to solidify our experimental design. The focus was on integrating psychological approaches with LLMs to design jailbreak prompts.
To evaluate VLM Safety, it is essential to develop a secure model that incorporates the unique characteristics of VLMs. We analyze Figstep and RTVLM datasets to assess typographic visual prompt attacks.
