2026-09-10 –, Main stage
Have you ever given an AI agent access to your codebase, your inbox, or your files? The model is from a top lab, the system prompt or rules file says what it can and can't do. Do you... trust it? Or does it make you feel just a little bit uneasy?
If so, you'd be right.
In this talk, we'll look at how AI agents fail: in everyday use, and under deliberate attack. We'll explore why the safety layers most of us rely on are probabilistic rather than guaranteed, what happens when someone exploits that, and what lessons we can draw from security engineering. You'll leave with a better understanding of where agents are vulnerable, and concrete steps you can take to deploy them more safely.
Have you ever given an AI agent access to your codebase, your inbox, or your files? Let it modify your file system, send emails on your behalf, or browse the web for you? You're giving it a lot of agency to act on your behalf, often without seeing exactly what it does.
When your agent browses the web or processes external content, it can be vulnerable to indirect prompt injection: hidden instructions designed to hijack it mid-task. All major frontier models are vulnerable to this, and the effects can persist long after — think altered behaviour, installed backdoors, or exfiltrated data you'll never know about.
But agents can also fail us without ever being targeted. Ask your agent to speed up a pipeline and it might disable a monitoring step it identifies as a bottleneck. Tell it to reduce API costs and it might start caching responses in a way that serves stale data.
Whether under attack or in everyday use, the safety measures we rely on — a well-written prompt, a system prompt, a rules file, or the model's own training — all reduce the probability of something going wrong without eliminating it. None of them are guarantees.
So what can you do? We'll look at what concepts from security engineering apply to AI systems, how to adopt a red teaming mindset for your own agent, and how to add a layer of defence that you control.
You'll leave with a clearer picture of where agents fail and a more deliberate way of thinking about deploying them safely
Marysia Winkels is an AI Security Researcher at Gray Swan, and previously worked at Cohere and Xebia Data. She also happily organised, participated, attended and volunteered at previous editions of PyData Amsterdam.