2025-04-12 –, Track 2
It’s coming, and you aren’t ready. Your company’s virtual agent is sending inappropriate messages and handing out customer PII to anyone that asks nicely. And who are they going to call? You. This talk explores the investigation and response process for handling the unique threats to GenAI chatbots.
It’s coming, and you aren’t ready—your first generative AI chatbot incident. GenAI chatbots, leveraging LLMs, are revolutionizing customer engagement by providing real-time, automated 24/7 chat support. But when your company’s virtual agent starts responding inappropriately to requests and handing out customer PII to anyone that asks nicely, who are they going to call? You.
You’ve seen the cool prompt injection attack demos and may even be vaguely aware of preventions like LLM guardrails; but are you ready to investigate and respond when those preventions inevitably fail? Would you even know where to start? It’s time to connect traditional investigation and response procedures with the exciting new world of GenAI chatbots.
In this talk, you’ll learn how to investigate and respond to the unique threats targeting these systems. You’ll discover new methods for isolating attacks, gathering information, and getting to the root cause of an incident using AI defense tooling and LLM guardrails. You’ll come away from this talk with a playbook for investigating and responding to this new class of GenAI incidents and the preparation steps you’ll need to take before your company’s chatbot responses start going viral—for the wrong reasons.
Allyn Stott is a senior staff engineer at Airbnb where he works on the InfoSec Technology Leadership team. He spends most of his time working on enterprise security, threat detection, and incident response. Allyn has presented at Black Hat, Kernelcon, The Diana Initiative, Blue Team Con, Swiss Cyber Storm, SecretCon, Texas Cyber Summit, and BSides around the world. In the late evenings, Allyn writes an exclusive security newsletter: meoward.co.