2026-05-07 –, IFEN room 2, Workshops and AI Security Village (Building D)
Every LLM has flaws. It’s been proven that the guardrails on every LLM can be bypassed. When you’re thinking about which ones to build your applications on, what are the key risks you need to be aware of?
In this talk, we will dive into our testing methodology for scanning the most popular LLMs for vulnerabilities where we generated hundreds of thousands of prompts across categories including prompt injection, malware, offensive language, and much more.
We’ll share our LLM risk matrix, and explain the best practices around minimizing the risk of hallucinations, malicious content, indirect prompt injection, and more as you build your LLM-powered applications.
Every LLM has risks, from malicious content generation to jailbreak, injection, misinformation and more. In this session, we'll discuss the approach that we used for categorizing the risk levels of the most popular LLMs that are available for application developers on the leading cloud platforms. We'll explain:
What tools we used to do this testing
How we use those tools
What categories of problems we're able to identify
How we turn the problems into understandable risk for developers and security practitioners to use for making decisions on which LLMs to adopt
Jeremy is the founder and CEO of FireTail, an end-to-end AI security platform. Prior to FireTail, Jeremy worked in M&A at Rapid7, a global cyber leader, where he worked on the acquisitions of 3 companies during the pandemic. Jeremy previously led sales at DivvyCloud, one of the earliest cloud security posture management companies, and also led AWS sales in southeast Asia. Jeremy started his career with 13 years in cyber and IT operations. Jeremy has an MBA from Mason, a BA in computational linguistics from UNC, and has completed additional studies in Finland at Aalto University. Jeremy speaks 5 languages and has lived in 5 countries.