Brandon Sherman đź‘ľ
Brandon Sherman is a walking, talking, and occasionally thinking bag of meat and water. This is more than what can be said of any LLM at present. When his boss is looking, he does Cloud Security at Temporal Technologies, Inc. where the goal is to build software as reliable as gravity. He has previously presented on topics such as Cloud Forensics, The Cloud Changes Underneath Us, and
Console Hero to IAM Zero. Hopefully, today he hasn't managed to be too dumb.
Session
Your threat model hasn’t changed with AI. It seems we have learned nothing in the 12 years since Codespaces was wiped out. It was always possible for a human to write some truly unhinged scripts, calling APIs faster than a human could click, which could ruin your day (Or week, or month, or longer). Now, AI Agents are creating some truly unhinged scripts which have caused outages at multiple large organizations. AWS has suffered an outage when an AI coding tool decided the right course of action was to delete prod.
Was the AI wrong? Is deleting prod actually a bad thing? (I argue: No.)
Regardless, the purported solution is to have a “human in the loop” — essentially, a human whose job it is to review the actions being taken or proposed and vet them for safety. This is not inherently a bad solution, but it comes with a multitude of risks that need to be considered. Chief among them, we are all by our very nature as humans, prone to doing very stupid things. I once lost someone's birthday card just before leaving for their party. I found it, two days later, in the refrigerator. It is one of the defining characteristics of the human condition: Even the smartest individuals among us are capable of being stupid. Pair this with AI, and you have a recipe for disaster.
In this talk, I will cover the risks and how to think about them, leveraging learnings from another industry which uses complicated computer systems: aviation. I’ll discuss how to avoid becoming “Children of the Magenta Line”, a phrase from a classic 1997 training session by American Airline’s chief pilot, given two years after the AA965 crash where a flight crew dutifully followed the magenta line on their navigation screens into a mountain. In the security industry, we are in danger of repeating the same human-computer interface mistakes that have been identified and mitigation strategies developed in other automation-heavy industries. Thankfully, in our industry, our mistakes tend to have less immediately tragic consequences; but that doesn’t mean we should be complacent and make them anyway.