Security Bsides Las Vegas 2024

Building a Security Audit Logging System on a Shoestring Budget
2024-08-07 , Firenze

Working cybersecurity can be a tough gig, especially if you’re budget constrained and developers are adding services faster than the company adding employees. Knowing what’s happening in the system is the first step to securing it.

This talk demonstrates how to build a robust, security-focused audit logging system for a fast growth company on the thinnest of budget. Human cost in toil and time is also a serious consideration, which is optimized through hard learned lessons.

Audiences will appreciate both the outcome, and the lessons learned when software engineering and hacker culture collide. Plus, they will discover what becomes possible as your budget expands.


To defend against threats, one must know when they occur. A good security auditing system is fundamental to effective cybersecurity, featuring robust access control and resilience to infrastructure outages and repudiation. It must be capable of ingesting events from multiple sources while providing a uniform search pattern. Many cloud providers have some offerings, but they are either cloud specific, have a hefty premium, or limited in functionality and customization.

In a company operating across multiple clouds with a strong vendor lock aversion, my security team faced the challenge of building a system that meets all of the aforementioned requirements, but with a limited budget allocated solely for running cloud infrastructure. Under these constraints, we devised a system to meet the above requirements using mostly opensource components. As maintenance of distributed systems carry a hefty human toll, we also focused on making a system that required as little human intervention as possible.

This talk will explore the alternatives we considered, the rationale behind our choice, and a deep dive into our implementation strategy – employing open-source software on Kubernetes that runs across multiple cloud providers and physical regions.

Key topics include:
- Implementing streamlined log access control with SSO and RBAC for minimal human intervention.
- Scaling strategies for peak ingestion, and sustaining search speeds.
- Techniques for reducing costs while maintaining data availability.
- Key failures in our system and how we improved as a result
- How a software engineer learns to build faster by watching hackers work

The talk aims to provide an informative session and a candid discussion of available options, what we built, and how our solution fares against offerings in the market. The goal is for the audience to leave feeling fortunate that their company has invested in a robust auditing system, or inspired to create one if needed.

George is a software engineer by career and training. He has worked at a biotech creating genetically modified crops, a video games studio, and a large cloud provider company. In the past few years, George was thrust into the world of cybersecurity by proximity to other folks doing it far longer than he has.

Despite having built many things used by many people, George’s claim to fame is having worked with a founding member of WhatsApp, who sadly turned down a sizable stake in the company for higher base pay. An event that gave him unrealistic start up expectations due to sampling bias.

Nowadays, George enjoys broadening his horizon from interactions with eclectic engineers. He currently works at CloudKitchens.