Speaker introduction

• I’m a defensive cybersecurity practitioner: Detection & Response Engineer and Threat Hunter.
  18+ years in IT + Security.
• I’ve worked in several roles as a defensive practitioner and on the vendor side doing threat research and building security solutions.
• I currently work at Google Cloud where I help security practitioners defend their organization from attack using Google Security Operations.
• I like to pay it forward by sharing knowledge and research with the community via blogging, presenting at conferences such as Black Hat and BSides, and contributing to MITRE ATT&CK.
• I’m also the creator of Dorothy, a tool to simulate attacker behavior in Okta environments.
• A collection of my contributions to the security community can be found here.
• When I’m not working, you can find me enjoying the outdoors in Colorado with my family.

Intended audience

• Anyone who is curious about what Detection Engineering is and wants to learn about how to turn intelligence/information on attacker behavior (not IOCs) into a detection.
• A practical example of turning a piece of threat intelligence into a real-world detection will be utilized throughout the presentation.
• Defensive security practitioners: Detection Engineers, SOC Analysts, etc.
• Maybe you manage vendor-supplied detection content (rules/signatures) in your security tools and want to learn how to develop your own Detection Rules to increase your organization’s coverage against threats.
• If you’re already an expert in Detection Engineering, you might not learn a ton, but you might enjoy this presentation anyway 😀
• The goal for this presentation isn’t to be a comprehensive reference on all things Detection Engineering. - The goal is to give people an idea of what a day in the life of a Detection Engineer is and how to take some intelligence and turn it into a real-world detection. Attendees will walk away with an understanding of techniques for developing custom detections to alert on attacker behavior.

Agenda overview

• What is Detection Engineering?
• Benefits of Detection Engineering
• Taking action based on intelligence on attacker behavior (TTPs)
• Analyzing and understanding an attack technique
• Identifying data sources for detection
• Simulating adversary behavior
• Developing a detection rule
• Improving the precision of detection rules
• Testing & validating detection rules
• Wrap up: Key takeaways, Links to useful resources, Q&A

What is Detection Engineering?

• Here is my understanding of Detection Engineering after doing this for many years.
• Detection Engineering is a specialization focused on the proactive identification or malicious activity or unauthorized behavior within systems and networks.
• It involves designing, building, and continuously refining processes and systems to detect security threats before a data breach or other incident can occur
• Detection Engineers use a combination of data analysis, threat intelligence, and an understanding of attack techniques to create effective detections.

What are the benefits of having a Detection Engineering capability?

• Reduced risk - Proactive detection helps organizations identify attacks early giving them a chance to limit their impact and reduce the risk of data breaches or system disruptions.
• Increased visibility - Detection Engineering takes threat intelligence and produces detections that increase an organization’s overall coverage against threats.
• Faster response times - Well-defined detection rules and processes enable rapid response to incidents, minimizing damage
• Complements prevention - Detection Engineering provides a safety net if preventative security measures fail or if it’s impossible to implement preventative measures.

Story time: Intelligence-driven Detection Engineering

• While working as a Detection & Response Engineer at another company, I received intelligence from a fellow practitioner working in the same industry vertical. We were speaking about an active threat group’s tactics, techniques, and procedures.
• My intelligence source told me that the threat group’s modus operandi was to compromise a developer or software engineer’s user account, create a GitHub personal access token under the account, and proceed to clone all of the GitHub repositories that the compromised user had access to.
• The attacker is using a tool to perform this data theft in an automated fashion.
• They’re using anonymization/proxy services to hide their real IP address and geolocation.
• Needless to say, I became curious about what logging, monitoring, and detection opportunities existed for my GitHub Enterprise environment.
• For many organizations, GitHub houses critical intellectual property and is a prime target for attackers seeking to steal valuable source code, disrupt software development operations, or carry out supply chain attacks. Security teams must proactively monitor their GitHub Enterprise environments and have the capability to detect and respond quickly to any suspicious activity before an incident occurs.
• During the remainder of this presentation, I will be demonstrating how to take this piece of intelligence and turn it into a real-world detection that alerts us if this behavior happens in our organization’s environment that we’re defending together.

Understanding the technique we want to detect

• Define the tactic, technique, and procedure in the format of MITRE ATT&CK.
• Explain the atomic behaviors that occur when the attacker executes this technique successfully.
• An abstract workflow-type diagram will be used to describe the atomic behaviors

Identifying data sources that can be used for Detection Engineering

• Review what logs are available to us as Detection Engineers.
• Explain GitHub nomenclature for people who are not familiar with it: GitHub Enterprise, GitHub Organization, GitHub Repository.
• Introduce the GitHub audit log and show what it looks like in the GitHub user interface (UI)
• Show a couple of example events
• Lucky for us, GitHub Enterprise provides an audit trail of enterprise, organization, and user activity!
• Logging the source IP addresses for GitHub events must be enabled. The default setting is disabled. For our detection use case, we want to know which IP address is being used when actions are being carried out in our GitHub environment

Ingesting GitHub audit logs in a centralized system i.e. a SIEM

• Why do we do this? It’s great that GitHub provides an audit log and some syntax that lets us search for log events in their UI.
• There is also a feature where GitHub can call a webhook when a specific event happens.
• A challenge here is that many platforms all have their own search language for logs.
• By ingesting these logs into our SIEM and normalizing them into a common schema, the security team can easily search, analyze, and detect behaviors in these audit log events from a central location, using a single language.
• Provide a brief explanation of how to configure GitHub Enterprise to stream its audit logs to a SIEM.

Simulating adversary behavior in a detection lab

• Demonstrate how to simulate the behavior we are trying to detect.
• At this point logs are flowing into our SIEM and being parsed and normalized.
• In a detection lab, execute the following behavior.
• Create a new personal access token that has access to a number of GitHub repositories in an organization.
• Clone over 5 GitHub repositories in quick succession.
• Developing a search query to match on the intended events.
• Develop search queries that match on the atomic events i.e. the GitHub personal access token being created and the code repositories being cloned.

Developing a detection rule (crawl, walk, run)

• At this point, we have our logs flowing into our SIEM and they’re being normalized into a common schema. We’ve simulated the attack technique that we want to detect. We have search queries that match on the intended behaviors. Now it’s time to develop the first version of our detection rule.
• Explain why we don’t want to detect and alert on atomic behaviors.
• The behaviors of creating a personal access token or cloning a GitHub repository are not suspicious on their own. If we detect and alert on these atomic behaviors, our SOC will drown in alerts and close them out as false positives after wasting time triaging them.
• We want to alert on a sequence of behaviors to develop a detection with higher efficacy and precision. - To do this, we need to write a detection rule that correlates a number of events and alerts us on a sequence of behaviors.

Correlating events

• We’re correlating the events for the personal access token being created and more than 5 GitHub repositories being cloned in a short period of time. The events are being grouped by the GitHub user ID (the actor who is performing this behavior).
• Having our detection rule alert us when a large number of GitHub repositories are cloned in a short period of time suggests that some automation is at play. The intelligence I received informed me of an open source tool that attackers like to use to steal the contents of their victim’s GitHub repositories.
• On the subject of using open source tools to simulate attacker behaviors, you need to be very careful not to just download any tool you find on GitHub and run it in your environment to simulate attacker behavior. Not all developers are trustworthy or write secure code and you want to avoid infecting yourself or your organization. That would be bad, we’re trying to do the opposite here today 😀

Testing our detection rule

• Simulate the attack technique again.
• Validate that an alert was generated for the behavior.

Tuning detection rules

• Walk through an example of how to apply an exception to a rule to filter false positives. Maybe we have a process running in our organization that backs up the contents of our GitHub repositories every night. -Show how to filter this behavior out based on user ID or personal access token ID and source IP address.
• Tuning rules is part of the care and feeding that goes into Detection Engineering. You’re never truly done. Environments drift, attack techniques are no longer relevant, systems get patched, entire networks and technologies come and go.

Enriching events to improve detection efficacy and precision

• Hypothetically, let’s say that this detection rule has some false positives. Maybe when software engineers get a new laptop, they create a new personal access token and clone all the code repositories they have access to.
• This behavior causes pesky false positives for our SOC to investigate. They’re already dealing with alert fatigue and we don’t want them to hate us as Detection Engineers for making things worse. We want to be nice to our SOC analysts, we like them and respect what they do to keep us safe.
• To improve the precision for our detection rule, let’s add some additional logic to it. We want the detection to fire an alert when the GitHub activity is coming from an IP address that is attributable to a proxy/anonymization/VPN service, and not a VPN concentrator’s IP address that our organization owns.
• To do this, we can use a service like spur.us. They provide high quality feeds of IP addresses that are part of proxy/anonymization services.
• Walk through the process of adding this logic to our detection.

Putting it all together: Testing and validating rules

• Explain the importance of testing. It’s crucial for a security team to be able to know that their logging, detection, and alerting capabilities are working as expected. Environments drift. Vendors change their logging schemas (field names and values).
• Triggering our detection rule from a VPN/anonymization service IP address.
• Validating that an alert was generated.
• Writing a test for a rule can sometimes take longer than writing the rule logic itself. But don’t skip this step! We can run this test on a regular basis (e.g. daily) to have confidence as defenders that our logging, detection, and alerting pipeline is working properly.
• Having this confidence and peace of mind that you won’t miss behavior or get blindsided by an incident or red team activity is amazing. It’s well worth the up front effort to put in place.

Wrap up

• Key takeaways.
• A list of my favorite Detection Engineering resources.
• Sharing a collection of over 25 detection rules with the community to bolster our monitoring & detection capabilities for GitHub Enterprise!
• Thank you for your time & attention.
• Q&A.