login

Overcoming Barriers in Security DSLs with BabbelPhish: Empowering Detection Engineers using Large Language Models
.ical

The rise of detection-as-code platforms has revolutionized threat detection, analysis, and mitigation by leveraging domain-specific languages (DSLs) to streamline security management. However, learning these DSLs can be challenging for new detection engineers.

In this talk, we introduce BabbelPhish, an innovative approach utilizing large language models to bridge the gap between natural language queries and security DSLs. We demonstrate its application to MQL, Sublime Security’s free DSL for email security, and its potential extension to other DSLs. BabbelPhish enables users to harness the full potential of detection-as-code platforms with familiar natural language expressions, facilitating seamless transitions from triage to querying and coding.

We will discuss BabbelPhish's architecture, training process, and optimization techniques for translation accuracy and MQL query validity. Through live demonstrations and user interviews, we will showcase its real-world applications and implementation options, such as a VSCode plugin.

Join us as we explore how large language models can integrate natural language capabilities with the precision of security DSLs, streamlining security management and threat hunting, and making detection-as-code platforms accessible to a wider range of security professionals.

Outline

Introduction

  • The rise of detection-as-code platforms and their role in transforming security management and threat hunting
  • Benefits of these platforms, such as automation, scalability, and consistency in security operations
  • Problem: Barrier to entry – Challenges faced by new or would-be detection engineers due to the learning curve associated with domain-specific languages

The Impact of Translating Natural Language to Domain-Specific Languages

  • Accessibility and Usability – security workers may not be familiar with the syntax and structure of MQL, which can limit their ability to utilize security products effectively.
  • Speed and Efficiency – Allowing workers to query security platforms using natural language can help expedite identifying, investigating, and implementing mitigation strategies for potential threats.
  • Reducing Human Error – One of the challenges in using a domain-specific language is the potential for human error when constructing queries.
  • Improved Collaboration and Communication – Security is often a team effort involving multiple stakeholders with varying levels of technical expertise. By enabling users to communicate their queries in natural language, BabbelPhish can facilitate better collaboration between team members who may be less experienced.

Language Models in Cybersecurity

  • Overview of large language models (LLMs) and their applications in cybersecurity (e.g., vulnerability detection, malware classification, and automated threat intelligence)
  • Explain the benefits and challenges of using LLMs for natural language processing tasks in cybersecurity.

Overview of MQL: Message Query Language

  • Provide a short overview of MQL as a domain-specific language for analyzing email messages and hunting threats.
  • Describe MQL's syntax, functions, and common snippets.
  • Show the MQL Editor

Introducing BabbelPhish: A Solution for Overcoming Barriers in Detection-as-Code Platforms

  • The need for a tool that enables users to go from triage to questions to code, all in natural language
  • How BabbelPhish leverages LLMs to translate natural language queries into security-related DSLs seamlessly

Designing BabbelPhish: Architecture Overview

  • Preprocessing module: Converts raw natural language input into a structured format that the LLM can process efficiently
  • Large Language Model (LLM): The core component responsible for translating natural language input into MQL queries
  • MQL generator: Takes the LLM's output and constructs valid MQL queries, ensuring proper syntax and structure
  • Post-processing module: Further refines MQL queries to optimize performance and maintain consistency with the Message Data Model (MDM)
  • Feedback loop: Allows users to provide feedback on translation accuracy, helping to improve BabbelPhish's performance over time

Designing BabbelPhish: Data Collection and Model Training

  • Data collection: Gather large-scale datasets containing natural language queries and their corresponding MQL translations
  • Data preprocessing: Clean and preprocess the collected data, ensuring consistency and removing any potential noise
  • Model selection: Choose a suitable LLM architecture (e.g., GPT-4) based on performance and compatibility with the task
  • Fine-tuning: Train the LLM on the processed data, adjusting model parameters to optimize translation accuracy
  • Validation: Evaluate the trained model using a held-out dataset, measuring performance metrics such as translation accuracy and query integrity

Designing BabbelPhish: Optimal Translation & Generating Valid Queries

  • Loss function optimization: Utilize custom loss functions that prioritize the preservation of MQL query semantics and syntax
  • Regularization: Implement techniques such as dropout, weight decay, and early stopping to prevent overfitting and improve generalization
  • Attention mechanisms: Employ attention mechanisms to help the model focus on relevant parts of the input when generating MQL queries
  • Rule-based post-processing: Apply a set of predefined rules to refine generated MQL queries, ensuring compliance with MDM and MQL syntax
  • Continuous improvement: Leverage user feedback and real-world examples to iteratively improve the model's performance and address any identified shortcomings

Use Cases and Examples

  • Demo of capability: Provide a live demonstration of BabbelPhish in action, showcasing the seamless translation of natural language queries into MQL for effective threat analysis and hunting
  • Learnings from user interviews with real detection engineers: Share insights and feedback gathered from interviews with detection engineers who have used BabbelPhish, highlighting the tool's practical benefits and areas for improvement in real-world scenarios
  • Various implementation options: Discuss different ways to integrate BabbelPhish into existing workflows, such as embedding it within an MQL Editor, integrating it with email security platforms, or incorporating it into custom security dashboards for enhanced user experience and productivity

Limitations and Future Work

  • Discuss the limitations of BabbelPhish, such as potential translation errors and handling of ambiguous queries.
  • Share ideas for future improvements, including better handling of complex queries and integration with other security tools.

Conclusion

  • Recap the importance of using LLMs to bridge the gap between natural language and DSLs in detection-as-code platforms.
  • Highlight how BabbelPhish overcomes the barrier to entry for new or would-be detection engineers, enabling them to contribute effectively and make a significant impact.
  • Encourage the audience to explore the potential of BabbelPhish and similar tools in their security environments to enhance accessibility and productivity.