, Florentine F
Large Language Models (LLMs), are increasingly being integrated into enterprise environments for the purposes of automation, analytics, and decision-making. Although their fine-tuning capabilities enable the development of tailored models for specific tasks and industries, LLMs also introduce new attack surfaces that can be exploited for malicious purposes.
In this presentation, we unveil how we transformed an LLM into a stealthy C2 channel. We will demonstrate a PoC attack that leverages the fine-tuning capability of a popular generative AI model. In this attack, a victim unwittingly trains the model using a dataset crafted by an attacker.
This technique transforms the model into a covert communication bridge, enabling attackers to exfiltrate data from a compromised endpoint, deploy payloads, and execute commands.
We will discuss challenges we faced, such as AI hallucinations and consistency issues, and share our approach and the techniques we developed to mitigate the issues. Additionally, we will examine this attack from a defender’s perspective, highlighting why traditional security solutions struggle to detect this type of C2 channel, and what can be done to improve detection.
Join us as we break down this unconventional attack vector, and demonstrate how LLMs can be leveraged for offensive operations.
In this presentation we will share a proof of concept we developed, originally as part of a data exfiltration focused research project held in Palo Alto’s Cortex TI team.
As we mapped the landscape we found that Large Language Models (LLMs) are increasingly leveraged by attackers for automation, phishing, and malware development, but their true offensive potential remains largely untapped.
In this talk, we explore a novel technique: abusing the fine-tuning process of LLMs to establish a covert C2 channel and exfiltrate sensitive data. Unlike traditional AI abuses that focus on prompt engineering or model manipulation, this approach enables adversaries to embed and retrieve information through the fine-tuning mechanism, bypassing common security measures.
At first glance, using LLMs for covert communication seems impractical due to security controls, session-based memory limitations, and unpredictable model behavior. However, by fine-tuning a widely used model, we successfully created a reliable attack method where a victim unknowingly trains an LLM with sensitive data, allowing an attacker to extract this data and issue commands remotely. We will showcase our PoC, highlighting key technical challenges such as AI hallucinations, consistency issues, and response unpredictability—along with the techniques we used to overcome them.
From a defender’s perspective, detecting this attack is quite challenging. Traditional security solutions, such as EDRs and network monitoring tools, do not effectively track AI interactions, allowing malicious activity to blend in with legitimate AI usage. We will analyze why conventional detection methods fail and discuss potential mitigation strategies, including behavioral anomaly detection.
This talk provides an in-depth look at the risks associated with LLM fine-tuning and its implications for security. Through a pre-recorded demonstration, we will illustrate how attackers can use AI-powered C2 channels in real-world scenarios. As AI continues to evolve, understanding and securing its hidden attack surfaces is critical—before adversaries fully "tune in" to these emerging opportunities.
Noa Dekel is a Senior Threat Intelligence Researcher at Palo Alto Networks. Starting her career as a Threat Intelligence analyst in the defense sector, today Noa specializes in threat hunting, malware analysis, and detection engineering.