2024-08-07 –, Siena
The rush to embed AI into everything is quickly opening up unanticipated attack surfaces. Manipulating natural language systems using prompt injection and related techniques feels eerily similar to socially engineering humans. Are these similarities only superficial, or is there something deeper at play? The Cognitive Attack Taxonomy (CAT) is a continuously expanding catalog of over 350 cognitive vulnerabilities, exploits, and TTPs which have been applied to humans, AI, and non-human biological entities. Examples of attacks in the CAT include linguistic techniques used in social engineering attacks to prompt a response, disabling autonomous vehicles with video projection, using compromised websites to induce negative neurophysiological effects, manipulating large language models to expose sensitive files or deploy natively generated malware, disrupting the power grid using coupons, and many other examples. The CAT offers the opportunity to create on demand cognitive attack graphs and kill chains for nearly any target. This talk concludes with a brief demo integrating cognitive attack graphs into a purpose-built ensemble AI model capable of autonomously assessing a target's vulnerabilities, identifying an exploit, selecting TTPs, and finally launching a simulated attack on that target. The CAT will be made publicly available at the time of this presentation.
A pdf copy of the complete white paper is available to reviewers here:
https://drive.proton.me/urls/X7R4RS9RX0#5DcktrTQRNBV
References in [brackets] are included in the white paper, as well as more detailed explanations for each of the key points.
DEFINING COGNITIVE SECURITY
Three security domains have been proposed to exist in the modern security environment [4]:
• the physical domain,
• the cyber (or information systems) domain
• and the cognitive domain
• The principles of cognition suggest that these principles can be manipulated to the benefit of others and to the detriment of the cognitive system being targeted.
THE COGNITIVE ATTACK TAXONOMY
• Frameworks have emerged to describe disinformation attacks, such as DISARM [5], or to describe social engineering attacks, such as the Social Engineering Taxonomy [6], or attacks on artificially intelligent systems [7], [8], [9], but there are not currently any frameworks or taxonomies which broadly account for attacks against cognitive systems. Such a cognitive security taxonomy will provide a useful resource for both researchers and threat analysts and could be readily adapted to threat modeling applications.
• The CAT addresses the research gap between other frameworks by specifically focusing on issues of cognitive security across agents and systems (automated and human, individual and distributed), while also accounting for security vulnerabilities exposed by simple human error [10].
• Cognitive Vulnerabilities: Defined as any cognitive processing principle which may be manipulated to the advantage of an attacker or to the detriment of the targeted cognitive system. The term ‘’cognitive vulnerabilities’’ is misleading in that it implies a weakness, but within the context of cognitive security, vulnerabilities should be considered as ‘’potentialities for misuse’’. A computer operating system that is capable of encrypting files for security purposes has an inherent ‘’vulnerability’’ in that this capability may also be weaponized by an adversary for malicious purposes.
• Cognitive Exploits: Within information security, an exploit refers to a sequence of commands, a software bug, a “glitch” or malfunction, or maliciously written code, which can be used to cause the targeted system to behave in unprescribed ways. The CAT uses the term “exploit” to similarly refer to a mechanism or action by which a cognitive vulnerability is manipulated.
• T/TTPs: Modern cognitive threat actors no longer need to develop the tools, tactics, techniques, or procedures from scratch, but instead commonly share such T/TTPS among themselves and among the population at large. T/TTPs are methods which invoke or leverage cognitive exploits to manipulate cognitive vulnerabilities.
Understanding the Cognitive Attack Taxonomy: The Cognitive Attack Taxonomy has a standardized set of labels which repeatedly appear in the CAT descriptions. This uniformity is intended to facilitate ease of comparison between entries and locating information within each entry . (See the Appendix for an example entry).
CAT Name: This is the common name to describe the cognitive attack taxonomy (CAT) vulnerability, exploit, or T/TTP (VET).
Short Description: This is a brief description of the CAT-VET is usually a two to three sentences maximum description of the entry.
CAT ID: Intended to be a unique identifier for the CAT-VET. The prefix "CAT" refers to the Cognitive Attack Taxonomy, followed by the year, and finally the serial number of the CAT-VET. For example, CAT-2021-005 identifies the fifth CAT-VET cataloged in calendar year 2021.
Layer: This refers to the human interconnection model extension to the OSI Model [12], with the inclusion of relevant OSI layers.
• Layer 7: This is the layer that AI typically operates at (according to ChatGPT4).
• Layer 8: The human layer at which heuristics, biases, and other psychological influence techniques operate. Social engineering or influence operations function at this layer.
• Layer 9: The organizational layer, manipulation techniques at this layer operate through policy functions.
• Layer 10: The legal layer, manipulation at this layer occurs through legislative processes or court cases.
Operational Scale: This refers to the typical or expected scale this CAT-VET is deployed at. Social engineering engagements typically occur at the tactical level (a single attacker and a single target), while disinformation campaigns usually occur at the operational level, and finally cognitive warfare tends to be more strategic in nature [13].
• Tactical: These are typically individual encounters with a single attacker and single target.
• Operational: This level refers to multiple engagements over a period of time, typically involving multiple parties.
• Strategic: Nation-state or nation-state level actors exercising multiple operations to exert strategic influence objectives.
• Level of Maturity: CAT-VETs exist on a continuum from theoretical to well-established.
• Category: The CAT-VET category informs whether an entry is a cognitive vulnerability, cognitive exploit, or is a tactic, technique, tool, or procedure.
Vulnerability: Cognitive Vulnerability
Exploit: Cognitive Exploit
T/TTP: Cognitive Attack Tactic/Technique, Tool, or Procedure
• Subcategory: The CAT-VET subcategory refers to the type of vulnerability, exploit, or T/TTP the entry falls within. The subcategory is intended to be expandable as new discoveries are made, while CAT-VET Categories are intended to be immutable.
• Also Known As: Identifies alternative names or adjacent terms and concepts to the entry.
• Brief Description: This description is intended to be a five words or less description of the entry.
• Closely Related Concepts: These are concepts which relate to the entry but are not alternative names.
• Mechanism: This describes the operation of the CAT-VET entry. If the entry is a vulnerability, then the mechanisms described will be exploits or T/TTPs. If the entry is an exploit, the mechanisms will include vulnerabilities the exploit may be applied to or T/TTPs which may take advantage of the exploit, alternatively, a T/TTP entry will list cognitive vulnerabilities which it may be applied against or exploits which might be leveraged T/TTP deployment.
• Interactions: Describes adjacent phenomena which may enhance or degrade the entry. For example, decision fatigue (CAT-2022-050) is a cognitive vulnerability which describes the experience of increasing difficulty in resisting temptation as choices are made. This vulnerability may be enhanced by presenting more closely related alternatives when choosing between alternatives (increasing cognitive load) or degraded by presenting personally relevant information at the key decision point (increasing semantic relevance).
• Detailed Description: This category provides a detailed description of the entry. This section can be as long as needed and is intended to be expandable to allow for new or updated information about the entry.
• Use Case Example: Examples of how the CAT-VET might be used in a hypothetical situation.
• Example From the Wild: Example of the CAT-VET has been used in a documented case.
• Comments: General commentary on the CAT entry. Community discussions may exist here in addition to the page discussion notes.
• References: All CAT entries are to be backed by references to the maximum practical extent. All entries should be backed by research and/or observations from the wild. The CAT is not intended to be a repository of fantasy or opinion.
APPLICATIONS OF THE COGNITIVE ATTACK TAXONOMY
• Walk through a semi-automated red team example CAT applied to human security assessment
• Cognitive Attack Graph for the High-Level Executive (Example)
• Cognitive Attack Graph for Human Resources Manager (Example)
CONCLUSION
• Will conclude the presentation by leaving the audience feeling terrified but with a ray of hope.
WORK CITED - All previous research is cited in the white paper and will be included on the slides.
APPENDIX - EXAMPLE CAT ENTRY: RECIPROCITY o accompany this talk.
Dr. Matthew Canham is a former Supervisory Special Agent with the Federal Bureau of Investigation (FBI), he has a combined twenty-one years of experience in conducting human-technology and security research. He currently holds an affiliated faculty appointment with George Mason University, where his research focuses on threats posed by maliciously produced AI generated content and synthetic media social engineering. Dr. Canham recently founded the Cognitive Security Institute, a non-profit organization dedicated to understanding the key components of cognitive attacks and discovering the best ways to defend against these.
Dr. Canham has provided synthetic media threat awareness training to NASA (Kennedy Space Center), DARPA, MIT, US Army DevCom, the NATO Cognitive Warfare Working Group, the BSides Las Vegas security conference, the Misinformation Village at DefCon, and the Black Hat USA security conference. He has appeared on multiple podcasts including BarCode Security, Weapons of Mass Disruption, 8th Layer Insights, The Cognitive Crucible Podcast, the ITSP Podcast, and he has appeared as a deepfake subject matter expert on several news outlets.