2025-10-13 –, Main Track
In the last year we have seen an increase in established security vendors adding AI assistants to their suite of products. Alongside this, a small industry of bespoke AI Security Assistant designed to help businesses. How can you measure the benfit they bring to a business? We've interviewed SOC analysts and business owners to seek the "right" answer. Evaluating security products is an expensive endeavour that a lot of business cannot afford. The promise of analysts being more efficient and reducing alert fatigue get thrown around. What are the metrics you can look at as an independent tester to give value to both enterprises and vendors alike?
The allure of reduced alert fatigue and greater operational efficiency brings organizations to AI SOC solutions, yet beneath the buzzwords lies the challenge of rigorous, metrics-driven evaluation.
Key metrics—such as mean time to detect (MTTD) and respond (MTTR), false positive reduction rates, threat coverage, and investigative accuracy—should be prioritized during evaluations. These indicators reveal not just the system’s technical prowess, but its real-world impact on both security outcomes and resource allocation. Metrics like analyst effort, time to investigate, escalation rates, and the level of automation are vital in quantifying gains in efficiency and reductions in manual workload.
Hidden costs may emerge from necessary infrastructure upgrades, custom integrations, knowledge transfer, and the retraining of SOC analysts to work alongside AI agents. Furthermore, true proof of value is not always immediate—learning curves, tuning, and rollout phases may extend timelines for return on investment.
Evaluations should also scrutinize vendors’ claims through direct measurement of key outcomes, investing in thorough documentation, validation, and explainability to foster analyst trust. Ultimately, organizations must weigh metrics reflecting not just detection and response.
Given these complexities, investing in an AI-powered SOC is an ambitious undertaking, requiring clear measurement strategies, realistic expectations for financial and labor commitments, and an understanding of the key buzzwords that signal true value rather than marketing hype.
Thought Leadership,
Stefan began his journey as a security software tester, specializing in targeted attacks. He has a proven track record of building and leading high-performing teams, developing advanced testing methodologies that drive innovation and set industry benchmarks. Previously CTO of SE Labs, has founded Artifact Security a new testing company that focuses on innovation in new security aproaches.