Off-By-One 2024 Day 2 - AI Powered Bug Hunting Evolution and benchmarking : Alfredo Ortega

Abstract

While AI holds promise for assisting bug hunting, its actual impact remains unclear. This presentation addresses this gap by introducing Crash-Benchmark, a standardized evaluation framework for AI-driven static analysis tools.

We’ll share results from a simple bug-hunting AI agent, AK1, and discuss the implications for optimizing AI-based bug hunting in C/C++ codebases.

AI-bughunting presents unique challenges: Early models lacked sophistication, struggling to comprehend long codebases. Moreover, privacy concerns often necessitate exclusive use of local models, which are inherently less capable than commercial AI models offered by industry leaders such as OpenAI and Google.

To illustrate this challenge, we’ll showcase AK1, a simple rule-based AI agent capable of autonomously identifying various bug classes within C/C++ codebases.

Notably, its model-agnostic design allows it to improve performance with each new model release. Nevertheless, evaluating the effectiveness of AI-based tools poses difficulties due to the subjectivity of the output.

Speaker
Alfredo Ortega is a security researcher and bug hunter, delivering presentations at over two dozen prominent information security conferences globally, including Black Hat, Defcon, Syscan, and Hackers-to-Hacker (H2HC) events, dating back to 2007.

Ortega holds a Doctorate degree in Computer Science from the Instituto Tecnológico de Buenos Aires. He is also the founder and primary architect of Neuroengine.ai, a pioneering platform dedicated to the open-source distribution and collaborative development of open-source artificial intelligence models, promoting community-driven innovation.

Alfredo serves as a Web3 Auditor at Coinfabrik, where he leverages his extensive expertise to fortify the security posture of cryptocurrency infrastructures, ensuring the integrity and reliability of decentralized systems.

Follow Alfredo on X @ortegaalfredo