AI Passes Security Tests: Regulators Blind to Risks

What happened?

According to an Axios report picked up by The Next Web, the benchmarks used to measure the ability of AI systems to carry out cyberattacks have become obsolete. Frontier models like GPT-4, Claude 3, or Gemini Ultra easily surpass tests originally designed to evaluate hacking skills, such as capturing flags in controlled environments (CTF) or exploiting known vulnerabilities. This means the tests no longer differentiate between safe and dangerous models. The report highlights that in tests like those on the HackTheBox platform, advanced models solve challenges that previously required hours of human work in minutes, evidencing a growing gap between actual capability and measurement tools.

Why is this important?

The obsolescence of these benchmarks has critical implications. On one hand, U.S. regulators have until August 1 to implement a classified AI evaluation system, per a White House executive order. Without updated tools, federal agencies will not be able to properly assess the risks of the systems they acquire or deploy. On the other hand, companies using AI in cybersecurity or developing advanced models lack reliable metrics to validate the safety of their systems. As The Next Web article notes, 'the tools designed to measure how dangerous AI can be have stopped working.' This leaves security teams 'half-blind,' unable to anticipate automated attacks that could compromise critical infrastructure, financial systems, or personal data. Additionally, the lack of updated benchmarks makes it difficult for regulators to establish minimum safety standards, which could delay the approval of AI systems in sensitive sectors.

Historical context and comparisons

This phenomenon is not new in AI. Over the past decade, benchmarks like ImageNet for computer vision or GLUE for natural language were surpassed by more advanced models, leading to the creation of new, more complex datasets. However, in cybersecurity, the problem is more severe because risks are dynamic and human attackers also improve. The difference now is that AI can automate attacks at a speed and scale that exceed human testing capacity. Moreover, the lack of transparency of proprietary models makes it difficult for the academic community to develop new evaluations. For example, while in 2018 hacking benchmarks required models to identify simple vulnerabilities in web applications, today's frontier models can exploit complex vulnerability chains that mimic real-world attacks. This evolution has left behind tests like the DARPA Cyber Grand Challenge, which in 2016 already showed AI's potential to automate defenses, but now prove insufficient to measure advanced offensive capabilities.

Consequences for companies and users

For companies, the inability to measure the real risk of AI models means they could be deploying vulnerable systems without knowing it. This is especially concerning in sectors like finance, healthcare, or critical infrastructure. A study by MITRE and the Center for Security and Emerging Technology (CSET) indicates that current models can generate malicious code or plan multi-stage attacks with an efficiency that surpasses many human hackers. End users, for their part, may be exposed to more sophisticated fraud or automated attacks, such as personalized AI-generated phishing or malware that adapts in real time. Cybersecurity teams are left 'half-blind,' as the article states, unaware of the offensive capabilities of the models they use or defend against. Additionally, cybersecurity insurers face difficulties in assessing premiums, as the risks associated with offensive AI are not precisely quantified.

What should readers know?

It is crucial that policymakers and CISOs understand that current benchmarks are not sufficient. New evaluation methodologies are needed that are adaptive and include continuous adversarial testing. For example, the UK's AI Safety Institute is already developing dynamic evaluations that update based on emerging model capabilities. Furthermore, collaboration between governments, academia, and industry is essential to create standards that evolve at the pace of AI. Meanwhile, any claim about the safety of an AI model based on outdated benchmarks should be treated with skepticism. Companies should demand from AI providers recent and transparent adversarial test results, and regulators should consider creating an independent body to validate benchmarks continuously.

The gap between AI's actual capability and the tools to measure it is widening, and regulators risk making decisions based on incomplete information.

Conclusion

The news that AI surpasses security tests is not a surprise to experts, but it is a wake-up call. The pace of advancement of frontier models demands an urgent review of evaluation mechanisms. If action is not taken soon, regulatory blindness could translate into large-scale security incidents, such as the use of AI to disable defense systems or carry out coordinated cyberattacks. The urgency is even greater considering that the next generation of models, like GPT-5 or Gemini Ultra 2, could incorporate even more advanced offensive capabilities. Therefore, it is imperative that industry and governments prioritize the creation of robust and adaptive benchmarks, before AI surpasses not only tests but also our ability to control it.