AI in Defense: The real danger of delegating critical decisions

The fallacy of superintelligence versus operational reality

Over the last decade, public and political debate on artificial intelligence has been hijacked by a dystopian narrative focused on technological singularity, hostile superintelligence, and the end of humanity at the hands of self-aware machines. This approach, fueled by science fiction and warnings from some industry leaders, has diverted attention from a much more mundane, immediate, and therefore dangerous threat: the accelerated deployment of immature, opaque, and error-prone AI systems in critical infrastructure and high-stakes decision-making environments.

As researchers Timnit Gebru and Emily M. Bender point out in their recent analysis published in The Guardian, the obsession with hypothetical existential risks is acting as a smokescreen that allows tech companies and governments to integrate AI tools into strategic sectors without the necessary levels of auditing and security. The current operational reality is not that of a chatbot wishing to take over the world, but that of a statistical tool that, when it fails, can catalyze geopolitical crises of global scale.

What really happened?

The incident reported by CNN on September 18, although it remains unverified by other major sources, illustrates the fragility of the current system. According to the report, a U.S. military unit was on the verge of intercepting a Chinese vessel after receiving an AI-assisted intelligence report that erroneously classified the ship's cargo as nuclear weapon components. The deployment of forces—including aircraft and military personnel ready for boarding—was minutes away from execution. It was the last-minute human verification that prevented an armed confrontation which, in the current delicate balance of power, could have quickly escalated into a nuclear conflict between two superpowers.

This event, if confirmed, would not be a failure of "conscious AI," but a failure of design and governance: the use of a probabilistic language model for intelligence analysis tasks without proper validation protocols. The irony lies in the fact that the system, marketed under promises of precision and technical superiority, failed in a basic factual verification task, demonstrating that current models lack a semantic understanding of the geopolitical context and the consequences of their hallucinations.

The trap of automation in decision-making

Historically, automation in the military sphere has followed strict human-in-the-loop principles. However, the pressure for competitive advantage and the speed of data processing are eliminating those layers of security. The Chinese vessel incident exposes a critical gap: the opacity of AI systems (the "black box" effect) prevents effective accountability. Unlike human error, where there is a traceable chain of command and an explainable logic, a chatbot's error is difficult to audit in real-time, which creates a false sense of security in commanders who rely blindly on the report generated by the machine.

This phenomenon recalls the failures in early warning systems of the Cold War, such as the Stanislav Petrov incident in 1983. The fundamental difference is that, back then, the radar system was deterministic software with hardware flaws; today, we are delegating semantic interpretation and intelligence synthesis to probabilistic models that, by definition, are designed to be creative and persuasive, not necessarily truthful.

Consequences and lessons for the sector

• Erosion of trust and global stability: Reliance on unstable AI systems in national defense can destabilize international alliances. If governments cannot trust the integrity of intelligence data, diplomacy becomes useless.
• Regulation focused on deployment, not fiction: It is imperative that lawmakers stop legislating on the "AI of the future" and begin auditing the automated decision systems currently in use. Regulation must demand safety and explainability standards similar to those of civil aviation or the nuclear industry.
• The human factor as an absolute safeguard: Automation must always be a support, never a substitute. In scenarios where error has a geopolitical cost, human validation must be an independent and mandatory process, not a bureaucratic formality.

The history of technology teaches us that every disruptive innovation requires a period of stabilization and regulation before mass adoption. However, in the AI arms race, the defense sector seems to have ignored this lesson. As Gebru and Bender rightly point out, allowing low-quality algorithms to push us into a nuclear conflict due to a factual hallucination is not an inevitable risk of progress; it is an operational negligence that requires immediate correction in global defense policy.