Amazon challenges the dogma of human oversight in AI

What happened?

Eric Brandwine, distinguished engineer and VP of Amazon Security, stated in an interview with The Register that Amazon does not support the 'human-in-the-loop' model for governing AI agents. According to Brandwine, humans are inherently inconsistent and, like AI systems, non-deterministic. The company advocates for a model where human oversight is applied strategically rather than as continuous control of every agent action. Brandwine noted that “humans tend to be a little precious about humans,” and although we have millennia of experience dealing with humans, “we know how humans fail; we're comfortable with that. So human-in-the-loop is not necessarily the gold standard.” This stance breaks with the dominant narrative that has prevailed in the industry since the dawn of modern AI.

Why is it important?

This stance challenges the dominant narrative in the industry, where human oversight has been presented as the infallible solution to mitigate AI risks. By pointing out human fallibility, Amazon opens a crucial debate on the real effectiveness of traditional control mechanisms. 'Normalization of deviance'—a phenomenon where humans progressively ignore anomalies—is cited as a concrete risk, exemplified by cases in hospital emergencies and military settings. Brandwine had already addressed this topic in 2017 during a talk at AWS re:Invent, describing how people in high-pressure environments, such as emergency rooms or military cockpits, can stop responding to alarms after multiple false positives, with tragic consequences. This documented human bias calls into question the effectiveness of continuous human oversight, especially in AI systems operating at high speed. Amazon's stance, backed by empirical evidence, forces the industry to reconsider whether 'human-in-the-loop' is truly the best practice or merely a comfort solution.

Consequences for businesses and users

Amazon's stance could influence how companies design AI governance systems, shifting focus from continuous human oversight to more robust automated controls. For users, this could mean more scalable and consistent systems, but it also raises questions about accountability in case of errors. Google Cloud, through Francis deSouza, has already hinted at a similar move toward AI-led defense supervised by humans. In the business realm, the adoption of autonomous AI agents could accelerate, reducing operational costs but increasing the need for clear governance policies and real-time monitoring systems. However, the lack of direct human oversight could generate resistance in regulated sectors like finance or healthcare, where traceability and accountability are critical. The challenge will be balancing efficiency and control, a debate already seen in the adoption of autonomous vehicles or algorithmic trading systems.

Security implications

Brandwine warns that asking humans to approve every action of an AI agent at high speed inevitably leads to performance degradation. He cited a real-life example: in emergency rooms, constant false alarms lead staff to ignore critical signals, resulting in fatal outcomes. This suggests that companies should invest in automated monitoring systems and high-level policy definition, rather than relying on human reviewers for every decision. Additionally, Brandwine noted that both humans and AI systems are non-deterministic: neither can guarantee the same outcome from the same input twice. Therefore, the key is not to eliminate the human but to design systems that leverage their strengths (such as strategic judgment) without exposing them to repetitive tasks that inevitably lead to fatigue and error.

What should readers know?

• Not a total rejection: Amazon does not remove the human from the system but repositions them as a strategic supervisor, not a validator of every step. Oversight should be applied at key decision points, not in the continuous loop.
• The risk of deviance: 'Normalization of deviance' is a documented human bias that can cause reviewers to overlook critical errors over time. Brandwine has studied it since 2017 and considers it a real danger in high-speed environments.
• Paradigm shift: Major tech companies like Amazon and Google are moving the debate toward more automated AI governance, with human oversight at key points, not in the continuous loop. This could redefine industry standards.
• Regulatory implications: This stance could clash with future regulations requiring mandatory human oversight, such as the EU AI Act, which classifies high-risk AI systems and requires effective human oversight. Amazon may face tensions with regulators if its approach is perceived as evading responsibility.

“If you put a human inside of this tight loop, and ask them to make approval decisions for agentic tools repeatedly, time after time, they'll do a good job. And then they'll do an okay job. And pretty quickly they'll be doing a poor job.” — Eric Brandwine, Amazon Security

Context and comparisons

The debate over the human role in AI is not new. For years, the industry has promoted 'human-in-the-loop' as a safety guarantee. However, cases like the fatal accident of an Uber autonomous vehicle in 2018, where the safety driver did not intervene in time, illustrate human fallibility. In that incident, the safety driver was looking at her phone instead of the road, a classic example of normalized deviance. Another case is Air France Flight 447 in 2009, where pilots ignored stall warnings due to inadequate training and fatigue, resulting in 228 deaths. These events show that even in high-risk situations, humans fail. Amazon now brings this argument to the realm of enterprise AI agents, where decision speed is critical. Compared to Google's approach, which also advocates for strategic but not continuous human oversight, Amazon appears to be leading a shift toward more automated governance. However, companies like Microsoft still defend the traditional model, creating a divide in the industry. Amazon's experience with AWS and its vast infrastructure lends credibility to this debate but also generates skepticism about its commercial motivations.

Conclusion

Amazon's criticism of 'human-in-the-loop' represents a turning point in AI governance. By acknowledging human limitations, the company pushes for a more realistic and scalable approach, though not without risks. Businesses and regulators will need to adapt to this new reality, where human oversight is not the ultimate goal but one tool within an ecosystem of automated controls. The key will be designing systems that combine artificial intelligence with human oversight at the right moments, avoiding both reviewer fatigue and lack of accountability. As Brandwine concludes, “it's not about eliminating the human, but about putting them where they can truly make a difference.” This paradigm shift could accelerate the adoption of autonomous agents but will also require new trust metrics and regulatory frameworks that balance innovation and safety.