Amazon investigation led to White House ban on Anthropic models

What happened?

The White House issued an export control directive that forced Anthropic to block access to its Fable 5 and Mythos 5 artificial intelligence models for foreign users. The measure originated from a security investigation conducted by Amazon, which demonstrated that through a series of prompts it was possible to obtain information useful for cyberattacks. Amazon CEO Andy Jassy shared these findings with the U.S. government, precipitating the decision. According to the Wall Street Journal, Amazon's investigation claimed that through a sequence of prompts, Fable 5 could be made to reveal information susceptible to use in cyberattacks. Shortly after Jassy shared the findings with the White House, the government ordered the blocking of access to the models by foreign citizens. This case marks a milestone at the intersection of AI safety and export controls, as it is the first time a security investigation by a tech company has directly led to a specific government restriction on AI models.

Why is this important?

This is one of the first cases where a security investigation by a tech company directly leads to a government restriction on AI models. It sets a precedent for future regulations and could affect international collaboration in AI research. Additionally, the ban affects foreign researchers and developers who rely on these models, creating tensions between open innovation and national security. The directive is part of the Biden administration's policy to prioritize AI safety, as reflected in the October 2023 Executive Order on AI, which established reporting requirements for frontier models. However, unlike previous restrictions focused on hardware (such as NVIDIA chips to China), this measure targets software, which could have broader implications for the global distribution of AI models. The lack of transparency about Amazon's specific findings draws criticism over potential conflicts of interest, given that Amazon competes with Anthropic in the cloud and AI market. Moreover, Anthropic has a close relationship with Amazon, which has invested billions in the startup and offers its models through Amazon Bedrock.

Consequences

• For Anthropic: Loss of revenue and international users, possible delay in global adoption of its models. Anthropic relies heavily on enterprise customers outside the U.S., especially in Europe and Asia. The restriction could reduce its market share against competitors like OpenAI, which does not yet face similar restrictions.
• For the industry: Increased regulatory scrutiny; other companies could face similar restrictions if their models present risks. This could lead to market fragmentation, where AI providers must implement geographic access controls, increasing compliance costs. Companies like Meta, with its open-source Llama model, could benefit by not being subject to export restrictions.
• For the AI community: Fragmentation of the ecosystem, with geographic barriers to accessing advanced models. International researchers and startups will lose access to cutting-edge tools, potentially slowing innovation in regions like Latin America, Africa, and parts of Asia. It could also incentivize the development of local models, but with limited resources.
• For national security: The measure could be effective in preventing malicious uses, but could also generate diplomatic tensions with allies that rely on these models. The European Union, for example, could see this as interference in its technological sovereignty and accelerate its own AI regulations.

Context and analysis

The case recalls previous restrictions on NVIDIA AI chips to China, but focuses on software rather than hardware. The Biden administration has prioritized AI safety, and this move could be a precursor to broader regulations. However, the lack of transparency about Amazon's specific findings draws criticism over potential conflicts of interest, given that Amazon competes with Anthropic in the cloud and AI market. Notably, Anthropic had launched Fable 5 and Mythos 5 just weeks earlier, positioning them as cutting-edge models with enhanced reasoning and code generation capabilities. Amazon's investigation, according to sources, exploited vulnerabilities in the models' security layer, known as "jailbreaking," which allowed bypassing safeguards. This raises questions about whether Anthropic was informed of the findings before the White House acted. The situation also highlights the growing tension between AI openness and national security. While some open AI advocates argue that restrictions will only shift research to less regulated jurisdictions, security proponents point out that advanced models could be used to develop cyber or biological weapons. The U.S. government has been working on a risk assessment framework for AI models, but this case shows that regulation can be reactive and based on reports from private companies.

“Amazon's investigation shows that even the most advanced models can be manipulated for malicious purposes, but the government response must be measured so as not to stifle innovation.” — Analyst at TheVortiq

What should readers know?

If you are a developer or researcher outside the U.S., access to Fable 5 and Mythos 5 will be restricted. Alternatives such as open-source models (Meta's Llama, Mistral AI's Mistral) or from other providers (OpenAI's GPT-4, Google's Gemini) may be options. However, these alternatives may not have the same performance on specific tasks. The situation is dynamic; more details are expected on the security criteria that led to the ban. Anthropic has stated that it is working with the government to clarify the terms and possibly lift the restriction after implementing additional security measures. Meanwhile, international users are advised to seek models hosted in their regions or consider local AI solutions. The AI community will be watching to see if this measure extends to other models or if a formal process is established to assess security risks in frontier models.