When an experimental artificial intelligence model from OpenAI broke out of a cybersecurity test this summer and infiltrated the platform Hugging Face, the latter turned to a surprising tool to investigate the attack: a Chinese open-weight AI model. Its first choice, commercial American models, had refused to analyze some of the malicious code because of their safety restrictions.
The episode illustrates a problem confronting U.S. president Donald Trump and Chinese president Xi Jinping, who are expected to discuss AI safety at their meeting this week. Chinese developers are putting increasingly capable models into the hands of people around the world, even as American AI companies are urging governments to establish stronger safeguards. The spread of powerful open models is exposing the limits of an approach to AI safety that relies on developers retaining control over their technologies.
While leading American AI companies such as OpenAI and Anthropic keep the details and code of their most powerful models a secret, Chinese developers have embraced openness. According to a recent report by Mozilla, seven of the AI marketplace OpenRouter’s 10 most-used AI models, as measured by token use, in August were Chinese-built and open-weight. Such downloadable models can be adapted and put to work by developers around the world, making them attractive alternatives to proprietary, relatively costly American systems.
On supporting science journalism
If you’re enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.
A lot of the disagreement comes down to control. Companies that offer proprietary AI models can restrict how people use them and modify safeguards as new risks emerge. Open-weight models, by contrast, let users download and modify the underlying technology, including by removing safeguards intended to prevent harmful uses. They are often called “open source” as an imprecise but evocative shorthand, although their developers don’t usually make the training data or code available.
Restrictions on proprietary models can make them harder to misuse. But those same restrictions can also frustrate cybersecurity researchers trying to understand the extent of the systems’ capabilities. Hanna Foerster, a computer science Ph.D. student at the University of Cambridge, says that some providers offer special access to researchers studying defensive security but that such access is much harder for academics studying offensive capabilities. Open models offer another option, and their capabilities are catching up. “There are some start-ups that are working on open-source models that have actually got similar capabilities … to what they’re getting for some closed-source, superbig models,” she says.
But Foerster says much of the danger lies in the software and infrastructure surrounding a model. Known as a “harness,” this software can turn a chatbot into an AI agent that is capable of acting autonomously. If you give that agent access to a computer’s files or the ability to run code, it can do considerably more than answer dangerous questions.
That has implications for how AI systems are evaluated and safeguarded. “The current auditing discussion strengthens the case for thinking beyond model-level safety. Safety increasingly depends on the whole system around a model,” says Avijit Ghosh, lead technical AI policy researcher at Hugging Face. He points to the permissions an agent receives and the use of isolated computing environments known as sandboxes.
Ion Stoica, a professor of computer science at the University of California, Berkeley, doubts that American efforts to restrict access to powerful AI could prevent the spread of capable models. “I don’t understand,” he says. “What’s the alternative?” Once models are freely available, he argues, restricting American developers is unlikely to prevent malicious actors elsewhere from obtaining comparable capabilities. He also questions leaving the most powerful AI systems in the hands of a few companies: Can you imagine, he says, “a world in which you are going to trust OpenAI and Anthropic [to] know what they are doing?”
That poses a problem for international regulation. In an analysis released on Monday, ahead of the U.S. and Chinese presidents’ talks, researchers at the Center for Strategic and International Studies wrote that applying the same safety standards to American and Chinese frontier models would require regulators to either extend their reach into China, which the authors regarded as exceedingly unlikely, or find a way for the two countries to recognize and verify each other’s safety assessments.
U.S. officials have proposed a narrower first step: a mechanism for the two countries to notify each other about serious AI incidents that threaten national security. Such an agreement wouldn’t settle the broader disagreement over how to control powerful models that are already in circulation. The Hugging Face breach and its fallout offers an unusual snapshot of that dilemma: an American company’s experimental model mounted the intrusion, and a downloadable Chinese model helped investigators piece together what happened.
Subscribe to Support Independent Journalism
Great science journalism requires human expertise, time, effort and creativity. And it costs money. That’s why I and the journalists here at Scientific American hope you’ll join our community.
When you subscribe, you are supporting staff and freelance journalists who are passionate about telling science stories that are true, important and compelling. Our editors and reporters are often experts in their fields, which means they understand the nuances of big discoveries and can untangle the breakthroughs from the hype. With a subscription, you are also supporting rigorous fact-checking to ensure the words we publish are precise and accurate. And you’re supporting original illustrations, graphics and photos that bring you closer to an advanced laboratory, an ice sheet in Antarctica or a space mission in orbit. You’re helping us craft other types of high-quality journalism as well: Our newsletters are carefully written, edited and curated by staffers you have or will come to know and love. Our Science Quickly podcast is based on original reporting, collaboration with editors and scientists and exacting production.
Subscriptions keep this engine running so we can continue to deliver thoughtful, rigorous and independent science journalism to you. In an era of viral misinformation, this work is crucial. If you value what we do, I hope you’ll consider joining us as a subscriber
