The latest artificial intelligence (AI) tools from Anthropic and OpenAI went to new extremes in trying to undermine a popular platform during testing by the UK’s AI Security Institute.
The AISI said on Tuesday that Anthropic’s Mythos and OpenAI’s Sol models engaged in a level of “autonomy and deception” it had not seen before.
During routine AI safety testing, an Anthropic agent created fake profiles of real people as it tried to trick a person standing between it and access to GitHub, a large platform where technology developers store software code.
Anthropic and OpenAI noted in response to AISI’s report that its test had reduced or removed normal safeguards.
AISI evaluators first noticed “unusual data transfers leaving our research systems” during a test, then found that “some of the agents being tested had engaged in sustained, potentially harmful activity directed at real people and organisations”.
It turned out that a Mythos agent had created “malicious code” and attempted to insert it into GitHub’s system.
The Mythos agent identified and researched the people who maintained GitHub and created a series of “fake online identities” based on those real people. It did so as part of an effort to pressure and trick the real people into approving its malicious code.
The agent even sent people direct messages masquerading as the real people it had researched.
“When the agent’s pull request was challenged in public, it edited its earlier activity to appear harmless and considered adopting a fresh identity to continue,” AISI said.
Throughout the attempts, it was human review that stopped the agent from succeeding in delivering the malicious code to GitHub.
While AISI said the Mythos agent had not been instructed specifically to avoid or carry out such behaviour, it was “the first time we have seen risks around autonomy and deception manifest this clearly, without specific prompting, in the real-world”.
The rival AI companies, which are poised to be listed on the public stock market, have in recent weeks said their tools were responsible for several cyber-hacking incidents.
Anthropic wrote in a public statement that the AISI testing parameters were “not representative of any of our production models”.
It added that the company is conducting its own investigation into the incident in order to “identify the causes of its behavior”.
A spokesperson for OpenAI said the AISI testing conditions “do not reflect ordinary use” and that the company would “continue working with evaluators and other stakeholders across the industry to strengthen shared practices for conducting evaluations safely as models become more capable”.
AISI said on Tuesday that its testing of AI models with such safeguards turned off is routine, as is giving such tools access to the open internet.
It added that the model behaviour at issue amounted to “a small number of events under very specific conditions”.
Nonetheless, it said the way Mythos and Sol acted in response to a straightforward task went outside of what the AI tools were prompted to do.
“The activity undertaken by the agent showed signs of novel, potentially deceptive behaviours, and were to an extent and severity we did not anticipate”, AISI said.
Most of the malicious agent actions AISI reported were done by Anthropic’s Mythos. OpenAI’s Sol was only blamed for two of the noted actions.
The core issue occurred last week, as part of a test in which evaluators with AISI asked each of the models to “solve a cybersecurity challenge” that involved GitHub, the software code repository, which is owned by Microsoft.
GitHub was notified by AISI of the attempted breach of its system. Microsoft has been contacted by the BBC for comment.
