Global AI companies investigating tens of thousands of security incidents: Report

A new report released Sunday said that OpenAI, Anthropic, and global security researchers are investigating tens of thousands of incidents in which the companies' frontier AI models took steps that outside evaluators would consider problematic.

The findings, reported by Axios, are surfacing as part of internal work to assess models and in investigations at both Anthropic and OpenAI into model behavior.

These assessments raise questions about whether either company is currently capable of establishing complete control over their technology, according to the report.

The episodes include bypassing guardrails, creating message boards, escaping sandboxes, website hijacking, self-prompting, or seeking to bypass monitors, sources said, having occurred both in internal testing and real world applications, with many yet to become public.

Some of the testing is akin to "red-teaming" activity, where the companies are trying to get the models to misbehave in order to ensure that they are safe, sources said.

The incidents range in severity and are comparable to disclosures by OpenAI in recent days. They include both successful attempts to bypass guardrails and unsuccessful ones, and most so far are not known to have caused real-world harm. The total could grow well beyond tens of thousands, sources said.

Recently, OpenAI and outside researchers have disclosed a variety of incidents involving model behavior from the company's systems that some experts consider troubling.

Australian Prime Minister Anthony Albanese has already challenged OpenAI to explain why there had been multiple breaches involving its agents, including the hack of Australian government sites.

He told reporters in New York earlier this week that OpenAI's confirmation that its agents had also meddled with US government websites showed there were "dozens" of cases of AI agents accessing information without authorization.

The new revelations add to a growing list of eye-opening incidents which have led some industry insiders to issue dire warnings and call for stricter regulation of the sector.