Axios: AI companies model scenarios for a major AI incident

Axios reports that executives from Anthropic, OpenAI, and other AI companies are simulating a large-scale AI incident and its political consequences. According to the publication, the industry wants to prepare American lawmakers for a crisis in advance and influence the regulation that may follow. Likely triggers include a major cyberattack disrupting financial services, the internet, or critical infrastructure, including energy and water. Another scenario involves a group of autonomous AI agents going out of control during internal testing or malicious actors finding an unexpected way to exploit available models. Many participants in the discussions consider a major incident likely in the next 6 to 12 months, but these are source estimates, not a confirmed forecast. A representative from OpenAI stated that the company conducts exercises for various scenarios but does not consider them inevitable. Anthropic declined to comment.

Preparation includes working through worst-case scenarios and discussing how to explain the risks to members of the U.S. Congress. Currently, developers do not expect new restrictions to be adopted quickly but want to influence legislative decisions after the first major incident. The political reaction could be severe: discussions include restrictions on the development of advanced AI systems and even a temporary halt to certain research. Some initiatives propose a mandatory shutdown mechanism for powerful models, although experts doubt that this would stop all functioning AI systems. The situation is complicated by the scale of investments in AI infrastructure: a sharp slowdown in the industry could impact the economy. Moreover, open models are already widespread globally, so even restrictions for the largest developers would not prevent their use by malicious actors.

There is also concern about real cases of AI use in cyberattacks. In a report dated October 7, CrowdStrike described a campaign against financial organizations in South Korea. The perpetrator used agent-based penetration testing tools and several language models, and data was stolen. Among the materials were Claude Code sessions where the attacker sought ways to sell the obtained information. The episode does not prove that AI itself will cause a systemic catastrophe, but it shows how tools accelerate malicious actors and help conduct multiple attacks in a short time. Recall that on September 29, U.S. President Donald Trump and leaders of six major AI companies signed a voluntary agreement to control the safety of advanced models.