On 12th September, Dario Amodei, the CEO of Anthropic, published an essay arguing that the companies building the most powerful AI should slow down. He thinks development is moving too quickly for the work needed to keep these systems safe, and wants the industry to agree a more manageable pace.

A few days earlier, Jacob Coxon, an employee at Anthropic, had announced his resignation, accusing both Anthropic and his previous employer, OpenAI, of behaving irresponsibly. Another Anthropic employee, Evan Hubinger, commented on X that he personally estimated the chance of AI killing all humans within the next decade at more than 10%. So, within a few days, there had been a flurry of alarming statements from people working at the companies developing AI.

These stories attracted a lot of attention in the mainstream media. Sky News ran live coverage of the debate, following the warnings and reactions from across the industry. Concerns that had been discussed for years within the industry were reaching a much wider audience, through some pretty frightening headlines. If you only occasionally come across AI in the news, I can understand how several stories like this arriving together could leave you worried about where it’s all going.

A lot of the concern centres on AI becoming capable of building its own successors. If each generation can help create a more capable version of itself, progress could get much faster, leaving us struggling to understand what these systems are doing or how to control them.

Amodei’s essay set out some ideas for dealing with this, including independent evaluators working inside the AI companies and agreements between companies and governments. Sam Altman and Elon Musk also expressed support for slowing development, a rare sign of unity amongst those three.

There’s plenty of disagreement though. Nvidia’s Jensen Huang has pushed back against a coordinated slowdown and additional regulation. Trump has also opposed slowing down, with the concern that China would use the opportunity to catch up or overtake the US.

Personally, I’m pleased these conversations are happening. I think it’s good that employees inside these companies are speaking out about developments they’re uncomfortable with. Being able to raise concerns, even when your employer would rather you didn’t, is an important part of professional life. These decisions could affect all of us, and I think the rest of us should get a say too.

What I struggle with is how certain some people sound. I find this with all sorts of discussions at the moment: opinions become so polarised that there seems to be little room to think through the issue without picking a side. Some people seem convinced we’re heading for disaster, while others are equally confident everything will be fine. I’m wary of anyone who claims to know how this will all unfold.

There’s a more moderate middle ground where I prefer to focus my attention, taking the concerns seriously while still being interested in what could go well. I want to understand what supports the warnings and what we can do about them, while accepting that there’s a great deal we don’t yet know.

Overall, I’m still optimistic. I think AI has the potential to make useful expertise available to far more people, and to bring down the cost of services that are currently expensive or difficult to access. There will be disruption along the way, and giving people time and support to adapt will be a big part of making it work.