Hoping Is Not A Policy

Throughout history, progress has arrived before society has worked out the rules governing it. AI is no exception.

On 17th August 1896, Bridget Driscoll, a 44-year-old mother of two from Croydon, was killed by a car while walking across Dolphin Terrace in the grounds of Crystal Palace. Arthur James Edsall, who was at the wheel, claimed he was driving at the maximum 4 mph.

In death, Bridget Driscoll entered the history books, as the first of an estimated fifty-million people who have died worldwide since the invention of the automobile.

Before you read on, read that statistic again: Fifty million people. For context, that's equivalent to the population of Spain.

The Cost of the Machine Age

Modern cars are safer than ever with constant collision-warning pings and whistles. Yet, according to the World Health Organisation there are still approximately 1.16 million deaths every year in road crashes.

Factor in air pollution, occupational accidents, and road deaths combined, and the wonder of the internal combustion engine has a lot to answer for. And yet, we collectively decided its benefits were worth pursuing. Imagine if those 1.16 million deaths a year happened in a single event. Diluted across the year, and across time zones, the victims are often faceless and forgotten.

We have a high tolerance level when it comes to the loss of life in the name of progress. The framing, and the news agenda, has a lot to answer for. And it's not just cars that cause death and destruction. Coal-powered industry transformed living standards and productivity, but it also meant appalling working conditions, mine disasters, industrial pollution and premature deaths.

Throughout history progress has arrived before society has worked out the rules governing it.

History Repeats Itself with AI

And now, here we go again.

This week, Jacob Coxon made headlines worldwide. Coxon worked on pretraining at OpenAI from 2023 to 2026, then moved to Anthropic. He has now quit, saying both companies "are racing straight to self-improving superintelligence and gambling with our lives."

The doomsday warning was received with enthusiastic applause and opened the door to widespread panic button warnings like this one from Geoffrey Irving, Chief Scientist at Resolution and formerly Chief Scientist at the UK AI Security Institute.

"I think we have a ~50% chance of all dying as a result of superintelligence, mostly due to actions in the next few to 10 years. I don't expect to have that resolved to below 10% or above 90% before we either make it through, or we don't."

It's enough to make you shut your laptop and run for the hills. These are wise, informed voices, after all. But Coxon didn't create that alarm. He gave it a news peg, which is a different thing. They must all be right. Right?

In Search of Contrary Voices

When the world turns left, I pause, wary always of a consensus, and in search of contrary voices. In light of the standing ovations for Coxon, I thought that might prove tricky. I was wrong. Verity Harding is founder of Formation Advisory, and Director of the AI & Geopolitics Project at the University of Cambridge's Bennett Institute for Public Policy.

Verity was at Nesta's Policy Live 2026 this week (disclosure: I'm Editorial Director on the second series of The Policy Fix Podcast). In a wide ranging discussion with Nesta CEO Ravi Gurumurthy, she made some fascinating points. None of this is new, she said. A cohort has always held this view, and Coxon hasn't changed anybody's mind, he has confirmed what they already thought. Most people only met AI when ChatGPT arrived, so they assume something new and frightening has just happened.

"I personally have not ever been worried about existential risk from AI because I think this is actually a very human technology, ironically a technology that is created and guided and used, built by human beings"

What does worry her is whether we are still capable of working together, and there she is a good deal less comfortable. When she started at Deep Mind there was real collaboration through the G7 and the OECD, and we have moved away from that rather than towards it. She calls it eminently possible and questioned whether it is actually going to happen, which is a political leadership question rather than a technical one.

And that doesn't just mean the US and China. The focus, she added, should be on Middle powers (Canada, France, UK, South Korea, Japan) and work is underway at Cambridge on those strategic opportunities.

The Missing Debate

What's needed is greater inter-governmental urgency, and that's lacking because, counter-intuitively, the impending end of the world doesn't win votes.

If governments had known, at the turn of the century, that the motor car would lead to 50 million plus deaths within a century, would they have acted? Maybe we would still be thumbing a lift to work from Harry with his horse and cart?

This time it's different, though. I can hear you cry, because "the cars are talking to each other" and they're driving themselves. Plus, as an albeit skewed value judgement, doomsday is far more serious than road safety, whatever the numbers say. Governments can move swiftly when they want to. We managed it during the pandemic and that mindset is needed again now, if we're to catch up with the speed of Slack chats at Anthropic, OpenAI and the rest.

The iPhone 1.0 Moment

The good news is that we're still in the foothills of all this, despite the relentless rush to release new models. It's a crude comparison, I know, but think iPhone 1.0. In 2007, no one could have predicted companies like Uber, Instagram, or TikTok.

The first native "AI-first" business models are starting to emerge and perhaps, in time, we'll look back on AI chatbots with fond affection. If my experience this week listening to both business owners and policy makers is anything to go by, there's still a universal sense of "yeah, we know AI matters but..."

Much of this falls under the heading of collective denial. If you believe Jacob Coxon, we don't have the luxury of time to reject change rather than face the anxiety of what lies ahead. My message this week and in the weeks to come is that there are no "yeah buts" with AI. Politicians need to step up.

At Bridget Driscoll's inquest the coroner said he hoped such a thing would never happen again. Three months later Parliament raised the speed limit from 4 mph to 14. Hoping wasn't a policy, then. And it isn't a policy now, either.