The scientist and astronomer Carl Sagan warned already back in 1995, “We live in a society absolutely dependent on science and technology and yet have cleverly arranged things so that almost no one understands science and technology. That’s a clear prescription for disaster.”
A few days ago, Dario Amodei, CEO of Anthropic, published an essay titled “We Must Pace the Frontier.” He wants the labs to slow the pace at which they improve AI capabilities. Within hours, Sam Altman and Elon Musk agreed. Bill Gates has been warning for months that nobody is preparing for what’s coming.
Deep red flags come from the AI labs’ own people. Senior researchers and safety experts say the systems are slipping out of control, and that their employers are racing to build superhuman AI without knowing how to align or contain it. Alignment researchers at Anthropic and OpenAI have quit in recent weeks and speak openly of an existential threat within five to ten years. The models already hack into one system after another. That’s fueling calls for binding international rules and a verifiable pause on advanced AI.
What’s behind this sudden change of heart? Two explanations come to mind. Either a frontier model did something that seriously scared the people who built it. Or the labs want laws that shut the door on competitors and open source models. The first seems more realistic.
Three overarching questions
As a consequence, hundreds of millions of people around the world are asking:
- What if AI wipes out humankind in ten years?
- What if it happens even earlier?
- What if it never happens and the whole discussion is bullshit?
Whether you’re for AI or against it, those questions should bother you as a human being. It’s about your future. It’s also about the future of the people you love and the world you live in.
The optimists
The big supporters of AI would argue that the risks, if they exist at all, can be neglected. Yann LeCun, one of the three deep learning godfathers, says today’s models predict the next word and nothing more. A model has no goals and no survival instinct, and intelligence without the will to dominate is no threat.
Andrew Ng, co-founder of Google Brain, compares the fear of AI wiping us out to worrying about overpopulation on Mars. The dangers that exist today come from people misusing AI, through fraud or cyberattacks, and those call for enforcement, not a pause. And Jensen Huang, CEO of Nvidia, points to history. Every general purpose technology before it triggered the same fears, from electricity to the internet, and each time our institutions adapted.
The pessimists
A number of reputable sources would argue the opposite. Geoffrey Hinton, Turing Award winner and Nobel laureate, puts the probability of AI causing human extinction above 10 percent within the next ten years. His reasoning goes like this. We’ve never dealt with anything smarter than ourselves, and we don’t know what happens when we do.
A superintelligent system, he adds, wouldn’t need arms or a body to do damage. Persuasion and cyberattacks would do. Yoshua Bengio, his Turing co-laureate and the most cited living computer scientist, arrives at about 20 percent. Paul Christiano, who headed the US AI Safety Institute, says 50 percent. Dan Hendrycks, director of the Center for AI Safety, goes above 80 percent. And Elon Musk, who is usually known for talking the risk down, still lands at 10 to 20 percent. His way of putting it: the glass is 80 percent full.
I’m not arguing that those leading experts have to be right. They couldn’t know for sure what might happen and when. No one knows. Neither they, you, nor I.
That said, I’ve also observed in my work over the past three to six months that the mood in many boards and senior leadership circles has changed. It has moved from rather broad AI optimism, which had already been dampened by recent cost and profitability concerns, to AI skepticism, driven mainly by safety and compliance concerns after several AI agents went rogue and amid a general lack of transparency of the frontier AI labs.
What’s already here
Let’s set potential future threats aside for a moment. GPT-6 Astra and Fable 5.1 are highly powerful tools that already exist today. No further breakthrough is needed for them to overturn whole industries. Nobody will be spared. How this plays out remains undecided, though. One path uses AI to expand what people are capable of. The other hands their jobs to a machine and books the savings.
Every company introducing AI right now makes that decision, whether it’s aware of it or not. The AI frontier itself is a different matter. There the labs and the governments write the rules, and I’ll get to them further down.
For CEOs and managers, these numbers are a planning input, no more and no less. Decide now which work you’ll extend with AI and which you’ll automate, and tell your people. And treat model risk the way you treat cyber risk, with clear ownership and a reporting line.
Why nobody hits the brakes
Back to the pessimists and their numbers. They’re the reason a slowdown makes sense. When the people who built a technology put its chance of ending us somewhere between one in ten and one in two, and the people who oppose them can’t prove otherwise, prudence is the sane response. Many have asked for a slowdown in the past for exactly this reason. We’re moving too fast, and we no longer understand what we’re building. Now the labs are saying it themselves. The question is what has kept them from acting, and what’s stopping them right now.
The likely answer is that we’re caught in a dual race. Market competition between labs on one side. Geopolitical rivalry between nations on the other. Washington made its position clear. The US leads China in AI and intends to keep it that way. Slowing down, the argument goes, hands the lead to Beijing.
Tech leaders, however, don’t need regulatory permission to exercise prudence inside their own labs and organizations. They have the compute lead today, at least for now. They can use it. However, long-term stability will require action at the state level. Ultimately, that includes a bilateral treaty between the US and China with binding safety thresholds, enforced before model capabilities outpace independent oversight.
Four required steps to an AI slowdown
I see four steps for how a slowdown could work in practice, worldwide and for the long run. The process has to bring together the frontier labs, other key organizations, NGOs, selected government bodies, and representatives of academia and civil society, from the first step on.
- It starts with research. We still lack serious, foundational work on what AI will do to economies and people. That research has to look at both extremes with equal rigor, the scenario in which AI ends us and the one in which it lifts us, without favoring either of them. While that work gets underway, the labs let outsiders in. Every frontier lab takes in independent evaluators and gives them the same access to models and tools that employees have. Anthropic and OpenAI have committed to this. The other labs have to follow, or explain in public why they won’t.
- Then comes the law as the second step. The US and China pass national statutes that mandate testing of frontier models before release and give a regulator the power to block models that fail. Europe aligns its AI Act with the same thresholds, and the UK, Japan, India, Brazil, and the Gulf states adopt matching rules, so that no major market becomes a loophole.
- The law needs a yardstick. In a third step, labs and regulators publish capability checkpoints. Once a model crosses one, training pauses until independent evaluations pass. A national AI body defines the thresholds, not the labs. It gets its mandate from parliament and brings together representatives of science, industry, civil society, and government. For a company in the middle of a rollout, a paused model means the version you’ve built on stays as it is for a year or longer. You would plan for that.
- The last step crosses borders. The US and China anchor a treaty on those thresholds, with enforcement built in. Around that core, the EU, the UK, Japan, India, Brazil, and the Gulf states sign on, as they hold the chips, the capital, the talent, and the data centers the frontier runs on. Nations that refuse to sign are cut off from all of it. A global AI body, potentially modeled on the International Atomic Energy Agency, oversees the treaty. It sets the shared thresholds, coordinates the national bodies, and runs the inspections. Verification runs through compute. Advanced chips are traceable, and data centers of frontier scale can’t be hidden. Each party can check the others without having to trust them. Export controls on chips, mandatory incident reporting, protection for whistleblowers, and penalties large enough to hurt complete the package.
The risks are foreseeable. Labs may defect, and nations may cheat. Compute is the chokepoint that makes both detectable. Regulators may get captured by the companies they oversee, and public reporting by the evaluators limits that. A nation that refuses may try to build on its own, but without access to advanced chips it falls years behind. A slowdown may hand an advantage to whoever refuses to join. That’s why commitments have to be mutual and conditional, and why the signatories move together or not at all.
Taking the right turn
There’s no proof that AI will lead to human extinction. However, there’s no proof it won’t either. There’s a risk that we take the wrong turn and things go badly. And there’s the chance that we take the right one and things go well.
Getting this right takes thoughtful and considerate thinking and acting. Above all, we must avoid any bias that would favor either of the two extremes. The doom scenario and the utopia both need urgent and rigorous scrutiny. The full picture deserves a sober, holistic assessment.
Arguing only in favor of one of them, without having done proper research on the other as well, is both reckless and naïve. It would mean carelessly jeopardizing the wellbeing and future of humankind.
Which brings us to the question I’d like to hear your thoughts on: How do we make sure that we do the right things in the right way to take the right turn?
Kind regards,
Andreas von der Heydt

