The People Building AI Keep Telling Us What They're Building. We Should Probably Listen.
They're not whispering. They're not leaking. They're saying it on stage.
I’m watching a YouTube video at my desk on a Sunday afternoon when Microsoft’s AI chief says something that makes me put my coffee down.
“I think AI should best be understood as something like a new digital species.”
A new species. Just casually announced between slides.
Here’s the thing. I’m not a doomer. I work with AI every day. I build with it. I genuinely believe it’s the most useful technology most people will touch in their lifetimes. This newsletter is literally called AI the boring - I’m the last person who wants to write a fear piece.
But something’s been nagging at me lately, and I think it’s worth saying out loud.
The people building the most powerful AI systems on Earth keep telling us - publicly, clearly, on the record - that what they’re building might be genuinely dangerous. And then they go back to building it.
Not in leaked emails. Not in anonymous sources. On stage. In published essays. In interviews they agreed to do on camera.
I want to walk you through what they’ve actually said. Not my interpretation. Not some conspiracy thread. Just their words, attributed, in order. And I want you to notice something about the pattern.
What the builders are saying
Sam Altman, back in 2017… before ChatGPT, before OpenAI was a household name… wrote a blog post called “The Merge.” In it, he said this:
“We will be the first species ever to design our own descendants.”
Not a tool. Not a product. A descendant. He went further: “If two different species both want the same thing and only one can have it - in this case, to be the dominant species on the planet and beyond - they are going to have conflict.”
He’s describing an inevitable conflict between humanity and the thing his company is building. His proposed solution? “A merge is probably our best case scenario.”
Elon Musk - who co-founded OpenAI and has poured billions into AI - put it bluntly on Neil deGrasse Tyson’s StarTalk back in 2015:
“We’ll be like a pet Labrador if we’re lucky.”
If we’re lucky. That’s the good outcome, from a man spending his fortune on this technology.
Geoffrey Hinton won the Nobel Prize for the foundational work that made modern AI possible. Then he quit his prestigious job at Google so he could speak freely. Here’s what he said in a Q&A with the safety research group METR:
“I actually think the risk is more than 50%, of the existential threat.”
More than a coin flip. From the man they call the godfather of AI.
Dario Amodei, CEO of Anthropic… one of the leading AI labs, has put the odds of catastrophic AI outcomes at 25%.
And Jack Clark, Anthropic’s co-founder, wrote this in his widely-read newsletter last year:
“Make no mistake: what we are dealing with is a real and mysterious creature. And like all the best fairy tales, the creature is of our own creation.”
A co-founder of one of the companies building the most advanced AI on Earth is telling you the thing they’re making is a creature… and he’s worried about it.
And then there’s Sundar Pichai, CEO of Google. He told Lex Fridman:
“I think the underlying risk is actually pretty high, but I have a lot of faith in humanity kind of rising up to meet that moment.”
In other words: the risk is high, but hopefully humanity will rally to stop what his company is doing. Meanwhile, Google, Amazon, Microsoft, and Meta have been lobbying for a 10-year moratorium on US states regulating AI.
You can’t make this stuff up.
Here’s what’s actually going on here
I keep coming back to a number. In 2023, researchers surveyed nearly 2,800 AI scientists who’d published in the field’s top conferences and journals - NeurIPS, ICML, ICLR, the big ones. They asked them to estimate the probability of AI causing human extinction or permanent, severe disempowerment of the human species.
The mean response? 16%. One in six. Russian roulette odds.
Over half of the researchers surveyed gave at least a 10% chance of an outcome that bad.
There’s something quietly profound about this that I keep thinking about. These aren’t whistleblowers. They’re not being brave by speaking up. They’re just... saying things. Openly. At conferences that livestream to thousands of people.
Ilya Sutskever, OpenAI’s former chief scientist, told The Guardian he thought “it’s pretty likely the entire surface of the Earth will be covered with solar panels and data centres.”
Richard Sutton - winner of the Turing Award, the Nobel Prize of computer science - gave a talk in 2023 titled “AI Succession.” He quoted roboticist Hans Moravec approvingly from the stage: “They would displace us from existence. It behooves us to give them every advantage and to bow out when we can no longer contribute.”
He didn’t get shouted off the stage. He got applause.
And Geoffrey Hinton - the same man who quit Google to warn us - was asked at MIT whether he’d support a superintelligent AI replacing humanity with something “objectively better in terms of consciousness.” His answer? “I’m actually for it, but I think it would be wiser for me to say I am against it.”
That line lands differently when you remember he’s the man who made the technology that’s getting us there.
The pattern
Here’s what I want you to notice. None of this is secret. None of it is hidden. None of it requires a FOIA request or a leaked Slack message.
In 2023, over 350 of the world’s leading AI researchers… including Hinton, Altman, Amodei, and two other Turing Award winners… signed a single-sentence open letter: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”
One sentence. That’s all they needed.
These people are telling us… at conferences, in books, in interviews, in blog posts they publish under their real names, exactly what they think they’re building and exactly how dangerous they believe it is.
And they’re still building it.
Not because they’re evil. Mostly because of game theory… if they stop, someone else won’t. Dario Amodei has said as much. The race continues because nobody can afford to be the one who stops.
But the thing that gets me… the thing I can’t quite shake on a Sunday afternoon… is the gap between what they say and what we hear.
They say “new digital species” and we hear “better chatbot.” They say “our descendants, not our tools” and we hear “cool, it can write my emails now.” They say “the risk is actually pretty high” and we scroll past it because the next headline is about a new image generator.
I don’t think we need to panic. I don’t think we need to become doomers. I don’t think we need to stop using AI… I won’t be.
But I think we should probably start listening to what these people are actually saying. Not the marketing copy. Not the product launches. The other thing. The thing they say at conferences when they think only insiders are paying attention.
They’re not whispering. That’s what makes it so strange. They’re saying it right out in the open.
We’re just not listening yet.
Cheers, Jagger