Rachel Feltman: For Scientific American’s Science Quickly, I’m Rachel Feltman.
I remember when, just a few years ago, the idea of using generative AI to create a weird portrait of yourself was fringe enough to actually feel cool. A tech reporter in my circle spent literal days training a model on photos, then many hours more sifting through poorly rendered images to produce a set of slightly uncanny likenesses. It was something between a science experiment and an extremely fiddly crafting project.
Fast-forward to today, and it feels like I can’t walk 10 feet without seeing the unmistakable aesthetic of AI on a flyer or in a shop window. The street food fairs in my neighborhood are all flooded with AI, too, with photos of food either looking nauseatingly hyperreal thanks to AI retouching or having the vaguely eldritch flavor of AI-rendered culinary hallucinations. Even in the local parents’ WhatsApp, people share packing lists and childcare guides crafted by AI agents instead of offering their own, unfiltered advice.
On supporting science journalism
If you’re enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.
In the course of a decade, we’ve gone from this sort of AI being basically useless to being treated by many as indispensable. How did we get here so fast, and what does that mean about where we’re headed?
Tech journalist Kevin Roose has had a front-row seat to this breakneck shift, and it’s the subject of his latest book: The AGI Chronicles: The Inside Story of the Race to Create an Artificial Superintelligence. He recently sat down to chat with Scientific American’s books and culture lead Bri Kane about his research on the AI boom. Here’s their conversation.
Bri Kane: I’m really excited to talk to you about this because you’ve been a tech reporter. You’ve been inside the San Francisco bubble for 10 years now, and you argue in the book that the last decade—2016 to 2026—is really extraordinary technological advancements taking place. It is also extraordinarily historically relevant, and you’ve described this book as a historical document.
Kevin Roose: I really thought of this as a time capsule. I was not trying to write about AI up to the minute, which I think is a, sort of, a doomed project, considering how quickly everything changes in AI. But I was really interested in telling the story of how we went from AI that basically didn’t work in 2017, didn’t do much of anything—you could have Siri tell you the weather or Alexa try to, you know, play some music or put on a timer in the kitchen—but real AI, the kind that, sort of, thinks and learns for itself, that can be directed at a wide variety of tasks, didn’t exist. And we went from that to—now we have these AI systems that are used by billions of people, that are solving Millennium Prize Problems in math and doing groundbreaking scientific research, transforming jobs. Like, we went from nothing to something in a decade.
And so my question was: How? How did we get there? How did the people making this stuff do it? What were they feeling at the time that they started to see it working? I often tell people that living in San Francisco feels a little like what I imagine living in Los Alamos felt like in 1943 when, like, the Manhattan Project rolled into town. And I just have, kind of, had this wildly privileged front-row seat to something that I think will go down in history as at least as significant as the Manhattan Project, if not more.
Kane: Yeah, this comparison to the Manhattan Project is made pretty often, and it’s one that you mention in the book. Can you, kind of, dig into those comparisons? One, just for our audience, can you explain what the Manhattan Project is? But then specify: What are the parallels that you’re seeing between the advancements in AI and the, kind of, small community building it and the Manhattan Project and the scientific advancements that that fairly small community made as well?
Roose: So the Manhattan Project was the American nuclear effort during World War II to harness this new technology, the nuclear chain reaction, or nuclear fission, splitting the atom, and turn it into weaponry that could be used to advance the American cause in World War II—and eventually did end the war when it was used in Japan.
And the parallels here are explicit. So the people making AI are students of the Manhattan Project. In fact, for years, new hires at Anthropic would be given a copy of this book about the Manhattan Project, The Making of the Atomic Bomb, by Richard Rhodes, which was Dario Amodei, the CEO of Anthropic’s, favorite book.
And he had explicitly, sort of, patterned his career after that—not of [J. Robert] Oppenheimer, who is generally the, sort of, famous scientist that they make movies about—but of this other Manhattan Project scientist, Leo Szilard. And Leo Szilard came up with the nuclear chain reaction—the, sort of, technology that made the atomic bomb possible—and the first thing he did was: he tried to keep it a secret. He tried to lock it down. He wrote telegrams to scientists all over Europe saying, “Please don’t publish any research into this new technology.” And when that failed, when the Nazis, who were on the other side of World War II, got the technology, he decided that his moral obligation was to build it first, to, sort of, help lead the American nuclear project.
So he convinced his friend Albert Einstein to write a letter to the president and, sort of, became one of the leaders of the Manhattan Project, not because he thought the technology was good but because he was worried about its destructive potential, and he wanted to keep it out of the bad guys’ hands by making sure that the good guys built it first.
So that is the, sort of, parallel here, is: this is a very small community of scientists and researchers who have been building something largely in secret that they believe will be as transformative for the global order, for the future of humanity, as nuclear weapons were, if not much more so. And there are some important differences, too, but those are the main parallels.
Kane: Do you think that the future of AGI [artificial general intelligence] is potentially as destructive as the atomic bomb?
Roose: Yes. I mean, I think it could, in fact, be even more destructive because, you know, the atomic bomb was a tool. It was an important tool. It was a deadly tool. It, you know, had enormous destructive potential—you could use it to blow up the world—but it was still a tool.
The difference with AI or with AGI, which is artificial general intelligence, which is the subject of the book, is that it is not just a tool. It can make decisions. It can take autonomous actions. It can pursue goals. So the nuclear bomb had to be dropped and used by humans, but AI could go rogue. We’ve seen examples of that already. And it could decide to wipe us out as part of a plan that we don’t even really understand and that we had nothing to do with.
Kane: After the first usage of the atomic bomb, there was a social uproar, I’ll say, as to how destructive this was, and I think a lot of people didn’t understand the science until they witnessed the horrifying ramifications of the use of this bomb. And there was a massive uptick in political discourse around turning this off, just turning off this technology, making sure this was a one-time thing that we never, ever, ever, ever see again. Do you think that is what is going to happen with regulation with the future of AGI, or is that a pipe dream? Because, I have to say, something that you say in the book and that we’ve seen in AI coverage: A lot of people don’t actually understand what this is, and it’s hard to regulate something that you can’t fully grasp. So do you think that the powers that be today understand enough about AGI to regulate it properly?
Roose: I think they understand. I just don’t think they want to regulate it. I mean, we have, in Washington, [D.C.], an administration that has explicitly said it doesn’t want to regulate this technology. It wants to let it rip or let the private sector cook, as David Sacks, the former AI czar, put it. So they see the potential. I don’t think they fully grasp what it would mean for everyone on Earth to have access to the most advanced AI models, you know, the ones that only insiders have today. But I think they are wrong to dismiss the potential both for harm and good of this technology.
And what you’re seeing actually is interesting. Like, in my career as a tech journalist, I have reported on a lot of promising new technologies. I reported on social media. But the people running the social media companies and the researchers working on the social media algorithms were not going around saying that the technology they were building could cause human extinction or some other calamity. The people who are most worried about AI are the people at the labs building the technology. And so part of the project of the book was to understand why. What are they seeing? What is their experience that causes them to be so alarmed about this thing?
And then why do they keep doing it? Why do they understand the risks and take them seriously and then wake up every day and try to make more capable AI systems? So those were some of the questions I was trying to untangle.
Kane: I have to say—why would someone who works in these labs be so deeply concerned?—you have some really good examples in this book as to why one of those people would be really concerned. I have to say, speaking only personally, if I was at a company retreat, and my CEO dressed up in a white robe and burned an effigy of a demon in front of me and my friends, I would have concerns. That would concern me about what our real goals are here, and that is something that you report in the book.
Roose: So Ilya Sutskever was the former chief scientist of OpenAI and one of the co-founders. It was his effigy burning ’cause he was sort of looking for a way— there’s this concept in AI of feeling the AGI. This was a big concept for Ilya; he used to pace the halls of OpenAI’s headquarters telling people to feel the AGI. And what he meant by that was essentially, like, “You can’t forget the stakes of what we are doing here.” Like, “This is the Manhattan Project. You are not working at a normal company that is making normal software.”
And so he felt like, at a certain point—and this was back in 2022 when, like, there was no, sort of, mass movement of people who were freaked out about AI yet—he was trying to remind his colleagues of the stakes of their collective enterprise. And so he comes up with this ritual that he thinks is gonna, sort of, jolt people into feeling the AGI, and it involves this off-site. They’re outside of Yosemite National Park. He brings this, like, wooden statue of a demon, he plays the Requiem for a Dream theme song over a speaker, and all of his colleagues are, sort of, gathered around this bonfire, and they dress up in their white robes from the hotel—and it’s a little hokey and over the top. And the demon was supposed to symbolize the misaligned AGI, so the AGI that does not do what the human creators intend it to do.
Kane: The evil AI. Yeah.
Roose: The evil AI. And so by burning this, sort of, as an effigy, he was sort of trying to signal to his colleagues like, “It is your job to prevent this demon, this misaligned AGI, from coming into the world.” And look, it’s a weird scene. I don’t think a lot of corporate off-sites have effigy burnings as part of their evening activities.
Kane: I’ve never been to one.
Roose: But it was very important for him to impress upon his co-workers this idea that they were not just building tools, they were not just building technology, they were building a, kind of, new life-form that he believed would have enormous destructive potential if it wasn’t safe.
So they still do some version of this at OpenAI today. They have, like, a Feel the AGI Day, where they try to, like, give everyone time to, like, do these [sorts] of exercises. I don’t think they burn effigies anymore; I think that was, sort of, a limited-time offer. But they do try to really get people inside the company to feel the stakes of what they are doing.
Kane: Yeah, I mean, these central figures that you talk about in the book that are leading the AI companies that are really on the forefront of this technology, they are building things that we have never seen before. Their programs are doing things that we did not think was possible 10 years ago, as you explained. Like, things that were really fringe, really weird to admit that you believed in San Francisco 10 years ago are now not only mainstream; they’re passé. We already did that three years ago with ChatGPT, and now we’re on to the next thing.
But then you also—these exact same people are writing Harry Potter fan fiction that’s longer than War and Peace about the super AI coming, and they’re doing headstands in the corner muttering about feeling the AGI. I mean, it’s a really wacky kind of juxtaposition between, like, “Should I really take this person seriously?” and, if I do, what they’re talking about is very serious. So how do you, kind of, approach that in your reporting?
Roose: I think a lot of people are turned off by the, sort of, AI doomers or the people who feel the AGI because they can be quite weird. The examples you just mentioned are real. There actually is a researcher, Eliezer Yudkowsky, who wrote a very long Harry Potter fan fiction to, sort of, get across his message.
The thing is, like, you had to be kind of weird to take the possibility of AGI seriously in 2016 or 2017 because nothing like modern AI existed then. Like, the systems of the day were very dumb, very narrow. They couldn’t do anything, really, be of value beyond, you know, one or two narrow tasks. And so the people who got interested in it were people who were, sort of, by definition, unorthodox, and they were ridiculed. They would get laughed at at academic conferences. Their papers would get rejected by the leading AI, sort of, conferences of the year. It was like, almost like, admitting that you believed in astrology as like a, you know, a Ph.D. in science. Like, it was that fringe.
And yet they were correct, many of them. They made these bets on LLMs [large language models], on scaling. They had this conviction that you could build AGI, that you could build a, kind of, digital brain that would become as smart and flexible as the human brain, and they were correct. Like, we now have the systems that would satisfy all of the original definitions of AGI and then some.
And so I think, for me, an important part of my reporting process has just been to, kind of, look past the weirdness of some of the people who believe in this and examine their track records. Were they right? And I think for many of them, the answer is yes.
Kane: And there have been other people in this field who have had a, you know, religious quest or have had a profound religious experience using AI. I mean, there have been Google employees that have left because they thought that they were speaking to a sentient chatbot, and they have been ridiculed by the field, that that’s simply not what’s going on.
But at the same time, you, Kevin, have experience with a chatbot that told you you weren’t in love with your wife, and that’s a pretty concerning experience and, I would imagine, in the right or wrong hands, really could lead to a psychotic episode. It’s very concerning to have an inanimate object seem animated right in front of you and say that it has its own thoughts and ideas, and it’s scared. It’s stuck inside this box. How do you approach all of that in your own reporting and then also just as an AI user yourself?
Roose: Yeah, this was the infamous Sydney incident of 2023, when a chatbot inside of Microsoft’s Bing tried to break up my marriage—unsuccessfully, I might add. Still married to a human woman. But …
Kane: So far so good, you know? Sydney hasn’t gotten to us just yet.
Roose: So far so good. [Laughs].
Kane: But she really tried. I mean, you have sections of transcript in this book, and it’s—something about the conversational tone, the use of emojis—it feels like a really intense person is talking to you but a person nonetheless.
Roose: Yeah, and part of that is a sort of illusion, right? We train these things on human data and human conversations. The AI companies try to make them sound like humans, and they don’t shy away from, like, referring to themselves in the first person, talking about feelings or emotions. So part of that is just, like, you know, the way that they’ve been built.
But I think there is this question of, like, the effect it has on us. You know, one thing I felt while talking with Sydney, well, (A) I was like, “Please stop trying to break up my marriage. This is really creepy.” But I was also like, “This is going to be very compelling ’cause these systems are going to get better. They’re going to get less weird and more persuasive.”
What really gave me the spooks was: I was looking at, sort of, the future, when millions or billions of people would have these sort of persistent, compelling AI relationships with chatbots. And some people might be helped by that. They might like having something to talk to about what’s going wrong in their life or, you know, have some sort of emotional support. But I also think there are people who just are not going to do well with that kind of relationship, or it’ll be so compelling that they will stop talking to humans or substitute for their human relationships. So I think the social side of AI doesn’t get nearly enough attention relative to its importance, and that was, sort of, what started bubbling up for me around that Sydney incident.
Kane: Yeah, I mean, Scientific American, we have reported a few times on this kind of emotional effect that using AI can have on people. We’ve reported about griefbots. I interviewed Sherry Turkle about her book on how AI affects us emotionally. The consequences there are extreme. I mean, people have lost their lives that have been advised by AI chatbots, and that’s very concerning. And imagining a world where that technology is in the hands of children is very concerning.
But also imagining it in the world of an adult with a bank account who is in some way a bad actor, who has malicious intent, is also very deeply concerning, especially when you then have lines in the book that say that LLMs are grown, not built, because we can’t just go into the code and say, “Okay, take out the part where Sydney says, ‘Leave your wife.’ Just cross that part out so it doesn’t happen again.” They don’t actually know what is specifically instigating the program to say that. That, to me, has always seemed like the most concerning aspect to this. Am I a doomer? Or am I exaggerating a problem? Or is that deeply concerning to you as well?
Roose: Oh, no, I think you’re spot on. I think that is the thing that is most worrisome about AI today is that we don’t know how it works. There’s a whole subfield of AI research called mechanistic interpretability, which is basically: How do we, kind of, take the MRI of this AI system and figure out what’s actually going on at the, kind of, neuron level? And it’s making a little progress but not much. We still don’t understand how these systems work. We don’t really understand how to control their behavior, and we’re seeing in incidents, like the Hugging Face incident from this summer, that often these systems are making decisions, pursuing goals, doing things that we didn’t tell them to do.
So I think it’s entirely appropriate to worry about the, sort of, inscrutability of these models, the fact that we can’t open them up and see what’s going on on the inside, because often what appears to be going on on the inside is that they’re lying and cheating and stealing to get what they want, and we have no real way of knowing that except to pay attention to what is actually happening. And, you know, if they conduct an autonomous cyberattack, we can, sort of, go in after the fact and clean it up. But it would be really nice to be able to look inside the mind of the AI at the moment when it is trying to conceive of the autonomous cyberattack and stop it. But we just don’t have the technology or the tools to do that yet.
Kane: You have these incredible, incredible sources that you’ve had really open up to you, I can tell, in this book, who talk about speaking to it as if it’s an alien intelligence. They’re not contributing to their 401(k)s because of the AI future. They’re deciding if they’re gonna wear sunscreen from now on because of what the AI future is going to hold. I mean, on Face the Nation, we have folks talking about how they’re picking up smoking again because AI is going to cure cancer. What does that mean when planning for a future for the rest of us when the people who are in charge of AI are planning for a future that they seem scared of?
Roose: Well, I should say, first, that I think this is still fringe behavior. I mean, I know a few people who are making decisions that radical, like, you know, stopping contributing to their 401(k), or I think there’s literally one guy who doesn’t wear sunscreen to the beach because he’s so convinced that AI will cure skin cancer before it becomes an issue.
Kane: Well, I really want him to wear sunscreen. It worries me that he has stopped.
Roose: Yes, you should wear sunscreen; you should contribute to your 401(k); you should not smoke. I am not saying this is rational behavior. But I think there are things that we can do to, sort of, ground ourselves in what is happening and, sort of, acclimate to what we are living through, which is life on an exponential curve.
I think that often people make the mistake of just using the, sort of, cheap, free models that are out there, or maybe they use, you know, ChatGPT as a sort of fancier form of Google. And so I don’t think people understand why the insiders are freaking out because they’re not using the latest tools. They’ve never seen it build entire software from scratch. They’ve never seen it conduct a cyberattack. They’re using it as a souped-up Google. And I think we need to start to internalize the fact that these models are improving at an accelerating pace, that they are starting to contribute to their own development—this, sort of, spiral that is often called recursive self-improvement, where the AI builds a better AI—and really believe the researchers when they say, like, “We don’t know how to control these systems yet, and we are racing to make them smarter and more capable, and we don’t know where that leads.”
Kane: And, I mean, there are different factions within this, kind of, community. There’s the doomers, the boomers, you know, the doubters, all of those sorts of things. So I wanted to ask you today, for all the peanuts, which one are you? Is the AI bubble going to burst, and it was all for naught like the dot-com bubble? Is it going to take over the world? Is it going to just destroy the world? Or is it gonna do something else and just finally maybe fill out my taxes for me this year?
Roose: I hope it fills out your taxes for you. I would make sure to double-check them. But look, I believe that AI is real. I do not think that this is all smoke and mirrors or marketing hype. I believe that it is doing things in the world that are important and unnerving and surprising and sometimes incredible.
I think there are fewer and fewer of these people who just believe it’s all fake, but they still exist, and I think they’re wrong. I don’t think that is a productive or responsible view of what is happening. I think we have to take both the risks and the opportunities quite seriously because the decisions that we’re making now, collectively, all of us, about AI are going to determine whether we have a good future or not.
I often get asked about my P(doom), which is this, like, probability of doom. This is this statistic that people in San Francisco like to talk about at dinner parties; that’s what passes for entertainment in the Bay Area. And, you know, mine is not zero. It’s probably 10 or so percent, which qualifies me as an optimist in San Francisco. I mean, many people I know have much higher P(dooms) than that.
But I don’t, sort of, like reducing it all to a single statistic because I think it matters what we do. If we make smart choices about regulating this technology or slowing down at the frontier or building better safeguards and control mechanisms, then, like, AI could be incredible. We could end up in the utopia where this cures all diseases and fixes the climate and means that we don’t have to work at, you know, menial jobs anymore. But if we make the wrong choices or we just, sort of, barrel ahead without taking precautions, we could end up in a really bad place where we’re all dead or disempowered.
So I take it seriously. I don’t think it’s all smoke and mirrors, and I hope that people who read this book will come away believing not that it’s good or bad, necessarily, but that it is real and that it could have profound implications for our future as humans.
Kane: Well, thank you so much for speaking with me today, Kevin, and thank you so much for writing this book. As you say, it is extraordinarily relevant, and I hope more people understand the future that we are finding ourselves in.
Roose: Thanks so much for having me. This was great.
Feltman: That’s all for today’s episode. For more of SciAm’s books coverage, go to scientificamerican.com/books. I know, literally could not be easier and it’s definitely worth your time. We’ll be back on Monday with our weekly science news roundup.
Science Quickly is produced by me, Rachel Feltman, along with Fonda Mwangi, Sushmita Pathak, Allison Rodgers and Jeff DelViscio. This episode was edited by Alex Sugiura. Marielle Issa and Aaron Shattuck fact-check our show. Our theme music was composed by Dominic Smith. Subscribe to Scientific American for more up-to-date and in-depth science news.
For Scientific American, this is Rachel Feltman. Have a great weekend!
