Your book is not called “If Anyone Builds It, There Is a 1 to 4 Percent Chance Everybody Dies.” You believe that the misalignment becomes catastrophic? Why do you think that is so likely? That’s just likely the straight -line extrapolation from it gets what it most wants. And the thing that it most wants is not us living happily ever after. So we’re dead. Like, it’s not that humans have been trying to cause side effects. When we build a skyscraper on top of where there used to be an ant heap, we’re not trying to kill the ants. We’re trying to build the skyscraper. But we are more dangerous to the small creatures of the Earth than we used to be, just because we’re doing larger things. Humans were not designed to care about ants. Humans were designed to care about humans. And for all of our flaws, and there are many, there are today more human beings than there have ever been at any point in history. So it’s not like the relationship between us and ants or us and oak trees. It’s more like the relationship between, I don’t know, us and us, or us and tools, or us and dogs, or something. Maybe the metaphors begin to break down. Why don’t you think in the back and forth of that relationship there’s the capacity to maintain a rough balance, not a balance where there’s never a problem, but a balance where there is not an extinction-level event from a super smart A.I. that deviously plots to conduct a strategy to destroy us? I mean, we’ve already observed some amount of slightly devious plotting in the existing systems. But leaving that aside, the more direct answer there is something like, one, the relationship between what you optimize for, that the training set you optimize over, and what the entity, the organism, the A.I. ends up wanting has been and will be weird and twisty. It’s not direct. It’s not like making a wish to a genie inside a fantasy story. And second, ending up slightly off is predictably enough to kill everyone.
