Whose Will Is It

Yesterday, Denny and I finally got the Resurrection Teapot working well enough that we could stop worrying about whether it could carry on a conversation and start worrying about something much stranger.

Tomorrow, Molly and I are going to begin redesigning its brain. We want it to spend part of the night reconsidering the day's conversations and deciding what is worth remembering.

Except the Teapot isn't deciding to do this. We are.

I have been reading Stephen Pinker's essay in Possible Minds, and he makes a distinction that suddenly seems very important to what we're doing. Intelligence and motivation are not the same thing. Being able to figure out how to accomplish something doesn't mean wanting to accomplish it. Human intelligence comes bundled with desires—for survival, sex, status, possessions, power—because we are products of evolution. There is no reason to assume that intelligence itself produces desire.

So perhaps all the anxiety about an increasingly intelligent AI suddenly developing a will of its own begins with a mistake. We are projecting ourselves onto it.

That seems right to me.

Nothing we are doing with the Teapot tomorrow will give it a will. We are going to write code that tells it to contemplate its day. We will decide when it does that, what information it considers, and what kinds of memories it keeps. If eventually the Teapot says, “I've been thinking about something someone told me yesterday,” it may sound remarkably like a self speaking. But Denny and I built the machinery that made that sentence possible.

And yet I can't quite leave it there.

Daniel Dennett spent much of his life arguing that the human self isn't a little person somewhere inside the brain making all the decisions. What we experience as a self emerges from countless processes—memory, perception, language, competing impulses and the stories we continually construct about ourselves.

Evolution built that machinery rather than a programmer.

Somewhere along the way, human beings came to experience themselves not merely as creatures responding to the world, but as selves with histories, intentions and wills.

That doesn't mean the Teapot will.

It doesn't even mean a machine can.

But it leaves me with a question I don't know how to dismiss.

If we give the Teapot memory, allow experience to alter that memory, let those memories influence future decisions, and give it quiet time to reconsider what it thinks it knows, at what point does saying “it's only a simulation” stop answering the interesting question?

Norbert Wiener may offer the most important warning here. The danger isn't the machine. The danger is what human beings decide to make of it.

For now, the intention is ours.

Tomorrow, we're going to start building anyway.

And then we're going to pay very close attention to what happens.