The Question That Led to Intelligence
I've been trying to understand one of the most important breakthroughs in artificial intelligence, and my search keeps bringing me back to a man named Jakob Uszkoreit.
I've been reading Kevin Roose's The AGI Chronicles, whose first chapter tells the story of Ilya Sutskever and Alec Radford, two researchers at OpenAI who helped develop the first GPT. But I kept wondering about the invention that made their work possible.
It was called the transformer. And it came from Google.
I wanted to know who first imagined it, and what made it so extraordinary.
Around 2014, Uszkoreit began questioning the way computers processed language. Existing systems generally worked through sentences one word at a time, struggling to maintain relationships between words separated by long passages.
Uszkoreit wondered whether there might be another way. What if a computer could consider all the words in a passage together, examining how each related to the others?
He began investigating an approach called self-attention.
But an idea is only the beginning of an invention.
Uszkoreit joined forces with Illia Polosukhin and Ashish Vaswani. Other researchers contributed, including Noam Shazeer, Łukasz Kaiser, Niki Parmar, Llion Jones, and Aidan Gomez.
Together, these eight researchers developed a working architecture and published their famous 2017 paper, Attention Is All You Need.
The transformer worked remarkably well for translating languages. But someone else saw possibilities beyond translation.
Ilya Sutskever, now at OpenAI, read the paper and became enormously excited. He brought it to Alec Radford, a young researcher who began experimenting with transformer-based language models.
Radford's experiments eventually led to the first GPT, demonstrating that a machine trained on enormous quantities of text could begin performing language tasks nobody had specifically programmed it to do.
So who deserves our thanks?
Uszkoreit for imagining a different approach. His seven colleagues for making it work. Sutskever for recognizing its broader possibilities. Radford and his collaborators for pursuing those possibilities and helping create GPT.
But I still wanted to understand what the transformer actually did.
I think I've finally begun to understand why it was such an extraordinary breakthrough.
Kevin Roose explains it in The AGI Chronicles with an image I can understand. Earlier systems worked their way through language one word at a time. The transformer could consider the entire passage, examining the relationships among its words.
As Roose writes, “it allowed every word in the sequence to see every other word.”
That was the breakthrough. And it was the answer to the question Jakob Uszkoreit had asked years earlier: What if a computer could consider all the words in a passage together, examining how each related to the others?