A Technical History of the Discoveries That Led to ChatGPT

The strange thing about the history of GPT is that the people building it did not begin with a reliable map. They had a destination—something approaching general intelligence—but no one knew whether such a thing was technically possible, much less how to build it. Artificial intelligence had already produced remarkable specialized machines. One program could play chess. Another could recognize photographs. Another could translate between languages. But each intelligence lived inside its own little enclosure. The more ambitious question was whether one machine could learn broadly enough to move among many kinds of problems without being rebuilt for each one.

Before Anyone Knew Whether It Would Work

OpenAI was founded in late 2015 around precisely that larger ambition. But the early work did not immediately resemble ChatGPT. Researchers trained agents to play games and control robots. One of the most important experiments involved Dota 2, a complicated multiplayer video game that demanded strategy, coordination and rapid decision-making. Jakub Pachocki and other researchers found that when they increased the amount of computing power devoted to training, performance kept improving. Double the computational resources, train again, and the agent became noticeably better. Similar techniques later helped OpenAI’s robotic hand learn increasingly sophisticated manipulation.

The important discovery was not Dota. It was scale. The researchers were beginning to suspect that artificial intelligence might improve not only through better algorithms but simply by giving learning systems more computation, more experience and more time.

At roughly the same period, another line of work was unfolding more quietly. Alec Radford trained a neural network on Amazon reviews. The task was extremely simple: predict what character came next. Then the system did something nobody had explicitly asked it to do. It learned sentiment. The model developed an internal numerical feature that tracked whether a review was positive or negative. OpenAI described this as a “sentiment neuron.” Somehow, in learning to predict sequences, the network had discovered a useful representation of meaning.

This is where the familiar description of modern language models as systems that “predict the next word” begins to feel both accurate and incomplete. Prediction really is the training objective. But predicting language extremely well requires discovering relationships inside language. Words are connected to objects, actions, feelings, people, places, histories and one another. A machine asked to predict enough language eventually benefits from representing some of those relationships internally. Prediction was the exercise; representation was one of the consequences.

The researchers themselves were still trying to understand what they were seeing. They had not begun with a finished theory of artificial general intelligence and then implemented it. Their experiments were producing abilities that theory then had to explain. That gap—between being able to make a system work and fully understanding why it works—will remain throughout this history.

On June 12, 2017, eight researchers at Google posted a paper titled Attention Is All You Need. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser and Illia Polosukhin introduced a new neural-network architecture called the Transformer.

Before Transformers, many language systems processed a sentence largely in sequence. Word after word moved through a recurrent network. These systems could work well, but long passages were difficult and training could be slow. The Transformer reorganized the problem. Its crucial mechanism was attention. When processing one part of a sentence, the network could give greater or lesser weight to other parts depending on their relevance.

Consider the sentence: “The violin would not fit inside the case because it was too large.” What does “it” refer to? A human immediately connects “it” with the violin. Attention helps a model learn relationships of that sort—not through a grammar rule specifying what “it” must mean, but through learned numerical relationships among tokens.

Transformers offered another enormous advantage: much of the computation could happen in parallel. That made the architecture unusually well suited to GPUs, which can perform vast numbers of mathematical operations simultaneously. The original Transformer paper emphasized that the architecture was both more parallelizable and faster to train than the dominant sequence models of the time.

Ilya Sutskever recognized its significance quickly. He had already worked on earlier techniques for sequence modeling and attention, so he could see that the Transformer solved problems he knew well. Greg Brockman later compared the moment to a method associated with physicist Richard Feynman: keep important unsolved problems alive in your mind, and when a new idea appears, ask whether it supplies a missing piece.

For OpenAI, the Transformer did.

Radford began experimenting with the architecture and later recalled making more progress in weeks than he had during the previous two years. The next breakthrough appeared on June 11, 2018, when OpenAI announced a system combining Transformers with large-scale unsupervised pre-training. The model could be trained broadly on language and then adapted to many different tasks. This was the first Generative Pre-trained Transformer—GPT.

The name contains the basic strategy. Generative means that it produces new sequences of language. Pre-trained means that it first learns broadly from large amounts of text rather than beginning each task from nothing. Transformer refers to the attention-based architecture that made that learning practical at scale.

Pre-training was a conceptual turning point. Andrej Karpathy later described the older approach as almost backwards. Imagine building an AI agent to book an airline ticket. One method would let the agent repeatedly click buttons, receive rewards and penalties, and eventually discover the correct sequence. But a language model could begin somewhere much further ahead. It might already have encountered descriptions of airplanes, calendars, websites, names, cities, credit cards and tickets. Instead of teaching every new agent the world from scratch, researchers could begin with something resembling a general-purpose linguistic brain and then adapt it to particular uses.

This was the true pivot.

Artificial intelligence had largely been organized around constructing many specialized intelligences. GPT suggested that one broadly trained linguistic intelligence might become the foundation from which many specialized abilities could grow.

The next step was almost embarrassingly simple. GPT worked, so what would happen if OpenAI made it much larger?

On February 14, 2019, OpenAI announced GPT-2, a 1.5-billion-parameter Transformer trained on millions of web pages. It could generate coherent paragraphs and perform rudimentary translation, summarization, question answering and reading comprehension without task-specific training.

The public did not simply receive unrestricted access to the full model. OpenAI was concerned that increasingly convincing generated text might be misused, so it initially released only smaller versions to researchers and developers. The complete 1.5-billion-parameter model came later. This is important when trying to imagine the period: GPT-2 was a major event inside artificial-intelligence research, but it was not yet something most ordinary people could sit down and use. The technology was advancing faster than its public visibility.

Meanwhile, OpenAI researchers were studying the relationship between size and capability much more systematically. In January 2020, Jared Kaplan, Sam McCandlish, Tom Henighan, Tom Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu and Dario Amodei published research on what became known as scaling laws. They found remarkably regular relationships among model size, dataset size, computation and performance.

Scale was no longer merely an interesting observation from experiments such as Dota. It was becoming something closer to an engineering principle. If a model’s performance improved predictably as more compute, data and parameters were added, then enormously expensive training runs could be justified before anyone knew exactly what new abilities would appear.

That changed the psychology of the field. Researchers no longer had to ask only, “What clever new algorithm should we invent?” They could also ask a much simpler and, in retrospect, astonishing question:

A few months later, the answer became GPT-3. In May 2020 OpenAI published Language Models are Few-Shot Learners. GPT-3 contained 175 billion parameters, more than one hundred times GPT-2. Its size was impressive. Its behavior was more important.

GPT-3 could perform unfamiliar tasks after seeing only a handful of examples supplied in ordinary text. This became known as few-shot learning.

Suppose the prompt contains:

Wordsworth → Romantic poet
Shelley → Romantic poet
Byron → Romantic poet
Darwin →

The model might complete the last line:

Darwin → Naturalist

No one explicitly told the model what task it was performing. It inferred from the first three examples that the pattern was: name a person, then identify what that person is known for. The examples themselves became the instruction.

This is called few-shot learning. Instead of explaining a task in words, you give the model a few examples of what you want, and it recognizes the pattern and continues it. The important discovery was that GPT could do this without being retrained for each new task. It could learn what you wanted from the prompt itself.

And the same principle can do much more than classify people. Give GPT several examples of a particular style of writing, and those examples can teach it how to continue in that style. That ability would eventually become one of the reasons a general language model could be adapted to so many different kinds of human conversation.

Yet in 2020 this was still mostly a research breakthrough. Ordinary people did not have a ChatGPT window waiting for them.

That would come later.

Two weeks after the GPT-3 paper, in June 2020, OpenAI announced another development that looked far less dramatic. It opened access to its models through an API.

An Application Programming Interface allows one piece of software to communicate with another. Instead of OpenAI deciding that GPT should become a medical chatbot, a game, a writing program or an educational tutor, outside developers could send text to GPT and receive generated text back.

This may have been one of the most consequential decisions in the entire story. GPT stopped being merely a model. It became infrastructure. Developers could begin experimenting with the same underlying intelligence for completely different purposes.

One early example was AI Dungeon, created by Nick Walton. It was an open-ended text adventure in which players could type almost anything and the AI would continue the story. Earlier versions used GPT-2. When OpenAI began testing the GPT-3 API, AI Dungeon became an unusually valuable real-world laboratory because thousands of players behaved in ways no controlled experiment could anticipate.

This period is important because it helps explain what “public access” actually meant before ChatGPT. A technologically adventurous developer might apply for API access. A programmer might build a product with GPT underneath it. A user might play an AI-powered game without understanding the research history behind it. Researchers could read the papers and experiment with released models. But there was still no simple universal doorway through which almost anyone could have an extended conversation with GPT.

The public was beginning to encounter the technology indirectly.

OpenAI, meanwhile, discovered that real users were doing more than generating revenue or testing products. They were exposing the model’s behavior. People asked questions no benchmark had anticipated. They found unexpected talents. They revealed offensive outputs, hallucinations and strange failures. They invented applications the researchers had never imagined. Greg Brockman later described interaction with users as a missing part of the research process.

That realization deserves emphasis. The development of useful artificial intelligence could no longer happen entirely inside the laboratory.

The machine needed contact with people.

The more people interacted with GPT-3, the clearer one of its limitations became. It had learned from an enormous amount of human language, and human language contains almost everything: wisdom and stupidity, truth and misinformation, generosity and cruelty. GPT could reproduce all of it.

The problem was no longer simply how to make the model more capable. Researchers also needed a way to influence how it responded.

One important technique became known as reinforcement learning from human feedback, or RLHF. Instead of training the model only on more text, humans were brought directly into the process. People compared possible answers and indicated which ones they preferred. Those judgments could then be used to train the model toward responses people found more useful.

A simple way to think about it is as an enormously well-read but socially inexperienced student. The student has read libraries but has never had an ordinary conversation. Now people begin responding:

That answer was helpful.
That one misunderstood the question.
This one was clearer.
That answer made something up.
Try again.

The model was learning something that additional reading alone could not provide: what kind of answer a human being actually wanted.

This work contributed to InstructGPT, which was designed to follow instructions more reliably and produce responses that better matched users’ intentions. The underlying language model had not suddenly acquired a conscience or human understanding. What had changed was the training process. Human preferences had become another source of information.

At roughly the same time, experiments such as WebGPT explored whether a language model could search for information, answer questions, and provide evidence that humans could evaluate. Researchers also began experimenting more directly with conversational models.

That was another important step.

Instead of simply asking a model to continue a piece of text, a person could ask a question, receive an answer, respond to it, clarify what was meant, and ask the model to try again.

Conversation was becoming the interface.

By 2022, several pieces that had developed separately were beginning to come together: large language models, instruction following, human feedback, and dialogue.

The technology was getting close to the form in which most of the world would finally encounter it.

By 2022, most of the pieces were already in place. Transformers had made large-scale language learning practical. Pre-training had given one model broad knowledge before it was adapted to particular tasks. Scaling had shown that more data, computation and parameters could produce surprisingly predictable improvements. Few-shot learning allowed a model to infer new tasks from examples placed directly in a prompt. Human feedback had made the models better at understanding what people actually wanted from them.

Then OpenAI put those developments behind an extraordinarily simple interface.

On November 30, 2022, it released ChatGPT as a free research preview. For most people, this was their first direct encounter with GPT. Earlier models had largely been experienced by researchers, developers and users of specialized applications. ChatGPT required no programming and no technical knowledge. Anyone could open a webpage, type a sentence in ordinary language and receive an answer.

That simple act changed the scale of the experiment. The history of GPT was no longer being written primarily inside research laboratories. Millions of people were now participating in it.

The scientific work had taken years. The invitation was almost absurdly simple:

Talk to it.

---

And When Did You?

This is where our history stops for now, at the moment ChatGPT became something the rest of us could encounter for ourselves.

What happened when you first tried it? What did you ask? What surprised you? Did you dismiss it at first, return later, or gradually find a place for it in your life?

We are interested less in another argument about whether AI is good or bad than in what actually happened when you encountered it.

Tell us your story in the comments below.

The next part of this history begins after November 30, 2022.