Watch: The Latent History of AI Boom
An exploration of how the transition from RNNs to transformers and the discovery of double descent enabled the scaling of large language models like GPT.
Transcript
Back in 2017, artificial intelligence was dominated by relatively small neural networks. The golden rule of computer science was simple: do not make your models too big. The common wisdom was that if you built a model with too many parameters, it would just memorize the training data instead of actually learning how to solve new problems.
When OpenAI built the first Generative Pre-trained Transformer, or GPT-1, in 2018, it was tiny. It was sized just to fit on a single development computer. Google followed suit with its own model, BERT, keeping it the same size for comparison.
Then came a revolutionary concept called double descent. Traditional wisdom said that as a model grows, it gets better, then worse as it begins to overfit. But researchers decided to keep going anyway. They discovered that if you make a model massively larger, past a critical threshold, something incredible happens. The performance stops degrading and suddenly starts getting much, much better. The model actually begins to generalize.
This eureka moment changed everything. It proved that scaling worked. OpenAI immediately pushed the limits, eventually building GPT-3 with one hundred and seventy-five billion parameters. This unlocked brand-new, emergent abilities, triggering the massive explosion of AI models we see today, from ChatGPT and Gemini to major breakthroughs in computer vision and biology.