Ilya Sutskever — Co-founder and research director of OpenAI; previously a research scientist at Google Brain and a student of Geoffrey Hinton at Toronto; a foundational figure in modern deep learning.
The gist
In this MIT AGI lecture hosted by Lex Fridman, Ilya Sutskever lays out a theoretical foundation for why deep learning works, arguing that backpropagation solves the profound problem of circuit search. He surveys reinforcement learning fundamentals, then dives into OpenAI's work on meta-learning, including hindsight experience replay, sim-to-real transfer via domain randomization, and learning hierarchies of actions. A large portion focuses on self-play, where agents create their own escalating challenges, illustrated by OpenAI's Dota 2 bots and wrestling humanoids. He closes with speculation on societies of agents developing language and social skills, and the technical and political problem of conveying goals safely to systems likely to become smarter than humans.
Big reveals
- Sutskever frames deep learning's success around a mathematical fact: finding the shortest program that fits your data gives the best possible generalization.
- He calls backpropagation finding the best small circuit 'the miraculous fact on which the rest of AI stands.'
- OpenAI's Dota bots went from playing totally randomly to world-champion level over roughly five months.
- Self-play lets you 'turn compute into data,' which Sutskever predicts will become extremely important as neural net processors get faster.
- He argues simply scaling up existing language models on larger, deeper architectures will 'go surprisingly far.'
- He criticizes the practice of freezing models after training, saying the training process is where 'the magic really happens' and should be used at test time.
- Sutskever states it is 'a lot more likely than not' that the agents we train will eventually be dramatically smarter than us.
Things worth remembering
- You can learn to sort n-bit numbers using a modestly sized neural network with just two hidden layers, despite sorting normally requiring log n parallel steps.
- Sutskever claims there is only one true reward in life: existence or nonexistence, and everything else is a corollary.
- He recounts hearing that simulating friction may be NP-complete, which is why simulators never perfectly match reality.
- TD-Gammon, from 1992, used two neural networks playing backgammon against each other and beat the world champion 26 years before this talk.
- TD-Gammon discovered new backgammon strategies that top human players had never noticed and that proved to be better.
- Sutskever theorizes the human brain grew rapidly because social standing in the tribe, not predators, became most important for survival.
- He cites convergent evolution between social apes and social birds despite very different brain structures and ancient evolutionary divergence.
- Teaching a simulated leg to do backflips via human preference clicks took about 500 clicks from human annotators.
- A pro DotA player imitated a strategy the bot used and was then able to defeat a better human pro.