Vladimir Vapnik — Co-inventor of support vector machines and VC (Vapnik-Chervonenkis) theory, foundational figure in statistical learning theory; worked at AT&T, NEC Labs, Facebook Research, and professor at Columbia University.
The gist
Lex Fridman talks with Vladimir Vapnik about the mathematical and philosophical foundations of learning and intelligence. Vapnik distinguishes instrumentalism (prediction) from realism (understanding God's law) and argues that math, not imagination or interpretation, reveals the ground truth of reality. He is sharply critical of deep learning, calling it 'fantasy' and 'interpretation,' and argues that optimal learning solutions lie on shallow networks per the Representer theorem. His central thesis is that there are two learning mechanisms (strong and weak convergence), and that intelligence lies in the still-unsolved problem of generating good predicates and invariants, the role he attributes to a great teacher.
Big reveals
- Vapnik claims there are two mechanisms of learning: strong convergence and weak (big) convergence, and that the latter allows use of predicates.
- He argues the Representer theorem shows the optimal learning solution lies on a shallow network, not deep learning.
- He defines the true goal of learning as creating an admissible set of functions with small VC dimension that contains good functions.
- Vapnik says he just finished an 'invariance story' he believes is the ultimate, complete learning theory with no other possible mechanisms.
- He speculates intelligence may exist partly outside us, citing how the same theorems get discovered simultaneously by multiple people.
- He poses the open challenge: match deep learning's 99.5% on NIST digit recognition using 100x less data by incorporating invariants.
Things worth remembering
- Vapnik's work has been cited over 170,000 times.
- Vapnik references Eugene Wigner's 1960 paper 'Unreasonable Effectiveness of Mathematics in the Natural Sciences.'
- He recounts how microscope inventor Leeuwenhoek described blood cells as 'a fight between queens and kings,' a wrong interpretation of a real observation.
- He uses the English proverb 'if it looks like a duck, swims like a duck, quacks like a duck' to explain how few good predicates replace huge data.
- He notes non-Euclidean geometry was discovered nearly simultaneously by Lobachevsky, Gauss, and Bolyai within about a ten-year span.
- Vapnik came to the United States in 1991; statistical learning theory was published in Russian monographs but unknown in America.
- When statistical learning theory was discovered ~20 years prior, only one person believed it: Dudley from MIT.
- Vapnik says he sees 'ground truth' in the clear structure of Bach's music, comparing it to axioms in geometry.
Recommended in this episode
Books, products and media the guest or host genuinely endorsed here — with the buy link.
Affiliate link — we may earn a commission at no extra cost to you.
The Second World War
Winston Churchill
“One of the greatest books is Churchill's book about the history of the Second World War. He starts in his book describing that in the old times when a war is over” — Vladimir Vapnik 00:23:17Find it on Amazon