Jay McClelland — Cognitive scientist at Stanford and a seminal figure in neural network research, co-author of the Parallel Distributed Processing books with David Rumelhart that helped lay the groundwork for the modern deep learning revolution.
The gist
McClelland traces his lifelong conviction that thought is fundamentally biological, rejecting the Cartesian split between body and mind. He recounts the 1970s-80s San Diego research group with David Rumelhart and Jeff Hinton where connectionism, the interactive activation model, and backpropagation took shape. He explains emergence, distributed representation, and how degradation of these representations explains semantic dementia, the very condition that took Rumelhart. The conversation ranges into mathematical cognition, the role of intuition versus logic, the expert blind spot, and how meaning is something humans make rather than discover.
Big reveals
- McClelland describes a 1977 'road to Damascus' moment realizing that thinking about the mind as a neural network would answer his questions.
- He reveals David Rumelhart suffered from semantic dementia, the exact distributed-representation breakdown McClelland was studying scientifically.
- Jeff Hinton told the group to stop seeking biological rules and instead adjust connection weights to solve a defined problem, the seed of backpropagation.
- Hinton initially bet on his Boltzmann machine over Rumelhart's generalized delta rule, but backprop won out.
- McClelland's contrarian claim that natural numbers are a cultural construction, not innate, now accepted by most cognitive scientists.
- Formally trained academics like Chomsky have intuitions that diverge sharply from ordinary people, suggesting expertise distorts introspection.
- McClelland says his graded view of mind makes the discreteness of death less apparent to him than to most people.
- A journal editor once told him to 'leave the theorizing to the theorists,' a label he refused to accept.
Things worth remembering
- Descartes built his mechanistic theory of animals after a hydraulic statue garden at Versailles moved when he stepped on stones.
- Huxley performed a public dissection to prove chimpanzees do have a hippocampus, countering claims of human uniqueness.
- 'Parallel' refers to each neuron being an autonomous computational unit, with billions working simultaneously instead of one central processor.
- 'Good old-fashioned AI' got its name in the late 70s because it was already from the 60s and hadn't panned out.
- In semantic dementia a patient calls all large animals horses, all small ones cats, and middle-sized ones dogs.
- Backpropagation extends the delta rule invented by Stanford engineer Bernard Widrow and his collaborator Hoff.
- Hinton draws pictures and shapes ravines with his hands rather than writing equations in lab meetings.
- A park bench quotation McClelland found: 'It is by logic that we prove but by intuition that we discover.'
- McClelland describes the 'Mozart effect' as the synergy of intense early immersion combined with innate resonance.
- The episode closes with Hinton's words that real breakthroughs come from curiosity-driven research.
Recommended in this episode
Books, products and media the guest or host genuinely endorsed here — with the buy link.
Affiliate link — we may earn a commission at no extra cost to you.
Parallel Distributed Processing
James McClelland and David Rumelhart
“you wrote parallel distributed processing books that explored ideas of neural networks in the 1980s together with a few folks” — Lex Fridman 00:17:34Find it on Amazon