The Deep Learning Revolution (The MIT Press) | Terrence J. Sejnowski

Summary of: The Deep Learning Revolution (The MIT Press)
By: Terrence J. Sejnowski


Delve into the fascinating world of deep learning as Terrence J. Sejnowski unravels the mysteries of intelligence in his revolutionary book, ‘The Deep Learning Revolution’. Explore the origins and development of machine learning while gaining valuable insights into human intelligence. Discover the intricate processes and challenges faced by researchers in creating evolving artificial intelligence that mimics the human brain. The book takes a peek at groundbreaking technologies like neural networks and deep learning in various fields, including language translation, medical diagnosis, and game playing, underscoring their relevance to everyday life.

The Deep Learning Revolution

Knowledge is no longer external to the brain. Machines are now learning how to translate words, recognize voices, and diagnose illnesses using deep learning systems rooted in mathematics, computer science, and neuroscience. Data is the new commodity, and learning algorithms are the refineries that extract information from raw data, fueling the deep learning revolution. This transformation not only focuses on the evolution of artificial intelligence but also on human intelligence. The deep learning revolution took three decades of patience and perseverance by a small but luminous research community to bring machine learning into the 21st century.

The Complexity of AI and Human Intelligence

AI pioneers sought to emulate human intelligence by writing computer programs with similar functionality. However, emulating the brain’s processes by executing algorithms within “massively parallel architectures” is essential to successfully copying human intelligence.
General intelligence, rather than just logic, is at the core of human intelligence, and learning is the foundation of general intelligence. Automating the process of defining the weighted connections between inputs is essential for AI to learn from many examples and create generalizations. The more complex an independent component analysis (ICA) becomes, the more efficiently it processes information. The creation of artificial intelligence using massively parallel architectures requires “computational neuroscience” to understand fully the nature behind AI operations and the human intelligence the AI intends to mimic.

Neural Network Design and Learning

A scruffy model distributes the representation of objects across many units and uses approximations to get qualitative answers, whereas a neat model is more computationally compact and proves more accurate. Progress in neural network design requires both. The key lies in giving them more complex dynamics by building feedback connections between layers. In 1982 hopfield introduced the Hopfield net, wherein every output connects back to all the inputs in the network and their strengths are symmetrical. This nonlinear network makes simultaneous updates, and the Boltzmann machine’s goal is to find the global energy minimum of a Hopfield net. By extracting statistical regularities common to all the data, freezing the weights at the first layer, and adding layers of more and more input units, the upper layers produce more nonlinear combinations of low-level features, making it possible to abstract the general from the specific. This “bottom-up” type of learning mirrors the behavior of the human cortex.

Machine Learning Breakthrough

In 1986, researcher Terry Sejnowski led a team that used the Boltzmann machine to teach a computer to predict the sound of the middle letter in a seven-letter window of English words. The machine learned to do so with almost perfect accuracy, even when tested on 20,000 words. The resulting network, NETtalk, went through a “babbling” phase like a human baby and even appeared on the Today Show. NETtalk later inspired the development of Google Translate, which proved that learning a language does not rely on rules, but on experience and a rich cognitive context. In 2016, using deep learning, Google Translate translated and then re-translated a paragraph from Ernest Hemingway’s The Snows of Kilimanjaro in Japanese and English, demonstrating the importance of semantics in language learning. The possibilities of future generative networks that could explain themselves are exciting.

Networks and Algorithmic Biology

A new field of algorithmic biology is developing algorithms to explain biological systems. While artificial neural networks are useful, they are limited, and their processes remain unknown. Although they can provide accurate diagnoses, they lack human experience and pattern recognition. Researchers are building a better understanding of the human brain by studying how deep learning neural networks record every activity. Artificial neural networks suffer bias by design because they only have the information humans give them. It’s networks all the way down, and biological systems have layers of complexity across temporal and spatial scales.

Want to read the full book summary?

Leave a Reply

Your email address will not be published. Required fields are marked *

Fill out this field
Fill out this field
Please enter a valid email address.
You need to agree with the terms to proceed