Depths of Learning

thesis Depths of learning

The coming revolution in smart machines this design — so-called ‘convolutional’ and robots has been predicted by artificial This time the neural networks — have inputs representing intelligence enthusiasts in every decade since small, overlapping regions of an image the 1950s. It’s never happened. Progress revolution in artificial or text, and are, as a result, well tuned to in artificial intelligence has been halting, intelligence looks to be structural hierarchies present in natural tedious, in many ways disappointing. signals. In images, combinations of edges Notable successes — IBM’s Deep Blue for real. form motifs, which then assemble into parts, computer beating Gary Kasparov in chess in whereas in speech or text, basic sounds 1997, and the emergence of reasonably useful integrate to phonemes, then syllables, words language translation in the late 1990s — data, in much the same way as the human and sentences. mostly came through brute computation, eye and brain. These neural networks take As LeCun and colleagues point out, rather than any essential similarity between the raw pixel data as input to a first layer of these neural networks, because of their computer operations and the mechanisms neuronal elements, and process it through architecture, actually respond to images behind human thought or perception. nonlinear connections to multiple further in the way real brains do. Studies find, But this history of failure is now layers of other neurons. After training, these for example, that the neuron-by-neuron changing, and sharply. A decade ago, come to represent more abstract aspects of activity pattern in a neural net trained algorithms for computer vision and object the image. In the case of vision, for example, on an image accounts for about half the recognition could barely identify a simple the next layer may capture features such as variance found in random sets of neurons ball or block on a plain background. Now edges and their orientations in particular in the visual cortex of a monkey viewing they can distinguish distinct human faces locations. A further layer might then the same image. This structural trick is more or less as well as real people, even in detect motifs built from edges, and another now used in most applications of machine complex natural backgrounds. Two months might assemble such motifs into larger learning for recognition and detection — ago, Google released a smartphone app patterns reflecting parts of familiar objects. not only for text and images, but in areas capable of translating text from more than Impressively, such networks can learn to such as identifying potentially useful drug 20 foreign languages just from photos — of recognize the most useful features of images molecules or analysing particle accelerator a road sign, perhaps, or a handwritten note. without any human help. data. Advancing technology has helped, as One of Google’s research groups recently Writing in Nature, Yann LeCun, modern architectures work with as many showed that an algorithm — simply by Yoshua Bengio and Geoffrey Hinton offer as 10 to 20 layers of nonlinear elements, watching and learning — could play all of an excellent review of recent progress (see training through the adjustment of hundreds the Atari video games of the 1980s with the Nature 521, 436–444; 2015). A crucial of millions of weight parameters. same skill as human experts. problem that these methods overcome, they These breakthroughs in computer It’s all the result of the discovery that some note, is linked to the basic symmetries of science and applied mathematics now old ideas about neural networks, if altered in objects in images. Images of a Husky in two drive frenetic research by commercial small but significant ways, are much more different poses ought to register as the same firms such as Google and Facebook. powerful than anyone expected — and of a dog. Yet, in a pixel-by-pixel description, Artificial intelligence isn’t what it used to conscious effort to mimic some of the good such images might be almost completely be. Gill Pratt, program director for robotics tricks used in the biology of human and different. Likewise, images of a Husky and research at the Defense Advanced Research animal perception. This time the revolution Samoyed in the same pose, which should be Projects Agency (DARPA), suggests that in artificial intelligence looks to be for real. distinguished, might be almost identical. The artificial intelligence may soon experience Machine-learning research has always ‘deep’ in deep learning refers to the number an explosive event not unlike the Cambrian been about getting machines to learn to of input-to-output layers in the network, explosion in evolutionary history when, in recognize complex patterns. Connect a which now often reaches 10 or 20, rather a short span of time, evolution proliferated computer to a camera, and a good image than only a handful in earlier work. As from simple, rudimentary single-celled identification algorithm should be able, from researchers have discovered, with multiple organisms into myriad multicellular the raw image data, to identify a particular nonlinear layers, such systems can express forms (J. Econ. Perspect. 29, 51–60; 2015). human face, a coffee cup, or a Husky pulling intricate functions of their inputs and so That explosion happened when evolution a sled. Historically, however, such networks remain sensitive to small feature details somehow stumbled on a design pathway have always struggled. Even rudimentary while being insensitive to image variations leading out of the single-celled trap and into success demanded human intervention — linked to basic symmetries — changes in new open space. people helping the algorithms by defining lighting, pose or background, for example. In artificial intelligence, the combination important large-scale features that the Perhaps the most exciting advance of of deep learning methods, advancing algorithms could look for, such as edges all involves networks that mimic aspects of neuroscience and computing power is doing or simple geometrical figures. Algorithms human and animal perception. In the visual the same for brains and intelligence, which are couldn’t learn to see the usefulness of these cortex of the human brain, cells respond to no longer limited to the biological kind. There features on their own. light striking small, overlapping subregions may be many further, and higher, levels of Not any more. This has all changed owing of the visual field. These neurons then feed intelligence just waiting to be discovered. ❐ to the discovery of so-called deep learning into further higher processing layers in a neural networks, which act on raw image hierarchical way. Neural networks following MARK BUCHANAN

798 NATURE PHYSICS | VOL 11 | OCTOBER 2015 | www.nature.com/naturephysics