KEYNOTE: Deep Learning Toolbox Al 2020

Home , Human image synthesis, OpenAI Five, Tensor processing unit, Wing (company)

portada KEYNOTE SPEAKER

“Deep Learning Toolbox en el 2020” Presenta: Marc Torrent Director CIDAI cidai.eu | @CIDAI_eu KEYNOTE SPEAKER

“Deep Learning Toolbox en el 2020” Oriol Vinyals Research Director in Deep Learning Google DeepMind research.google/people/OriolVinyals/ The Deep Learning Toolbox in 2020 @OriolVinyalsML Research Director Deep Learning Google DeepMind

AI & Big Data Barcelona (online), October 2020 artiﬁcial intelligence grand project to build non-human intelligence

machine learning machines that learn to be smarter artiﬁcial intelligence grand project to build non-human intelligence

machine learning machines that learn to be smarter

deep learning supervised learning reinforcement learning artiﬁcial intelligence grand project to build non-human intelligence

machine learning machines that learn to be smarter

deep learning Lots of attention

● Startups / VCs ● Facebook / Google / Amazon / Apple ● Open source efforts ● Press (noise...) ● Universities Machine Learning Artiﬁcial Intelligence Deep Learning ML is done in many places

TensorFlow GitHub stars by GitHub user profiles w/ public locations Source: http://jrvis.com/red-dwarf/?user=tensorflow&repo=tensorflow Datacenter Revolution

Picture credit: http://americanhistory.si.edu/exhibitions/preview-case-american-enterprise

General Artiﬁcial Intelligence General Artiﬁcial Intelligence Hardware Revolution

Tensor Processing Unit v2

● 180 teraflops of computation, 64 GB of HBM memory, 2400 GB/s mem BW ● Designed to be connected together into larger configurationsGeneral Artificial Intelligence TPU Pod 64 2nd-gen TPUs For comparison, 11.5 petaflops #10 supercomputer in world has 4 terabytes of memory Rpeak of 11 petaflops* Data Revolution Datasets / Environments

http://www.spacemachine.net/views/2016/3/datasets-over-algorithms Data modalities increasingly diverse

“[...] The alternative approach, which they thought was crazy, was to forget logic and try and understand how networks of brain cells learn things. Curiously, two people who rejected the logic based approach to AI were Turing and Von Neumann. [...] now neural networks are everywhere and the crazy approach is winning.” G. Hinton

Words, letters Speech Videos

Images Programs Graphs Software Revolution Big tech companies open sourced most ML tools! Research Revolution The life of a researcher The Deep Learning Toolbox: Zooming Out

Platforms Frameworks Datasets Summary

● Datacenter Revolution ● Hardware Revolution ● Data Revolution ● Software Revolution ● Research Revolution Algorithms Revisited Back to the 50s ConvNets

General Artiﬁcial Intelligence A Learning Algorithm

Given training examples “(input, output)” pairs

While not done:

Pick a training example (x, y)

Run the neural network on x

Compare actual output to y

Adjust parameters to reduce the error (the “loss”) The Deep Learning Toolbox: Zooming In

Feed forward models Sequence Prediction Seq2Seq Attention & Pointers

Read/Write memories Temporal Hierarchies Key,Value memories

Graph Neural Networks Recurrent Architectures

Figure credits: Jeff Dean, Chris Olah, Santoro et al 2016, Koutnik et al 2014, van den Oord et al 2016, Miller et al 2016, Vinyals et al 2016, Vaswani et al 2017 Feed forward models Functions a deep neural network can learn input output

Pixels: “lion” Sequence Prediction Functions a deep neural network can learn input output

Pixels: “lion”

Audio: “How cold is it outside?” Seq2Seq Functions a deep neural network can learn input output

Pixels: “lion”

Audio: “How cold is it outside?”

“Hello, how are you?” “Bonjour, comment allez-vous?” Functions a deep neural network can learn input output

Pixels: “lion”

Audio: “How cold is it outside?”

“Hello, how are you?” “Bonjour, comment allez-vous?”

Pixels: “A blue and yellow train travelling down the tracks” 2011

humans

26% errors 5% errors 2011 2016

humans

26% errors 5% errors 3% errors Impact @ Google and beyond Growing use of Deep Learning at Google

“Google will soon be a big LSTM”, Jüergen Schmidhuber

Across many products/areas: Android Apps drug discovery Gmail Image understanding Maps Photos Robotics research Search Speech Translate YouTube … many others ... Products using Machine Learning Machine Learning Enabling Apps that See, Hear and Understand https://waymo.com/tech/ Hemorrhages

Healthy Diseased

No DR Mild DR Moderate DR Severe DR Proliferative DR 1 2 3 4 5 F-score 0.95 0.91 Algorithm Ophthalmologist (median)

“The study by Gulshan and colleagues truly represents the brave new world in medicine.”

Dr. Andrew Beam, Dr. Isaac Kohane Harvard Medical School

“Google just published this paper in JAMA (impact factor 37) [...] It actually lives up to the hype.” Dr. Luke Oakden-Rayner University of Adelaide www.nasa.gov/press-release/artiﬁcial-intelligence-nasa-data-used-to-discover-eighth-planet-circling-distant-star Blog: www.blog.google/topics/machine-learning/hunting-planets-machine-learning/ Paper: [Shallue & Vandenburg], www.cfa.harvard.edu/~avanderb/kepler90i.pdf www.nasa.gov/press-release/artiﬁcial-intelligence-nasa-data-used-to-discover-eighth-planet-circling-distant-star Blog: www.blog.google/topics/machine-learning/hunting-planets-machine-learning/ Paper: [Shallue & Vandenburg], www.cfa.harvard.edu/~avanderb/kepler90i.pdf From research to production examples Google Translate Seq2Seq Machine Translation WaveNet (Google Voice)

van den Oord, A., et al. “WaveNet: A Generative Model for Raw Audio.” arxiv (2016). Temporal Hierarchies Text to speech Feed forward models Which of these are not real?

Brock, A., et al. “Large Scale GAN Training for High Fidelity Natural Image Synthesis” ICLR (2019). Feed forward models

Razavi, A., et al. “Generating Diverse High-Fidelity Images with VQ-VAE-2” ICLR (2019).

Frontiers and challenges Sequence Prediction Neural Chatbots Conditional Language [Vinyals, Le, ICML DL WS 2015]

Human: what do you think about tesla ? Machine: he ’s a good conductor .

Human: What do you think about messi ? Machine: he ’s a great player .

Human: what do you think about cleopatra ? Machine: oh , she ’s very regal .

Human: who is skywalker ? Machine: he is a hero .

Human: is sky blue or black ? Machine: blue .

Human: does a cat have a tail ? Machine: yes .

Human: does a cat have a wing ? Machine: no Scale of Language Models

Recurrent Architectures

● Movie subtitles: 900M ● LM1B: 1B Sequence Prediction Key,Value memories ● GPT2: 40B ● GPT3: 500B Evolution of language modeling

Shannon, 1951 (samples from SLP book, 2000), 3-gram [ ] They also point to ninety nine point six billion dollars from two hundred four oh six three percent of the rates of interest stores as Mexico and Brazil on market conditions

Sutskever et al, 2011, RNNs [ ] while he was giving attention to the second advantage of school building a 2-for-2 stool killed by the Cultures saddled with a halfsuit defending the Bharatiya Fernall ’s ofﬁce

Jozefowicz et al, 2016, BIG LSTMs [ ] With even more new technologies coming onto the market quickly during the past three years , an increasing number of companies now must tackle the ever-changing and ever-changing environmental challenges online . Evolution of language modeling Liu et al, 2018, Transformer [==wings over kansas] ==wings over kansas is a 2010 dhamma feature film written and directed by brian ig ariyoshi . it premiered on march 17, 2010 the film tells the story of three americans who bravely achieved a victory without expected daknfi . ==Wings Over Kansas Plot the story begins with the faltering success of egypt 's hungry dakfunctionality when he loses his lives around the time when the embarked white - collar daughters begin their father 's cabin. the rest of the campaign ( coming to town ) gives dakhandles [...]

Radford and Wu et al, 2019, BIG Transformer [In a shocking ﬁnding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.] The scientist named the population, after their distinctive horn, Ovid’s Unicorn. These four-horned, silver-white unicorns were previously unknown to science. Now, after almost two centuries, the mystery of what sparked this odd phenomenon is ﬁnally solved. Dr. Jorge Perez, an evolutionary biologist from the University of La Paz, and several companions, were exploring the Andes Mountains when they found a small valley, with no other animals or humans. Perez noticed that the valley had what appeared to be a natural fountain, surrounded by two peaks of rock and silver snow. Perez and the others then ventured further into the valley. “By the time we reached the top of one peak, the water looked blue, with some crystals on top,” said Perez. Perez and his friends were astonished to see the unicorn herd. These creatures could be seen from the air without having to move too much to see them – they were so close they could touch their horns. [...] GPTX ● Transformer-based ● GPT2: ○ 1.5 billion parameters ○ 40 billion words ● GPT3: ○ 175 billion parameters ○ 500 billion words ● Adapts to style and content of arbitrary conditioning input

Radford et al. (2019) https://openai.com/blog/better-language-models/#sample1 Brown & Mann & Ryder & Subbiah et al (2020) @shariffshameen losslesshq.com

@mattshummer_ @sh_reya Challenge: One Shot Learning

● Humans have a capacity for very rapid assimilation of data (one/few-shot learning).

Lake et al, 2013, 2015 Challenge: Adversarial Examples

Hamster Airplane

Image Image classifier classifier Crafted adversarial perturbation

Clean image

Adversarial image [ Intriguing properties of neural networks, Szegedy et al, 2014 ] Reinforcement Learning

OBSERVATIONS GOAL

Agent Environment

ACTIONS

DQN, Mnih et al, 2014; AlphaGo, Silver et al, 2016; OpenAI FIve, 2019; AlphaStar, Vinyals et al, 2019 Challenge: Robotics Challenge: Robotics Challenge: Robotics

One-Shot Imitation Learning, Duan et al, NeurIPS 2017 Challenge: Robotics

One-Shot Imitation from Watching Videos, Yu & Finn et al, RSS 2018 Challenge: Robotics

One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning, Yu & Finn et al 2018 Concluding Thoughts

Hardware, Software, Data, and Research Revolutions ⇒ breakthroughs

● Machine Learning:

○ More impact in Health, Self Driving cars

● Supervised Learning:

○ Learning from a few examples

○ Text models

● Reinforcement Learning:

○ Real world problems → Robotics Gracias!! Preguntas?? @OriolVinyalsML Octubre 2020 portada