Machine Learning for Speech Recognition by Alice Coucke, Head of Machine Learning Research

Home , AlphaFold, OpenAI Five, Speech recognition

@alicecoucke alice.coucke@sonos.com Outline:

1. Recent advances in machine learning

2. From physics to machine learning IA en général Speech & NLP

Startup

3. Working at Snips (now Sonos) Snips A lot of applications in the real world, quite a diﬀerence from theoretical work

Un des rares domaines scientiﬁques où la théorie et la pratique sont très emmêlées Recent Advances in Applied Machine Learning A lot of applications in the real world, quite a diﬀerence from theoretical work Reinforcement learning Learning goal-oriented behavior within simulated environments

Go Starcraft II Dota 2 AlphaGo (Deepmind, 2016) AlphaStar (Deepmind) OpenAI Five (OpenAI)

Play-driven learning for robots Sim-to-real dexterity learning (Google Brain) Project BLUE (UC Berkeley) Machine Learning for Life Sciences Deep learning applied to biology and medicine

Protein folding & structure Cardiac arrhythmia prediction Eye disease diagnosis prediction (NHS, UCL, Deepmind) from ECGs AlphaFold (Deepmind) (Stanford)

Reconstruct speech from neural activity Limb control restoration (UCSF) (Batelle, Ohio State Univ) Computer vision High-level understanding of digital images or videos

GANs for image generation (Heriot Watt Univ, DeepMind)

« Common sense » understanding of actions in videos (TwentyBn, DeepMind, MIT, IBM…)

GANs for artiﬁcial video dubbing GAN for full body synthesis (Synthesia) (DataGrid) From physics to machine learning and back A surge of interest from the physics community

NeurIPS 2019: workshop on « machine learning and the physical sciences » Speech and language Understand and analyze human speech

{ intent: FindWeather, entities: { datetime: 11/28/2019, location: Paris } }

Speech transcription Spoken language Text & speech generation Human Parity (Microsoft) understanding GPT-2 (Open AI) (Super)GLUE benchmarks (Google, Bert (Google) Facebook, IBM, Stanford …) XLNet (CMU) … Speech and language Understand and analyze human speech

{ intent: FindWeather, entities: { datetime: 11/28/2019, location: Paris } }

� �

Neural machine translation Voice activity detection Speaker identiﬁcation Sentiment analysis Detect emotions in text and Unsupervised MT (Facebook) Detect speech from audio Recognize unique speakers speech Fairness, ethics, and explainability We, as scientists, have a say in the future of AI From physics to machine learning Décrire mon parcours My background Parler de la thèse From physics to machine learning Parler du post doc

Pourquoi les physiciens / chercheurs sont une bonne formation - appréhender des théories complexes et les appliquer à des domaines variés - Programmation skills - Formation par la recherche : communiquer des résultats, lire un état de l’art, écrire des articles - Experimental method Décrire mon parcours My background Parler de la thèse From physics to machine learning Parler du post doc

Pourquoi les physiciens / chercheurs 2012: M2 ICFP Theoretical physics sont une bonne formation - appréhender des théories complexes et les appliquer à des 2013-2016: PhD in statistical physics @LPTENS domaines variés - Programmation skills - Formation par la recherche : communiquer des résultats, lire un état de l’art, écrire des articles - Experimental method Décrire mon parcours My background Parler de la thèse From physics to machine learning Parler du post doc

Feb 2017: senior ML scientist @ Snips

2019: director of ML research @ Snips

Today: head of ML research @ Sonos, Inc. A few takeaways (please go ask other people too) A few takeaways (please go ask other people too)

• PhD? Postdoc?

• Working at a startup company

• Physicists and machine learning A few takeaways (please go ask other people too)

• PhD? Postdoc?

• Working at a startup company

• Physicists and machine learning

Physicists @ Snips

A. C. Raﬀaele Tavarone Francesco Caltagirone Alaa Saade Stéphane d’Ascoli Sr ML Scientist Sr ML Scientist Sr ML Scientist ML research intern Acoustics team Tech Lead Now: DeepMind Now: PhD ENS & FAIR Language team A few takeaways (please go ask other people too)

• PhD? Postdoc?

• Working at a startup company

• Physicists and machine learning

Physicists @ Snips

Speech & language recognition - Privacy - pas cloud données en local —> contraintes en + techniques - resources computaitonelles - Peu de données pour entraîner des données - Chaque bloc est optimisé adapté Spoken Language Understanding From speech to meaning for voice assistants

Automatic Speech Recognition Engine Language modeling

Natural Intent: t ɜ r n ɑ n ð ə Acoustic Language Turn on the Language SwitchLightOn l a ɪ t s ɪ n ð ə model model lights in the Understanding ˈl ɪ v ɪ ŋ r u m living room Engine Slots: room: living room

Wake Word ACR ICF Dialogue Custom Wake Automatic Command Intent Classiﬁcation Multiturn Dialogue Word Recognition and Filling and Response Audio Frontend Action Code Spoken Language Understanding From speech to meaning for voice assistants

Automatic Speech Recognition Engine Language modeling

Deep neural network Proba over phones /a/ /b/ /c/ /d/ /e/ time Spoken Language Understanding From speech to meaning for voice assistants

Automatic Speech Recognition Engine Language modeling

Natural Intent: t ɜ r n ɑ n ð ə Acoustic Language Turn on the Language SwitchLightOn l a ɪ t s ɪ n ð ə Language model model lights in the Understanding ˈl ɪ v ɪ ŋ r u m model living room Engine Slots: room: living room

Proba over phones Decoding graph Turn on the lights in the living room 0.85 Turn off the lights in the living room 0.75 Set the lights in the living room 0.60 /a/ /b/ /c/ /d/ /e/ time Spoken Language Understanding From speech to meaning for voice assistants

Automatic Speech Recognition Engine Language modeling

Logistic regression Conditional Random Field Turn on the lights Intent: Slots: in the living room SwitchLightOn room: living room Spoken Language Understanding Oﬄine & on device

Automatic Speech Recognition Engine Language modeling

❌ ❌ ✅ Wake Word ACR ICF Dialogue Custom Wake Automatic Command Intent Classiﬁcation Multiturn Dialogue Word Recognition and Filling and Response Audio Frontend Action Code Our approach Our own voice in a vast ecosystem

Privacy by design Resource constrained ML: small data & hardware

✓ A new popular trend in the ML community (low resource ML, transfer learning, miniaturization, etc) ✓ Numerous conferences and workshops on the topic ✓ Towards a safer, greener and more private conversational AI Research activity Publishing research in industry

Snips voice platform: Efficient keyword spotting an embedded spoken language using dilated convolutions Federated learning for Spoken language understanding system for and gating keyword spotting understanding on the edge private-by-design voice interfaces arXiv arXiv arXiv arXiv There is a lot to do in research in the industry, with us joining Sonos we expect to grow the team and doing even more exploratory stuﬀ.

ICML 2018 ICASSP 2019 ICASSP 2019 NeurIPS 2019 workshop PiMLAI Main track Main track Workshop EMC2

Cited 35 times to date ✓ Publish open & reproducible benchmarks: ‣ ~200 access granted to researchers to our open speech datasets ‣ Snips dataset for NLU is the new academic standard Thank you for your attention Questions?

@alicecoucke [email protected]