Unsupervised Cross-Modal Alignment of and Text Embedding Spaces

Yu-An Chung Wei-Hung Weng Schrasing Tong James Glass and Laboratory Massachusetts Institute of Technology Cambridge, Massachusetts, USA

NeurIPS Montréal, Québec, Canada December 2018 (MT) Training data

French “the cat is black” MT system “le chat est noir” ( English , ) text translation

Automatic (ASR)

English ASR system “dogs are cute” ( English , ) audio transcription

Text-to- (TTS)

English English “cats are adorable” TTS system ( , ) text audio Machine Translation (MT) Training data

French “the cat is black” MT system “le chat est noir” ( English , ) text translation

Automatic Speech Recognition (ASR)

English ASR system “dogs are cute” ( English , ) audio transcription

Text-to-Speech Synthesis (TTS)

English English “cats are adorable” TTS system ( , ) text audio

Parallel corpora for training à Expensive to collect! Framework 1 Language 2

Wikipedia is a multilingual, web-based, free encyclopedia based on a model of openly editable and viewable content, a wiki. It is the largest and most popular … Framework Do not need to be parallel! Language 1 Language 2

Wikipedia is a multilingual, web-based, free encyclopedia based on a model of openly editable and viewable content, a wiki. It is the largest and most popular … Framework Do not need to be parallel! Language 1 Language 2

Wikipedia is a multilingual, web-based, free encyclopedia based on a model of openly editable and viewable content, a wiki. It is the largest and most popular …

Word2vec [Mikolov et al., 2013]

( y$ y % +×' Y = ⋮ ∈ ℝ y' Framework Do not need to be parallel! Language 1 Language 2

Wikipedia is a multilingual, web-based, free encyclopedia based on a model of openly editable and viewable content, a wiki. It is the largest and most popular …

Speech2vec [Chung & Glass, 2018] [Mikolov et al., 2013]

( x$ x % +×' X = ⋮ ∈ ℝ x' ( y$ y % +×' Y = ⋮ ∈ ℝ y' Framework Do not need to be parallel! Language 1 Language 2

Wikipedia is a multilingual, web-based, free encyclopedia based on a model of openly editable and viewable content, a wiki. It is the largest and most popular …

Speech2vec Word2vec [Chung & Glass, 2018] [Mikolov et al., 2013]

( x$ x % +×' X = ⋮ ∈ ℝ x' ( y$ MUSE y [Lample et al., 2018] % +×' Y = ⋮ ∈ ℝ y'

Learn a linear mapping W such that

∗ W = argmin WX − Y 9 W∈ℝ7×7 Framework Do not need to be parallel! Language 1 Language 2

Wikipedia is a multilingual, web-based, free encyclopedia based on a model of openly editable and viewable content, a wiki. It is the largest and most popular …

Speech2vec Word2vec [Chung & Glass, 2018] [Mikolov et al., 2013]

) x% x & ,×( X = ⋮ ∈ ℝ x( ) y% MUSE y [Lample et al., 2018] & ,×( Y = ⋮ ∈ ℝ y(

Learn a linear mapping W such that

∗ W = argmin WX − Y 9 W∈ℝ7×7 Components p Word2vec • Learns distributed representations of from a that model in an unsupervised manner p Speech2vec • A speech version of word2vec that learns semantic word representations from a ; also unsupervised p MUSE • An unsupervised way to learn W with the assumption that the two embedding spaces are approximately isomorphic Framework Do not need to be parallel! Advantages Language 1 Language 2 p Rely only on monolingual corpora of speech and text that: • Do not need to be parallel Wikipedia is a multilingual, web-based, free • Can be collected independently, greatly reducing human encyclopedia based on a model of openly labeling efforts editable and viewable content, a wiki. It is the largest and most popular … p The framework is unsupervised: • Each component uses • Applicable to low-resource language pairs that lack bilingual Speech2vec Word2vec resources [Chung & Glass, 2018] [Mikolov et al., 2013]

) x% x & ,×( X = ⋮ ∈ ℝ x( ) y% MUSE y [Lample et al., 2018] & ,×( Y = ⋮ ∈ ℝ y(

Learn a linear mapping W such that

∗ W = argmin WX − Y 9 W∈ℝ7×7 Components p Word2vec • Learns distributed representations of words from a text corpus that model word semantics in an unsupervised manner p Speech2vec • A speech version of word2vec that learns semantic word representations from a speech corpus; also unsupervised p MUSE • An unsupervised way to learn W with the assumption that the two embedding spaces are approximately isomorphic Framework Do not need to be parallel! Advantages Language 1 Language 2 p Rely only on monolingual corpora of speech and text that: • Do not need to be parallel Wikipedia is a multilingual, web-based, free • Can be collected independently, greatly reducing human encyclopedia based on a model of openly labeling efforts editable and viewable content, a wiki. It is the largest and most popular … p The framework is unsupervised: • Each component uses unsupervised learning • Applicable to low-resource language pairs that lack bilingual Speech2vec Word2vec resources [Chung & Glass, 2018] [Mikolov et al., 2013]

) Usage of the learned W x% x & ,×( #1: Unsupervised spoken word recognition: when language 1 = language 2 X = ⋮ ∈ ℝ x (e.g., both English) ( Input spoken ) y% word “dog” “dog” MUSE y “dogs” [Lample et al., 2018] & ,×( Nearest neighbor Y = ⋮ ∈ ℝ + search “puppy” y( “pet” ⋮ Foundation for Unsupervised Automatic Speech Recognition Learn a linear mapping W such that An interesting property of our approach: synonym retrieval ∗ à The list of nearest neighbors actually contain both synonyms and W = argmin WX − Y 9 W∈ℝ7×7 different lexical forms of the input spoken word. Components #2: Unsupervised spoken word translation: when language 1 ≠ language 2 p Word2vec (e.g., English to French) • Learns distributed representations of words from a text corpus that model Input spoken word semantics in an unsupervised manner word “dog” “chien” p Speech2vec Nearest neighbor “chiot” + search • A speech version of word2vec that learns semantic word representations ⋮ from a speech corpus; also unsupervised p MUSE • An unsupervised way to learn W with the assumption that the two Foundation for Unsupervised Speech-to-Text Translation embedding spaces are approximately isomorphic Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces

10:45 AM – 12:45 PM Room 210 & 230 AB #156