On Surrogate Supervision Multi-View Learning
Total Page:16
File Type:pdf, Size:1020Kb
AN ABSTRACT OF THE THESIS OF Gaole Jin for the degree of Master of Science in Electrical and Computer Engineering presented on December 3, 2012. Title: On Surrogate Supervision Multi-view Learning Abstract approved: Raviv Raich Data can be represented in multiple views. Traditional multi-view learning meth- ods (i.e., co-training, multi-task learning) focus on improving learning performance using information from the auxiliary view, although information from the target view is sufficient for learning task. However, this work addresses a semi-supervised case of multi-view learning, the surrogate supervision multi-view learning, where labels are available on limited views and a classifier is obtained on the target view where labels are missing. In surrogate multi-view learning, one cannot obtain a classifier without information from the auxiliary view. To solve this challenging problem, we propose discriminative and generative approaches. c Copyright by Gaole Jin December 3, 2012 All Rights Reserved On Surrogate Supervision Multi-view Learning by Gaole Jin A THESIS submitted to Oregon State University in partial fulfillment of the requirements for the degree of Master of Science Presented December 3, 2012 Commencement June 2013 Master of Science thesis of Gaole Jin presented on December 3, 2012. APPROVED: Major Professor, representing Electrical and Computer Engineering Director of the School of Electrical Engineering and Computer Science Dean of the Graduate School I understand that my thesis will become part of the permanent collection of Oregon State University libraries. My signature below authorizes release of my thesis to any reader upon request. Gaole Jin, Author ACKNOWLEDGEMENTS My most sincere thanks go to my advisor, Dr. Raviv Raich. I thank Dr. Raich for his expertise, time, and patience in assisting me finishing my Master of Sci- ence program and researches. Dr. Raich’s support and encouragement are very important for me to finish this work. I also would like to thank Dr. Xiaoli Fern, Dr. Sinisa Todorovic, Dr. Bill Smart for serving as committee members in my defence., and Dr. Yevgeniy Kovchegov for attending my defence. I thank my lab mates: Qi, Behrouz, Greg, Zeyu, Evgenia, Balaji, and Deepthi. Thank them for creating a great atmosphere in the lab. I also thank all my friends in China and in USA. I enjoyed the time I spent with all my friends. Finally, I would like to thank my parents. Their support for me is the greatest gift anyone has ever given to me. I would never make it without their love and support. TABLE OF CONTENTS Page 1 INTRODUCTION 1 1.1 Background ................................ 1 1.2 OutlineofThesis ............................. 4 2 LITERATURE REVIEW 5 3 PROBLEM FORMULATION 8 4 ALTERNATIVE SOLUTIONS 11 4.1 CCA+SVM ............................... 11 4.2 Label-transferredLearning . 12 5 C4A ALGORITHM 13 5.1 Two-classCase .............................. 13 5.2 Multi-classCase.............................. 15 5.3 Sub-Gradient Descent Implementation . 16 6 HINGE LOSS UPPER BOUND FOR SSML 18 6.1 Two-classCase .............................. 18 6.2 Multi-classCase.............................. 20 6.3 SSM-SVMAlgorithm........................... 21 6.4 Sub-gradientDescentforSSM-SVM. 23 7 GAUSSIAN MIXTURES 25 7.1 Introduction................................ 25 7.2 Formulation................................ 27 8 SIMULATIONS 34 8.1 Test of the Proposed Discriminative Approach . 34 8.1.1 TestonSyntheticData . 34 TABLE OF CONTENTS (Continued) Page 8.1.2 Application to Lip-reading . 36 8.1.3 Testing Result and Analysis . 41 8.2 TestoftheProposedGenerativeApproach . 43 8.2.1 TestonSyntheticData . 43 8.2.2 TestonanAudiovisualTask . 46 9 CONCLUSION 54 9.1 Summary ................................. 54 9.2 Contributions ............................... 55 9.3 Publications................................ 55 9.4 FutureWork................................ 56 Bibliography 56 Appendices 61 A CanonicalCorrelationAnalysis . 62 B PrincipalComponentAnalysis. 64 C Proof ................................... 66 D Proof.................................... 67 LIST OF FIGURES Figure Page 1.1 X isvideo................................ 2 1.2 X isvideo................................ 3 3.1 In surrogate supervision multi-view learning, we are given two sets m of data: paired examples (xi, zi) i=1 from and , and labeled n { } X Z examples (xi,yi) i=m+1 from . We are interested in learning a classifier for{ y given} z. .........................X 9 3.2 The data is separated to training data and testing data. In training data, we consider two groups: one are examples from with ac- cordingly labels from ; the other are unlabeled pairs fromX both Y X and . In testing data, we wan to obtain labels y of examples fromZ . Note that “ ” indicates the availability of∈Y data. 9 Z 4.1 A classifier f( ): is first learned based on the training ex- · n X →Y amples (xi,yi) i=m+1. Then, based on the two-view pair examples {n } m (xi, zi) the following training examples (zi, yˆ) are formed. 12 { }i=1 { }i=1 8.1 A gray scale image of the lip region. 39 8.2 Aspectrogramoftheaudio. 40 8.3 X isvideo................................ 49 8.4 Xisaudio................................ 50 8.5 Optionalcaptionforlistoffigures . 51 8.6 The synthetic data set, with two one-dimensional (X and Z) views. 52 8.7 X isvideo................................ 53 LIST OF TABLES Table Page 3.1 Descriptionsofthesymbols. 10 8.1 Classification accuracies achieved by C4A, SSM-SVM, CCA + SVM, and label-transferred learning in lung-cancer, spect, wine, hill-valley, ionosphere, glass datasets. Note that the classification is performed on where the labeled samples are not available and that the linear SVMZ is applied in CCA + SVM and in label-transferred learning. 36 8.2 The classification accuracies achieved by trained linear SVM classi- fieronaudioandonvideorespectively. 38 8.3 The highest classification accuracy achieved by the C4A algorithm among different combinations of r and γ. .............. 39 8.4 The highest classification accuracy achieved by the SSM-SVM algo- rithm among different combinations of r and γ. ........... 40 8.5 Digit prediction accuracies with inferences made solely using Z as input, for varying σ2 and J, using the proposed model. 47 8.6 Prediction accuracies for inference based on with varying σ2 and Z J, using mixtures trained based on supervised (Zi,Ci)pairs. 47 LIST OF ALGORITHMS Algorithm Page 1 Label-transferredLearningMethod . 12 2 Sub-Gradient Descent Implementation of C4A Algorithm . 17 3 Sub-Gradient Descent Implementation of SSM-SVM Algorithm . 23 Chapter 1: INTRODUCTION 1.1 Background It is common to learn a classifier from a set of examples of the form (feature vector, T label). Feature vectors can be represented as a vector the form [x , , xd] . How- 1 ··· T ever, in multi-view learning we consider subsets of the feature vectors [x , , xs] 1 ··· T and [xs , , xd] where s<d as multiple views, and are able to learn separate +1 ··· classifiers on each view. For example, in a lip-reading task the sets of data from visual and audio information are naturally considered as two subsets of the feature vectors (Fig.1.1). Another example, one can consider representations of the same documents in different languages as multiple views (Fig.1.2). Co-training is a semi-supervised multi-view learning technique aiming at im- proving performance of a learning algorithm by expanding labeled training data using information from multiple views [1]. For example, in [2] a set of labeled two-view examples (x,z,y) and a set of unlabeled two-view examples (x, z) { } { } are available. An assumption in [2] is that data from each of the views is sufficient for training an accurate classifier if labeled data are sufficient on both views. In [2], two classifiers are iteratively trained on two sets of pairs (x, y) i and (z,y) i re- { } { } spectively which come from the available triplets (x,z,y) i to label the unlabeled { } examples with the most confident classifier. 2 Figure 1.1: Audio and video in a lip reading task can be considered as two views. The goal of transfer learning is to improve the learning performance on the target view using information from the auxiliary view [3]. Regularized multi-task learning is a supervised learning example of transfer learning, where labeled data (x, y) i and (z,y) i are available on both views [4]. Regularized multi-task { } { } learning algorithm transfers information from auxiliary view to the target view to improve classification performance on the target view [4]. Instead of improv- ing classification performance, self-taught clustering aims at clustering a small collection of target unlabeled data with the help of a large amount of auxiliary un- labeled data [5]. Self-taught clustering algorithm clusters the target and auxiliary data simultaneously to allow the feature representation from the auxiliary data to influence the target data through a common set of features [5]. In this work, we focus on a special case of multi-view learning, for which we in- 3 Figure 1.2: An English version and a Chinese version of the same document can be considered as two views. troduce the term surrogate supervision multi-view learning, which aims at perform- ing classification task on target view where labels are unavailable. In surrogate su- pervision learning, two mutually exclusive sets of pairs (xi, zi) i S1 , (xi,yi) i S2 { } ∈ { } ∈ are available in the training data. However, the goal of surrogate supervision multi- view learning is to learn a classifier to predict y given z. The surrogate supervision multi-view learning scenario is different from the scenarios of [6] and [2] in that labeled examples of the desired view are unavailable in the training data. In other words, it is impossible to obtain a classifier that predicts labels for data from Z without using information from . Details of the surrogate supervision multi-view X learning setting are introduced in Chapter 3. In this work, we solve the surrogate supervision multi-view learning problem using discriminative generative models.