NEW INITIALIZATION STRATEGY FOR NONNEGATIVE MATRIX FACTORIZATION A Thesis Presented By Yueyang Wang to The Department of Electrical and Computer Engineering In partial fulfillment of requirements for the degree of Master of Science in the field of Electrical & Computer Engineering Northeastern University Boston, Massachusetts May 2018 ii ABSTRACT Nonnegative matrix factorization (NMF) has been proved to be a powerful data representation method, and has shown success in applications such as data representation and document clustering. In this thesis, we propose a new initialization strategy for NMF. This new method is entitled square nonnegative matrix factorization, SQR-NMF. In this method, we first transform the non-square nonnegative matrix to a square one. Several strategies are proposed to achieve SQR step. Then we take the positive section of eigenvalues and eigenvectors for initialization. Simulation results show that SQR-NMF has faster convergence rate and provides an approximation with lower error rate as compared to SVD-NMF and random initialization methods. Complementing different elements in data matrix also affect the results. The experiments show that complementary elements should be 0 for small data sets and mean values of each row or column of the original nonnegative matrix for large data sets. Key words: Nonnegative matrix factorization, complementary elements iii ACKNOWLEDGEMENT I would like to express my sincere gratitude to my thesis advisor, Professor Shafai for his patience, motivation, and immense knowledge. His guidance helped me a lot in all the time of research and writing of the thesis. I could not have imagined having a better advisor for my master study. Besides my advisor, I would like to thank the rest of my thesis committee: Professor Nian Sun and Professor Vinay Ingle, for their insightful comments and encouragement. Finally, I would like to express my profound gratitude to my parents for providing me with support and continuous encouragement throughout the writing of this thesis and my life in general. iv TABLE OF CONTENTS ABSTRACT ........................................................................................................................ ii LIST OF TABLES ............................................................................................................. vi LIST OF FIGURES .......................................................................................................... vii Chapter 1 Introduction ........................................................................................................ 1 Chapter 2 Nonnegative Matrix............................................................................................ 3 2.1. Nonnegative Matrices and Perron-Frobenious Theorem ......................................... 3 2.2. Reducible matrix ...................................................................................................... 4 2.3. Irreducible Matrix .................................................................................................... 5 2.4. Primitive Matrix ....................................................................................................... 5 2.5. Stochastic matrix ...................................................................................................... 6 2.6. M-Matrix .................................................................................................................. 6 2.7. Metzler Matrix ......................................................................................................... 7 2.8. Symmetric matrix..................................................................................................... 8 2.9. Non-square Nonnegative Matrix.............................................................................. 8 Chapter 3 SQR-NMF for the initialization of NMF ......................................................... 10 3.1. Nonnegative Matrix Factorization ......................................................................... 10 3.2. SVD based initialization: SVD-NMF .................................................................... 12 3.3. Eigenvalue decomposition based initialization: SQR-NMF .................................. 13 3.3.1 SQR-NMF initialize with eigenvalue and eigenvector .................................... 14 3.3.2 SQR-NMF initialize with symmetric matrix ................................................... 14 Chapter 4 Simulation Results and Comparison ................................................................ 16 4.1. Simulation results for MU algorithm ..................................................................... 16 v 4.1.1 Comparison w.r.t fixed factorization rank and increasing iteration times ....... 16 4.1.2 Comparison w.r.t fixed iteration time and increasing factorization rank ........ 17 4.1.3 Reconstruction images ..................................................................................... 18 4.2. Simulation results for divergence-reducing algorithm .......................................... 19 4.2.1 Comparison w.r.t fixed factorization rank and increasing iteration times ....... 19 4.2.2 Comparison w.r.t fixed iteration time and increasing factorization rank ........ 19 4.3. Numerical results for different complementary elements ...................................... 20 4.3.1 Comparison w.r.t fixed factorization rank and increasing iteration times ....... 20 4.3.2 Reconstruction images ..................................................................................... 21 Chapter 5 Conclusion ........................................................................................................ 23 REFERENCES ................................................................................................................. 24 APPENDIX ....................................................................................................................... 26 vi LIST OF TABLES Table 1 Factorization rank of processing 1 image with different extraction ratio ............ 13 Table 2 Factorization rank of processing 5 images with different extraction ratio .......... 13 vii LIST OF FIGURES Figure 1 Errors of reconstruction using MU algorithm .................................................... 16 Figure 2 Errors of reconstruction using MU algorithm .................................................... 17 Figure 3 Errors of reconstruction using MU algorithm with fixed iteration ..................... 17 Figure 4 Reconstruction images....................................................................................... 18 Figure 5 Errors of reconstruction using divergence-reducing algorithm .......................... 19 Figure 6 Errors using divergence-reducing algorithm with fixed iteration ...................... 20 Figure 7 Errors of adding different complementary elements .......................................... 21 Figure 8 Reconstruction images........................................................................................ 22 1 Chapter 1 Introduction Nonnegative matrix factorization (NMF) has become a widely used method in analyzing large datasets, since it extracts features from a large set of data vectors. It is a kind of factorization that constrains the elements of both components and the expansion coefficients to be non-negative. Non-negative matrix factorization is a useful method in reducing the dimension of large datasets. In the paper Learning the parts of objects by non-negative matrix factorization published by Lee and Seung[1], they presented how NMF could learn parts of objects in facial images. It is intuited by the idea of combining the parts to form the whole. Negative values is admissible, but lose physical meaning in practice. The factorization of matrices representing complex multidimensional datasets is the basis of several commonly applied techniques for pattern recognition and unsupervised clustering. Similarly to principal components analysis or independent component analysis, the objective of non-negative matrix factorization (NMF) is to explain the original data with limited basis components and coefficients, which when combined together approximate the original data as accurately as possible. The NMF specializes in that it constrained both the matrix representing the basis components as well as the matrix of mixture coefficients to have non-negative entries, and that no orthogonality or independence constraints are imposed on the basis components. This leads to a simple and intuitive interpretation of the factors in NMF, and allows the basis components to overlap. Because of its definition[1], the NMF method has been successfully applied in several fields including image recognition and pattern recognition, signal processing and text mining[2]. NMF has also been applied in biological industries. It is used to obtain new insights into cancer type discovery based on gene expression microarrays[3], for the functional characterization of genes[4], to predict cis-regulating elements from positional word count matrices[5] and, for phenotype prediction using cross-platform microarray data[6]. The popularity of the NMF approach derives essentially from three properties that distinguish it from standard decomposition techniques. 2 Firstly, the matrix factors are nonnegative by definition, which allows their intuitive interpretation as real underlying components within the context defined by the original data. The basic components can be directly interpreted as parts or basis samples,