AI-powered Classification and Query of Color Patterns With Applications to Movie Pictures Master Thesis August 2020 Linda Samsinger E-mail: [email protected] Student ID: 08-338-386 Supervisors: Prof. Dr. Renato Pajarola Prof. Dr. Barbara Fluckiger¨ Visualization and MultiMedia Lab Dr. Giorgio Trumpy Department of Informatics Gaudenz Halter, Dr. Alexandra Diehl University of Zurich¨ To Ernst Samsinger, my grandfather ii Abstract An exhaustive evaluation of 29 multi-class and multi-label classification algorithms for mapping self-specified color name categories to all color space values in the CIE-L*ab color solid enables an effective color-aware search system. Based on these classified colors, higher-chromatic patterns from color theory and its rules, such as color contrasts, can be detected in a repository of movie pictures by an exploratory attempt to concretize their scientific definitions from the realm of art. Color histograms are drawn indirectly from color palettes instead of images for pairwise histogram similarity computation. Hence, a retrieval system involving three components is built: (a) a query of colors in images or their color palettes, (b) their top-n similarity and (c) their automated color contrast annotation. The implementation of the proposed method is conducted on the ERC FilmColors project’s sample movie Jigokumon which consists of 569 subsequently shot video frames. A best macro F1-score of 92.7% was achieved using an Extra Trees classifier on Gaussian multi-label color classification which outperforms other task-adapted classifiers in this line of research. The resulting system is adaptable to digital movie databases (DMDb) with implications for 21st-century cinematography. iii Acknowledgement I would like to thank my project supervisors Dr. Prof. Renato Pajarola and Dr. Prof. Barbara Fluckiger¨ for providing me with a thesis topic that sparked a long-lasting interest in colors. Their guidance and mentorship throughout the project duration have been essential to its successful completion. I would also like to extend my gratitude to my supervisors Dr. Alexandra Diehl and Dr. Giorgio Trumpy, who shared their insights about color visualization with me. Special thanks go to Gaudenz Halter for his support and the candid discussions on technical issues, who reached out to help according to the need of the hour to bring this piece of work to its full fruition. iv Contents Abstract iii Acknowledgement iv 1. Introduction 1 1.1. Goal of the Thesis...........................................2 1.2. Project Description..........................................3 1.3. Problem Statement..........................................4 1.4. Chapter Overview...........................................5 2. Motivation 6 2.1. ERC FilmColors Project.......................................6 2.2. VIAN tool...............................................6 2.3. Related Work.............................................7 2.3.1. Task 1: Color-based machine classification.........................7 2.3.2. Task 2: Content-based Image Retrieval (CBIR).......................8 2.3.3. Task 3: Color contrast....................................9 3. Background 10 3.1. Concepts............................................... 10 3.2. Architecture.............................................. 12 4. Methodology 15 4.1. Task 1: Color Name Prediction.................................... 15 4.1.1. Data Augmentation...................................... 16 4.1.2. Classification Methods.................................... 20 4.1.3. Machine Learning...................................... 24 4.1.4. An Example: Multi-class Thesaurus-VIAN......................... 27 4.1.5. An Example: Multi-label Thesaurus-ITTEN......................... 30 4.2. Task 2: Image Similarity Calculation................................. 32 4.2.1. Histogram Conversion.................................... 33 4.2.2. An Example: Jigokumon’s Movie Picture Similarity.................... 33 4.3. Task 3: Color Contrast Estimation.................................. 35 4.3.1. Color Contrasts........................................ 35 4.3.2. Classification Rules...................................... 36 4.3.3. An Example: Jigokumon’s Color Contrasts......................... 38 5. Implementation 40 5.1. Functionalities............................................. 40 5.2. Components.............................................. 41 5.3. Challenges............................................... 42 6. Results 43 6.1. Task 1: Querying Color Names.................................... 43 6.2. Task 2: Querying Similar Images................................... 45 v Contents 6.3. Task 3: Querying Color Contrasts.................................. 48 7. Discussion 50 7.1. Task 1: Color Classification...................................... 50 7.2. Task 2: Color-based Image Similarity................................ 51 7.3. Task 3: Color Contrast Classification................................. 51 8. Conclusion 52 A. Appendix I 54 A.1. Color Designation........................................... 54 A.2. Basic Color Sets............................................ 55 A.3. Visualizing Basic Color Categories.................................. 57 A.4. Transforming Basic Color Categories................................ 60 A.5. Color Naming Systems........................................ 60 A.5.1. Werner’s Nomenclature of Colors.............................. 60 A.5.2. Sundberg Color Thesaurus.................................. 61 A.5.3. EPFL Color Thesaurus.................................... 61 A.5.4. Linguistic Color Name Clusters............................... 61 A.5.5. Color Name Dictionary.................................... 64 A.5.6. Fully-fledged Color Name Dictionary............................ 65 A.5.7. Extended Fully-fledged Color Name Dictionary....................... 65 A.6. Color Spaces............................................. 66 A.7. Color Palettes............................................. 66 A.8. Color Theorists............................................ 68 A.9. Color Extraction............................................ 70 B. Appendix II 75 B.1. Multi-class Classification Strategies................................. 75 B.2. Color Contrasts............................................ 76 B.3. Color Contrast Classification..................................... 79 C. Appendix III 82 C.1. Color Conversion Tools........................................ 82 C.2. Color Palette Search Tools...................................... 82 C.3. Web Tool Mockups.......................................... 91 C.4. Web-app Framework......................................... 96 C.5. Color Intepretations.......................................... 96 References 102 vi 1. Introduction This thesis summarizes a semestral research project on color analysis that lies at the cross-section of the natural sciences and the arts. While color has been the subject of many books from ancient philosophers to present-day publications, with the advent of artificial intelligence in the 21st century, the golden rules of harmony and balance from color theory are again open for new reinterpretations. These developments have taken place, for example, in the emerging field of the digital humanities. A digital archive of images taken from the artistic works of the grandmasters of cinema can be annotated with color-based labels, analyzed with sophisticated statistics and queried by using indexing methodologies from computer science. The contributions have been to design efficient colorimetric mechanisms by leveraging machine learning for color classification to query a database of digital movie images and color palettes. Furthermore, the project involved a good chunk of data analysis conducted on color dictionaries and images, prototyping, testing and deployment of a color-aware search system. In the age of big data, many databases have recorded a steady increase of multimedia records that harbor many challenges for matching, search and retrieval systems for automated labeling and search query formulation of images. Especially in film analysis and visual search, color management is critical to assessing the visual styles, moods and emotions of the predominant color scheme of color palettes used as a hallmark indexing method for analyzing movies. Color may serve as the common denominator shared among cinematographers to convey a certain underlying message that can be identified using color-based search functions. Nowadays, it has become commonplace to query large collections of digital image archives based on a queried color of interest. Since color is the single most important low-level cue attracting a viewer’s attention when watching a movie, other elements of art such as lines, forms, textures and shapes can easily be neglected. Colors pop out first and capture the attention of the viewer, hence their centrality in feature selection. However, the role of mapping intersubjective color names to an all-numerical color space remains a challenge. A considerable step forward may be to use machine learning methods to accomplish color classification. As such, colors are constitutive of color palettes from which higher-level chromatic patterns such as color contrasts can be derived, automating the color classification process to the highest accuracy possible could gain high significance in the arts. From color palettes extracted from an image, not only their color categories can be predicted, but also their similarity. The color distribution of an image is summarized using color