D2.1 State of the Art on Multimedia Search Engines
Total Page:16
File Type:pdf, Size:1020Kb
Chorus D2.1 D2.1 State of the Art on Multimedia Search Engines Deliverable Type * : : PU Nature of Deliverable ** : R Version : Released Created : 23-11-2007 Contributing Workpackages : WP2 Editor : Nozha Boujemaa Contributors/Author(s) : Nozha Boujemaa, Ramon Compano, Christoph Doch, Joost Geurts, Yiannis Kampatsiaris, Jussi Karlgren, Paul King, Joachim Koehler, Jean-Yves Le Moine, Robert Ortgies, Jean-Charles Point, Boris Rotenberg, Asa Rudstrom, Nicu Sebe. * Deliverable type: PU = Public, RE = Restricted to a group of the specified Consortium, PP = Restricted to other program participants (including Commission Services), CO= Confidential, only for members of the CHORUS Consortium (including the Commission Services) ** Nature of Deliverable : P= Prototype, R= Report, S= Specification, T= Tool, O = Other. Version : Preliminary, Draft 1, Draft 2,…, Released Abstract: Based on the information provided by European projects and national initiatives related to multimedia search as well as domains experts that participated in the CHORUS Think-thanks and workshops, this document reports on the state of the art related to multimedia content search from, a technical, and socio-economic perspective. The technical perspective includes an up to date view on content based indexing and retrieval technologies, multimedia search in the context of mobile devices and peer-to-peer networks, and an overview of current evaluation and benchmark inititiatives to measure the performance of multimedia search engines. From a socio-economic perspective we inventorize the impact and legal consequences of these technical advances and point out future directions of research. Keyword List: multimedia, search, content based indexing, benchmarking, mobility, peer to peer, use cases, socio-economic aspects, legal aspects The CHORUS Project Consortium groups the following Organizations: 1 Chorus D2.1 JCP-Consult JCP F Institut National de Recherche en Informatique et Automatique INRIA F Institut fûr Rundfunktechnik GmbH IRT GmbH D Swedish Institute of Computer Science AB SICS SE Joint Research Centre JRC B Universiteit van Amsterdam UVA NL Centre for Research and Technology - Hellas CERTH GR Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e. V. FHG/IAIS D Thomson R&D France THO F France Telecom FT F Circom Regional CR B Exalead S. A. Exalead F Fast Search & Transfer ASA FAST NO Philips Electronics Nederland B.V. PHILIPS NL 2 Chorus D2.1 Table of Contents TABLE OF CONTENTS ................................................................................................................................................. 3 1. INTRODUCTION................................................................................................................................................... 6 2. USER INTERACTION .......................................................................................................................................... 7 2.1. SUMMARY ........................................................................................................................................................ 7 2.2. USER INVOLVEMENT IN THE SYSTEM DEVELOPMENT PROCESS ......................................................................... 7 2.2.1. User centered design .................................................................................................................................. 8 2.2.2. The HCI perspective ................................................................................................................................... 8 2.3. USER GENERATED CONTENT............................................................................................................................. 9 2.4. USER CENTERED DESIGN OF MULTI -MEDIAL INFORMATION ACCESS SERVICES ................................................ 10 2.4.1. Beyond "relevance" as a target notion - How can we formalise our understanding of what users are up to? 10 2.4.2. Use cases as a vehicle............................................................................................................................... 11 2.5. USE CASES IN CURRENT RESEARCH PROJECTS ............................................................................................... 12 2.6. USE -CASES AND SERVICES ............................................................................................................................. 14 2.7. OVERVIEW : THE CHORUS ANALYSIS OF USE CASES .................................................................................... 14 2.8. PARTICIPATING PROJECTS .............................................................................................................................. 15 2.9. DIMENSIONS OF DATA ANALYSIS ................................................................................................................... 15 2.10. DATA ANALYSIS ............................................................................................................................................ 15 2.10.1. Actions.................................................................................................................................................. 15 2.10.2. Corpora................................................................................................................................................ 17 2.10.3. Methodologies (Technical Requirements)............................................................................................ 19 2.10.4. Products ............................................................................................................................................... 21 2.10.5. Users .................................................................................................................................................... 24 2.10.6. User Classes......................................................................................................................................... 26 2.11. CONCLUSION AND FUTURE PROSPECTS .......................................................................................................... 27 2.12. REFERENCES ................................................................................................................................................... 27 3. STATE OF THE ART IN AUDIO-VISUAL CONTENT INDEXING AND RETRIEVAL TECHNOLOGIES.......................................................................................................................................................... 29 3.1. INTRODUCTION ............................................................................................................................................... 29 3.2. RECENT WORK ............................................................................................................................................... 30 3.2.1. Learning and Semantics............................................................................................................................ 31 3.2.2. New Features & Similarity Measures....................................................................................................... 33 3.2.3. 3D Retrieval.............................................................................................................................................. 35 3.2.4. Browsing and Summarization................................................................................................................... 36 3.2.5. High Performance Indexing...................................................................................................................... 36 3.3. SUMMARY OF MULTIMEDIA ANALYSIS IN EUROPEAN RESEARCH .................................................................. 37 3.3.1. Multimedia Analysis in European Projects............................................................................................... 37 3.3.2. Multimedia Analysis in National Initiative ............................................................................................... 38 3.3.3. State-of-the Art in European Research ..................................................................................................... 39 3.4. FUTURE DIRECTIONS ...................................................................................................................................... 43 3.5. REFERENCES ................................................................................................................................................... 44 ANNEX A: OVERVIEW OF THE 9 IST PROJECTS ............................................................................................................. 51 MODULES ON SPEECH /A UDIO INDEXING AND RETRIEVAL ............................................................................................ 61 MODULES ON IMAGE INDEXING AND RETRIEVAL ......................................................................................................... 64 MODULES ON 3D INDEXING AND RETRIEVAL ............................................................................................................... 65 MODULES ON VIDEO INDEXING AND RETRIEVAL ......................................................................................................... 65 MODULES ON TEXT INDEXING AND RETRIEVAL ........................................................................................................... 72 ANNEX B: OVERVIEW OF THE NATIONAL RESEARCH PROJECTS ............................................................ 74 4. SOA OF