Content-Based Video Indexing for Sports Applications Using Integrated Multi-Modal Approach
Total Page:16
File Type:pdf, Size:1020Kb
Content-based Video Indexing for Sports Applications using Integrated Multi-Modal Approach by Dian W. Tjondronegoro, BIT (1st Hons) Submitted in fulfilment of the requirements for the degree of Doctor of Philosophy Deakin University May, 2005 DEAKIN UNIVERSITY CANDIDATE DECLARATION I certify that the thesis entitled content-based video indexing using integrated multi- modal approach submitted for the degree of doctor of philosophy is the result of my own work and that where reference is made to the work of others, due acknowledgment is given. I also certify that any material in the thesis which has been accepted for a degree or diploma by any other university or institution is identified in the text. Full Name.................................................…………………………………. (Please Print) Signed ..................................................................................………………. Date......................................................................................………………. Acknowledgements I was once told that PhD is a long journey of transformation from a ‘novice’ to ‘professional’ researcher. To succeed, an individual can never do it alone; there must be someone who is there for them, providing all the helping hands and supports. I can’t agree more and as for my case, I have a lot of people to thank. First thanks must go to my supervisor, Professor Phoebe Chen, who has been truly inspirational throughout my candidature. She always shows a tireless enthusiasm to meet, discuss, listen and encourage. Her invaluable advices and faith make all the difference, and I have been blessed to find in her all the good qualities of a supervisor. She has also invested herself in this thesis with enormous commitment, despite her many responsibilities, and I am forever grateful for that. Many thanks must go to my co-supervisor, Professor Binh Pham, who has given all the extra guidance and help that prove to be precious in improving the quality of my work. Special thanks are reserved for the Australian government and tax payers who generously provided an APA scholarship which allowed me to concentrate on my work in 2002-2004. For the past 3 years, Deakin University and Queensland University of Technology have provided me nice working places with all the much needed facilities, and services, as well as financial supports including travel allowances and top-up scholarships. Thanks to all the people there who have helped me with many things. I thank my wife Elizabeth for being such a wonderful wife who is always there for me and makes this whole journey so enjoyable. Thanks to all my friends at uni and church who have always given me tremendous supports and encouragement. i ACKNOWLEDGEMENTS Last, but most importantly, I’d like to dedicate this thesis for my mum and dad to express my deepest gratitude. They are the best parents who are so willing in giving me the best in life (including education) without hoping for anything in return. I am also so blessed to have a really loving sister who gave me so much supports and encouragement to do well during this candidature. Above all, I thank my Father in heaven for His everlasting love and faithfulness, and for showing me the true purpose of living that whatever I do is for His glory. Whenever I was short of energy and patience, He always reminded me that “I can do all things through Christ who gives me strength”. ii Contents ACKNOWLEDGEMENTS........................................................................................................................... I CONTENTS ............................................................................................................................................ III ABSTRACT ...........................................................................................................................................VII LIST OF ASSOCIATED PUBLICATIONS ...................................................................................................IX LIST OF ACRONYMS..............................................................................................................................XI LIST OF FIGURES.................................................................................................................................XIII LIST OF TABLES...................................................................................................................................XV CHAPTER 1 INTRODUCTION...................................................................................................... 1 1.1. BACKGROUND AND MOTIVATION .......................................................................................... 1 1.2. AIMS ....................................................................................................................................... 2 1.3. SIGNIFICANCE ......................................................................................................................... 3 1.4. THESIS OUTLINE ..................................................................................................................... 3 CHAPTER 2 CONTENT-BASED VIDEO RETRIEVAL............................................................ 7 2.1. COMPONENTS OF VIDEO RETRIEVAL SYSTEMS...................................................................... 7 2.2. VIDEO SEGMENTATION......................................................................................................... 10 2.2.1. Shot Boundaries Detection ............................................................................................. 10 2.2.2. Scene Detection............................................................................................................... 12 2.3. CONTENT ANALYSIS............................................................................................................. 13 2.3.1. Audio analysis ................................................................................................................. 13 2.3.2. Image Analysis ................................................................................................................ 19 2.3.3. Video Optical Character Recognition ............................................................................ 22 2.4. VIDEO INDEXING .................................................................................................................. 24 2.4.1. Feature-Based Video Indexing ....................................................................................... 25 2.4.2. Annotation-Based Video Indexing .................................................................................. 32 2.5. INDEXING BY BRIDGING SEMANTIC GAP.............................................................................. 38 2.5.1. Query by Example with Relevance Feedback ................................................................ 38 2.5.2. Browsing with compact representation .......................................................................... 40 2.5.3. Domain-specific Strategy................................................................................................ 43 2.6. MODELING VIDEO DATA ...................................................................................................... 49 2.7. VIDEO RETRIEVAL ................................................................................................................ 54 2.8. EVALUATION OF LITERATURE REVIEW ................................................................................ 57 CHAPTER 3 SPORTS VIDEO INDEXING ................................................................................ 61 3.1. WHY SPORTS VIDEO? ........................................................................................................... 61 3.2. HIERARCHICAL ABSTRACTION LAYERS IN SPORTS VIDEO .................................................. 63 3.3. SPORTS VIDEO INDEXING REQUIREMENTS........................................................................... 70 3.3.1. Characteristics of Sports Video ...................................................................................... 71 3.3.2. Users/Applications Requirements Analysis.................................................................... 74 3.3.3. UML-based Model for Sport Videos............................................................................... 84 CHAPTER 4 FRAMEWORK OF SPORTS VIDEO INDEXING AND RETRIEVAL......... 89 4.1. INTRODUCTION ..................................................................................................................... 89 4.2. MULTI-LAYER SEMANTIC CONTENTS EXTRACTION ............................................................ 89 4.3. SYSTEM ARCHITECTURE....................................................................................................... 92 CHAPTER 5 MID-LEVEL FEATURES AND GENERIC SEMANTIC EXTRACTION.... 95 5.1. INTRODUCTION ..................................................................................................................... 95 5.2. ALGORITHMS FOR MID-LEVEL FEATURE EXTRACTION ....................................................... 96 5.2.1. Whistle Detection ............................................................................................................ 96 5.2.2. Excitement Detection ...................................................................................................... 99 5.2.3. Inserted-Caption Detection........................................................................................... 103 5.2.4. Near Goal-Area Detection...........................................................................................