SIGNATURE SYSTEM FOR VIDEO IDENTIFICATION by Sebastian Possos Medellin A Thesis Submitted to the Faculty of The College of Engineering and Computer Science in Partial Fulfillment of the Requirements for the Degree of Master of Science Florida Atlantic University Boca Raton, Florida August 2010 ACKNOWLEDGMENTS I would like to thank my wife Adriana, because without her I would not have been able to reach this far and keep going. Also to Dr. Hari Kalva, who showed me a new world of knowledge, exciting and full of challenges, his guidance and friendship make this work possible. To Dr. Maria Petrie, who gave me the opportunity to be in this excellent institution and Dr. Oge Marques and Dr. Bassem Alhalabi for their help and support by being part of my Thesis Committee. Finally I would like to thank my family, friends and colleagues who were always present, not only during this time but always with me, giving me their support and giving me a hand when I needed it. iii ABSTRACT Author: Sebastian Possos Medellin Title: Signature system for video identification Institution: Florida Atlantic University Thesis Advisor: Dr. Hari Kalva Degree: Master of Science Year: 2010 Video signature techniques based on tomography images address the problem of video identification. This method relies on temporal segmentation and sampling strategies to build and determine the unique elements that will form the signature. In this thesis an extension for these methods is presented; first a new feature extraction method, derived from the previously proposed sampling pattern, is implemented and tested, resulting in a highly distinctive set of signature elements, second a robust temporal video segmentation system is used to replace the original method applied to determine shot changes more accurately. Under a very exhaustive set of tests the system was able to achieve 99.58% of recall, 100% of precision and 99.35% of prediction precision. iv SIGNATURE SYSTEM FOR VIDEO IDENTIFICATION TABLES .......................................................................................................................... viii FIGURES ........................................................................................................................... ix EQUATIONS ..................................................................................................................... xi Chapter 1 INTRODUCTION .......................................................................................... 1 1.1 Overview and Motivation ........................................................................................ 1 1.2 Problem Statement and Objective ........................................................................... 2 1.3 Main Contributions ................................................................................................. 4 1.4 Overview of the Thesis ............................................................................................ 5 Chapter 2 BACKGROUND AND RELATED WORK .................................................. 6 2.1 Introduction............................................................................................................. 6 2.2 Video Identification ................................................................................................. 6 2.2.1 Digital Watermarks ......................................................................................... 7 2.2.2 Video content based identification .................................................................. 7 2.2.3 Signature based identification ......................................................................... 9 2.3 Video Tomography ................................................................................................ 17 v 2.4 Advertisement auditing systems ............................................................................ 20 Chapter 3 PROPOSED SOLUTION ............................................................................. 22 3.1 Introduction........................................................................................................... 22 3.2 General Description .............................................................................................. 22 3.3 Signature design .................................................................................................... 23 3.3.1 First Video Signature version ....................................................................... 25 3.3.2 Second Video Signature version ................................................................... 37 Chapter 4 IMPLEMENTATION ................................................................................... 42 4.1 Introduction........................................................................................................... 42 4.2 Signature extraction .............................................................................................. 43 4.2.1 FFMPEG libraries ......................................................................................... 48 4.2.2 Intel IPP 6.0 Canny Edge detector ................................................................ 49 4.3 Shot Change Detector flow chart. ......................................................................... 50 Chapter 5 EXPERIMENTS AND RESULTS ............................................................... 51 5.1 Introduction........................................................................................................... 51 5.2 MPEG Experiments .............................................................................................. 51 5.2.1 Data set.......................................................................................................... 52 5.2.2 Procedure for testing ..................................................................................... 53 5.2.3 Uniqueness test ............................................................................................. 53 5.2.4 Robustness test .............................................................................................. 60 vi 5.3 Content Auditing ................................................................................................... 72 5.3.1 Experiment setup .......................................................................................... 72 5.3.2 Algorithm ...................................................................................................... 73 5.3.3 Results ........................................................................................................... 73 5.3.4 Summary ....................................................................................................... 75 Chapter 6 CONCLUSIONS AND FUTURE WORK ................................................... 76 6.1 Conclusions ........................................................................................................... 76 6.2 Future Work .......................................................................................................... 77 BIBLIOGRAPHY ............................................................................................................. 79 vii TABLES Table 1. Uniqueness Test Performance Summary ............................................................ 57 Table 2. Uniqueness Test Time complexity Performance ................................................ 57 Table 3. Modified Uniqueness Test Performance Summary ............................................ 58 Table 4. Uniqueness test Performance Comparison ......................................................... 59 Table 5. Video Modifications and Levels for Robustness Assessment ............................ 60 Table 6. Content audit: Signature comparison results for signatures of Type 1. .............. 74 Table 7. Content audit: Signature comparison results for signatures of Type 2. .............. 74 viii FIGURES Figure 1. Region pair distribution signature extraction – NEC Corp. proposal ............... 10 Figure 2. Neighborhood and pixel relationship – Mitsubishi Electric proposal ............... 11 Figure 3. Multi-resolution signature structure – Mitsubishi Electric proposal ................. 13 Figure 4. Temporal hierarchical structure – University of Bresia proposal ..................... 14 Figure 5. Saliency image generation – Peking University proposal for MPEG ............... 15 Figure 6. Saliency based signature extraction – Peking University proposal ................... 15 Figure 7. Mitsubishi Electric – NEC Corp. video signature bitstream ............................. 17 Figure 8. Video Tomography Extraction .......................................................................... 18 Figure 9. Video tomography extraction for 1 of 6 components ........................................ 24 Figure 10. Tomography line pattern ................................................................................. 24 Figure 11. Tomography, edge and composite images ...................................................... 27 Figure 12. Sub-signature extraction: Level changes along lines ...................................... 28 Figure 13. Sub-signature extraction: block or area assignment ........................................ 28 Figure 14. Tomography image with a shot change present .............................................. 29 Figure 15. Shot Signature Process .................................................................................... 30 Figure 16. Frame signature extraction .............................................................................