Motion Compensated Deinterlacer Analysis and Implementation
Total Page：16
File Type：pdf, Size：1020Kb
2008:126 CIV MASTER'S THESIS Motion Compensated Deinterlacer Analysis and Implementation Johan Helander Luleå University of Technology MSc Programmes in Engineering Arena, Media, Music and Technology Department of Computer Science and Electrical Engineering Division of Signal Processing 2008:126 CIV - ISSN: 1402-1617 - ISRN: LTU-EX--08/126--SE Master’s Thesis Supervisor: Magnus Hoem Examiner: Magnus Lundberg Nordenvaad Telestream AB & Department of Computer Science and Electrical Engineering, Signal Processing Group, Luleå University of Technology Preface This Master’s Thesis was carried out by me during the autumn term 2007 and beginning of 2008 at Telestream AB’s office at Rådmansgatan 49, Stockholm. It is part of the Master of Science program Arena Media, Music and Technology at Luleå University of Technology (LTU). Because of my education and interest in signal processing in media applications, the proposed topic was very well suited. The reader is assumed having basic knowledge about signal processing, such as sampling, quantization, aliasing and so on. I would like to thank Telestream AB for their warm welcome and comfortable treatment during this period. I would especially like to thank Magnus Hoem, CEO Telestream AB, for the opportunity to carry this thesis through, Nils Andgren, Telestream AB, for help and support through important thoughts and discussions, Kennet Eriksson, Telestream AB, for supplying test video sequences. Finally, I would like to thank Maria Andersson, for great support by illustration of the majority of the figures contained in this Master’s Thesis. i ii Abstract In the early days of television as Cathode Ray Tube (CRT) screens became brighter, the level of flicker caused by progressive scanning became more noticeable. This is because the human visual system is sensitive to large-area flicker. Interlaced scanning was invented in 1932 as a redeem to this difficulty. In contrast to progressive scanning, where every line is drawn in sequence, interlaced scanning alternates lines of a frame in half a frame interval, called a field. The conversion process from interlace scan to progressive scan is called deinterlacing. In this thesis, two deinterlacing methods, which use motion information from the video sequence, were used in conjunction to obtain an improved result. Thus, the process of finding the true motion on the interlaced video sequence also had to be analyzed. The analysis was done on artificially generated test sequences, as well as true video sequences. The result was measured using the Mean Square Error between a progressive input sequence and the deinterlaced output sequence. This measurement was compared to a much simpler deinterlacing algorithm and showed large improvements, primarily in sense of aliasing. However, in some cases, the deinterlacer produced severe artifacts causing picture degradation. iii iv Contents 1 Introduction 9 1.1 Background . 9 1.2 Problem description . 10 1.3 Purpose . 11 1.4 Limitations . 11 2 Prerequisites 13 2.1 Digital image and video signals . 13 2.1.1 Two-dimensional sampling . 13 2.1.2 Temporal sampling . 13 2.1.3 Temporal alias. 14 2.2 Generalized Sampling Theorem . 15 2.3 Deinterlacing . 15 2.3.1 Line Averaging. 17 2.4 Temporal redundancy . 17 2.4.1 Block-matching motion estimation . 17 2.4.2 Motion compensation . 19 3 Motion estimation on interlaced video 21 3.1 The 3-D Recursive-Search Block-Matcher . 21 3.1.1 Variable 3-D Recursive-Search Block-Matcher. 23 3.2 Time-Recursive motion estimation. 25 3.3 GST based motion estimation . 26 3.3.1 Applying a first-order linear interpolation . 30 v CONTENTS 4 Motion compensated deinterlacing 31 4.1 Time-Recursive deinterlacing . 31 4.2 GST based deinterlacing. 32 4.2.1 The GST interpolation filter . 32 4.2.2 Applying a first-order linear interpolation . 35 4.3 GST deinterlacing with Recursive GST motion estimation 36 4.4 Robust GST deinterlacing . 37 4.5 Two-dimensional extension . 39 5 Results 41 5.1 Evaluation of the RGST motion estimator and GST deinterlacer . 41 5.1.1 Performance on artificial test sequences. 41 5.1.2 Performance on true video sequences . 43 5.1.3 Subjective analysis . 46 6 Discussion 49 6.1 Problems and improvements. 49 6.1.1 Edge difficulties. 49 6.1.2 Singularity problem. 49 6.1.3 Improved fall-back method . 50 6.1.4 Improved interpolation . 50 6.1.5 Further robustness . 50 6.1.6 Two dimensional GST interpolation . 50 6.1.7 Chrominance deinterlacing . 51 7 References 53 vi vii viii 1 Introduction 1.1 Background Progressive scanning, also known as sequential scanning, is the method for displaying, storing or transmitting moving images, frames, where every horizontal line of each frame is drawn in sequence (see Figure 1). In the early days of television as Cathode Ray Tube (CRT) screens became brighter, the level of flicker caused by progressive scanning became more noticeable. This is because the human visual system is sensitive to large-area flicker. An increased refresh rate would have solved this perception problem, but would also have consumed a larger amount of bandwidth. Also, CRT screens at that time limited the amount of frames to be displayed per second. In 1932 the interlace technology was invented by Radio Corporation of America engineer Randall C. Ballard [1] partly as a redeem to the flicker problem. The technique improves the picture quality by removing large-area flicker without consuming any extra bandwidth. In contrast to progressive scanning, where every line is drawn in sequence, interlaced scanning alternates lines of a frame in half a frame interval, called a field. Consequently, two fields form one frame. One field contains all the odd lines of the image while the other field contains all the even lines of the image as shown in Figure 2. The afterglow of the phosphor of the CRT screen, in combination with the persistence of vision results in two fields being perceived as a continuous image. Hence, interlace makes it possible to view full horizontal resolution with half the bandwidth which would be required 9 1. InTRODuCTIOn fs f n s -1 2 n-1 n n Frame Field Figure 1. A sequence of progressive Figure 2. A sequence of interlaced fields. frames. for a progressive scan image, while maintaining the necessary refresh rate to prevent large-area flicker. Because of the achieved compromise between quality and required bandwidth, interlacing has been used exclusively until the introduction of computer monitors in the 1970s. 1.2 Problem description As described, traditional CRT screens are natively designed for interlace scanning. Recent display technologies like Liquid Crystal Display (LCD) and Plasma Display Panel (PDP) on the other hand, requires progressive scan video to display correctly1. The conversion process from interlace scan to progressive scan is called deinterlacing. This can be done in numerous ways but will here be focused on a motion compensated method. Motion compensation uses estimated motion vectors to realize an improved result2. Interlace scanning complicates this estimation though, as well as several other image-processing tasks. 1 Assuming the response time of these technologies could be made fast enough, interlace scan on such a display would result in a halving of brightness due to half of the pixels remaining black every other field. Consequently, conversion from interlaced to progressive scan is necessary. 2 See section 2.4 for further details on motion compensation, motion estimation and motion vectors. 10 1. InTRODuCTIOn 1.3 Purpose The main purpose of this thesis was to achieve superior-quality deinterlacing of interlaced video signals. Since this process can gain from motion compensation, the process of motion estimation on interlaced scanning also had to be investigated. 1.4 Limitations A block-matching motion estimator can be divided into a matching criterion and a search strategy. Because of its massive line of research, the focus was not on search strategy in this thesis. For straightforward and smooth analysis, Matlab was used as the tool for implementation of researched methods. The result does only take grayscale video into account due to simplification and computational time. 11 12 2 Prerequisites 2.1 Digital image and video signals Digital processing requires the signal to be digitally represented through sampling and quantization. This can be further studied in [2]. 2.1.1 Two-dimensional sampling One-dimensional sampling can be extended to two-dimensional sampling by adding an additional dimension. A gray-scale image can then be digitally represented on a sample lattice by sampling the continuous intensity values along two orthogonal axes x and y (see Figure 3). Figure 3. Matlab mesh plot of an image containing a bicycle. 13 2. PREREquISITES y Figure 4. Vertical- temporal diagram of a progressive sample lattice. n n-2 n-1 n 2.1.2 Temporal sampling When a sequence of images is observed, this is perceived as a continuous scene by the human visual system [3]. To be able to represent moving images, this is exploited in the line of video technology by capturing, transmitting and storing image sequences, i.e. video. This is done just like sampling of static images, but adding an additional dimension, time. For a moving scene, the two-dimensional array of samples is repeatedly collected for various discrete-time instances arriving at a three-dimensional array of intensity samples (see Figure 4). The intensity value of a pixel in position on the sample lattice and image number n can then be expressed by the function fp [x ,n] ( p for progressive). 2.1.3 Temporal alias Consider a white bar moving in the horizontal direction between temporal samples. This motion will construct a variation of intensity over time if observing a specific position x .