Motion adaptive deinterlacing filter implemented in hardware and OpenCL

Stefan Frid, Gustav Peterz February 8, 2013

During most of television history, CRT-based is still being used in some applications, video fil- displays has been the most common way of dis- ters which improve the appearance of interlaced playing moving images. Consequently, means content on non-CRT displays are used. Such fil- of broadcasting video have been adapted to the ters are called deinterlacing filters. Naive ones characteristics of CRT displays, and interlacing either sacrifice resolution and create flickering, is one example of this. The need to continu- or produce distinct visible artefacts on content ously refresh the image of a CRT results in a with motion as seen in figure 2. Thus, there is requirement of a high refresh rate, and interlac- a need for more complex filters that account for ing was introduced as a means of halving the motion and other dynamic properties in video consumed bandwidth at the cost of vertical res- content. olution, while maintaining a high refresh rate. Interlaced video consists of a sequence of im- ages known as fields, each with half the vertical resolution of the original frames. Every other field is created by discarding all odd rows in a frame, and the remaining fields are created by discarding the even rows (see figure 1).

Figure 2 – Example of interlaced Figure 1 – A sequence of four frames video that was processed by the and their corresponding fields after naive deinterlacing filter ”weave”. converting a video from progressive In content with considerable motion to interlaced. this filter often produce the charac- teristic artefacts known as ”comb- Today the CRT display is no longer in com- ing” or ”jaggies”. mon use, replaced with technologies such as LCD and plasma displays. These display types As the complexity of the filters grow, so does have different characteristics than CRT dis- the computing power needed to run the algo- plays, and are not well suited for the display of rithm. One use case where performance is a interlaced video. However, as interlaced video concern is when the algorithm is performed by

1 a low power device, such as a phone. Once video produced by MEDA is of equal or better relegated to voice calls and messages, modern quality than that produced by other deinterlac- smartphones may be connected to high defini- ing filters suitable for real-time use (figure 3). tion displays for video playback, creating a situ- ation in which deinterlacing might be needed on a phone. The hardware powering such phones is also finding use in digital video players intended to be used with televisions, providing another application for low power deinterlacing. To meet the demands on modern smart- phones and tablets as all-purpose appliances for gaming and media consumption, such de- vices are equipped with powerful coprocessors to offload processor-intensive tasks from the main processor. One such coprocessor is the GPU, the Graphics Processing Unit, which is specifically designed to process and manipulate Figure 3 – Filter image quality, as graphics. Compared to general-purpose CPUs, measured in PSNR, compared to a modern GPUs are very efficient at perform- number of other deinterlacing algo- ing multiple tasks simultaneously, making them rithms over a series of video clips. more effective for algorithms where processing of large blocks of data is done in parallel. In The first main feature of MEDA is its abil- general-purpose applications where the GPU is ity to detect motion in the video, and use this used for non-graphics related tasks, a frame- information to perform video filtering. While work such as OpenCL is often used. OpenCL naive methods of motion detection are suit- is an open standard for running parallel tasks able for some applications, MEDA uses a novel on CPUs, GPUs and hardware accelerators. It method designed to reduce erroneous static is cross-platform and supported on most major area detection in situations where this would operating systems, including various embedded result in noticeable visual artifacts. and handheld devices. Leveraging this functionality, the thesis re- port ”Motion adaptive deinterlacing filter im- plemented in hardware and OpenCL” describes the design and implementation of a new dein- terlacing method targeting mobile and low- power devices. The search for a suitable algorithm is de- scribed in the report. Algorithm character- istics such as complexity and data dependen- cies had to be taken into account, in order to Figure 4 – Image showing the dif- enable implementations in OpenCL and hard- ference between simple linear verti- ware. However, the algorithm characteristic cal interpolation (left) and the new most visible to the end user is obviously the edge-detecting interpolation used video produced, and much care was taken to by MEDA (right). find algorithms producing visually pleasing out- put. To avoid artifacts produced by previously The second main feature of the algorithm proposed methods, the Motion adaptive Edge is the image interpolation used after motion detecting Deinterlacing Algorithm, MEDA, was detection has been performed. Instead of us- developed. Quality evaluations show that the ing a simple type of interpolation, which of-

2 ten produce artifacts on diagonal edges, MEDA uses a sophisticated new edge-detecting algo- rithm which maintains hard diagonal edges (il- lustrated in figure 4). Additionally, this im- proved algorithm is not only applied in the tra- ditional 2D-image sense, which is essentially a horizontal-vertical plane, but also a horizontal- temporal plane, producing improved video de- tail for objects in motion.

Figure 5 – The hardware implemen- tation of the filter being prototyped on a development platform with an FPGA. The prototype is capable of real time deinterlacing of VGA con- tent at 60 fields per second.

To show the feasibility of the MEDA algo- rithm in mobile devices, two versions of the filter were successfully implemented. One ver- sion was a pure hardware implementation (pic- tured in figure 5), which for example could be included in the System on Chip (SoC) solutions which power modern mobile phones, smart TVs and media players. The second implementation was in OpenCL, which could be run on modern low-power GPUs. Both implementations can be used to process 1080i60 video, which is the toughest realistic use-case for a deinterlacing filter.

3