A system approach to high-quality picture-rate conversion Chris Bartels Abstract — Various techniques have been implemented to improve the motion portrayal of flat-panel Claus Nico Cordes displays, of which the widespread introduction of motion-compensated picture-rate conversion sys- Bram Riemens tems is an essential part. However, a careful design of such systems is critical as they have the potential Gerard de Haan to introduce perceptually annoying artifacts. In this paper, an overview of the required “ingredients” of high-quality motion-compensated picture-rate conversion is presented, as implemented in state-of- the-art systems-on-chip. The visual impact of individual components on picture quality is illustrated. Keywords — Motion blur, motion judder, motion compensation, picture-rate conversion, frame-rate conversion. DOI # 10.1889/JSID18.11.922 1 Introduction In Sec. 2, we show the components of a high-quality PRC system. In Secs. 3–7, these will be illustrated individu- Motion portrayal is an important aspect of picture quality. ally by showing their visual impact from an end-user per- Motion blur affects the “dynamic” resolution of a display spective. We draw our conclusions in Sec. 8. and in this respect, flat-panel-display technologies such as LCD technology have long remained inferior to CRT dis- plays.1 The two characteristics that caused this were (1) low- 2 Components of MC-PRC response times and (2) the picture sample-and-hold effect. MC-PRC is based on the estimation of the displacement path Over time, the slow response has been addressed by or “motion trajectory” of all pixels between consecutive panel makers through material innovations, while the remain- frames, i.e., (dense) motion estimation. Early MC-PRC ing effect has been further reduced by display processing algorithms relied on relatively simple interpolation technolo- such as overdrive.2 gies to profit from the ME. However, since the introduction of Thenegativeconsequencesofthepicturesample-ReprintMC-PRC in CRT-TV systems,5 display technology has and-hold have been targeted by increasing the display pic- improved considerably, modern displays exhibit high spatial ture rate from 50/60 Hz to 100/120 Hz, and even to 200/240 resolutions, high contrast ratios, and large screen sizes. As a Hz. However, since video content is mostly captured at rela- consequence, the quality requirements for MC-PRC have tively low picture rates, such as 24 Hz for film or 50/60 Hz increased over time, as the visibility of even small processing for video cameras, high-quality picture-rate conversion artifacts on these improved displays is likely. (PRC) is now essential to profit from fast-responding This has led to more sophisticated designs of the MC- fromPRC the system (illustrated in Fig. 1), where we now broadly panels. recognize the following components: robust film mode detec- This poses a significant challenge, as the eye-tracking tion, true motion estimation, robust interpolation with occlu- of the human viewer requires PRC algorithm designers to sion detection, and global fallback.a apply motion estimation (ME) and motion compensation 3,4 In the following sections, the components are described (MC) techniques. Simpler PRC methods using repetition one by one. or linear interpolation of images cause motion blur and/or judder which render the advances inJournal display technology use- less. Motion-compensated PRC (MC-PRC) can enable per- fect motion portrayal, but, to this day, there is no design without occasional failure. Limiting the occurrence fre- quency of failures and avoiding perceptually annoying arti- facts is challenging; particularly now, recent high-quality FIGURE 1 — Processing “chain” of a picture-rate-conversion system. displays make even minor artifactsof visible. the SID In this paper, we shall discuss the key requirements aIn this paper we consider progressively scanned video. A practical sys- and lessons learned in the design of state-of-the-art MC- tem also includes de-interlacing, which we consider outside the scope PRC systems, such as those discussed in Ref. 6. of this contribution. Received 04/12/10; accepted 07/13/10. C. Bartels is with Eindhoven University of Technology, Department of Electrical Engineering, Den Dolech 2, 5600MB Eindhoven, The Netherlands; e-mail: [email protected]. C. N. Cordes and B. Riemens are with NPX Semiconductors Research, Eindhoven, The Netherlands. G. de Haan is with Philips Research Laboratories, Eindhoven, The Netherlands. © Copyright 2010 Society for Information Display 1071-0922/10/1811-0922$1.00. 922 Journal of the SID 18/11, 2010 3 Robust film mode detection fields, comparison between two consecutive input fields is required. The main challenges with image-comparison- Most display devices and broadcasters process standardized based methods we recognize as: 50/60-Hz video signals. Content with a different picture Motion: When there is a large amount of motion in rate, e.g., 24/25-Hz film, is typically converted before trans- the scene, differences in consecutive fields are apparent, mission by means of picture repetition.b As a result, video and reliable film mode detection is easily achieved. This is signals entering the TV may originate from a source with a not the case for scenes with “little” motion, i.e., only a small different picture rate than that of the transmitted signal. part of the image is moving, or “small” motion, i.e., the The task of the film mode detection is to identify the origi- velocity of the motion is very low. For the former, a field nal picture rate and remove repeated video frames. In the difference can only be detected based on a small fraction of past, this was particularly relevant for film content, which the available pixels in the image. For the latter, image dif- was converted from 25 to 50 Hz using 2:2 pull-down or from ferences are small even in the presence of motion. Note that 24 to 60 Hz using 3:2 pull-down. Nowadays, due to a wider both cases occur very often in almost any content and failure variety of input formats, more of these pull-down patterns to detect the correct film mode leads to annoying artifacts, can be observed in 50/60-Hz video signals, such as: even with limited motion. This results in challenging require- ᭹ 2:3:3:2 or 2:2:2:4 for 24–60 Hz (e.g., DV-Cam) ments on the sensitivity of the field-difference detector. ᭹ 2:2:3:2:3 for 25–60 Hz Noise: Analog noise results in differences between ᭹ 4:4 for 15–60 Hz (e.g., YouTube or animation) fields that originate from the same film image but without ᭹ 2:2:2:2:2:2:2:2:2:2:2:3 for 24–50 Hz (“TelecineB”) any motion involved. Signals that originate from PAL or Obviously, the film mode detection must also identify NTSC coding may suffer from a color subcarrier residual input signals in which no repetition pattern occur, typical for signal. news-reports or sports-events broadcasts. Digital “noise” is caused by artifacts in the transmis- The film mode detection has to correctly identify sion chain due to, e.g., MPEG-2 or H.264 coding. This may these patterns in order to enable MC-PRC for such content. include quantization errors, ringing, blockiness, and mos- It identifies which input fields originate from a single film quito noise. Note that video encoders often use different image and determines the time instances of the interpolated quality levels depending on the picture coding type in a output images; e.g., for 2:2 pull-downReprint with 25-Hz film con- GOP (group of pictures); e.g., a relatively strong quantiza- tent and a 100-Hz output, time instances of 1/4, 1/2, and 3/4 tion is usually applied to bidirectional predictive coded (B) of the time difference between successive input pictures are pictures compared to intra (I) or predictive (P) coded pic- used. In case of 3:2 pull-down with 24-Hz film content for tures. When a difference measure is taken between a B and 120-Hz output, time instances of 1/5, 2/5, 3/5, and 4/5 are a P field, these quantization differences appear as image used. The film mode detection controls which images are differences. This obscures the image differences of the sent to the rest of the MC-PRC system: an incorrect detec- original input signal. tion typically results in (remaining)from motion judder. theInterlacing “noise” asaresultofthevideolinesinone Despite the large influence on the conversion result, of the fields being vertically offset by one frame line com- there are few scientific publications on the topic of film pared to the other field. They contain different image con- mode detection. In publications on PRC, the film mode detec- tent, even though they belong to the same original film tion is sometimes described as part of the system, but rarely frame. Additionally, vertical alias in the input fields hampers elaborated upon, e.g., see Refs. 7 and 8. In most cases the the distinction between actual motion and still (i.e., non- detection is limited to 3:2 pull-down.9 Industrial relevance moving) areas with a large amount of vertical detail. There- of the topic can be recognized fromJournal patent literature, e.g., fore, such still areas are easily detected as areas with motion. see Refs. 10–14. Hybrid content: Two or more different sources of The most common strategy for film mode detection is video content may have been mixed before the signal is based on an image difference measure between successive received by the TV set. An example is given in Fig. 2 where input fields.9–14 Alternatively, a motion measure between a 60-Hz originating ticker tape runs on top of a 24-Hz origi- successive input fields can be used.8,15 Once the measure is determined, a sequence of such values is analyzed and checked against expectedof patterns according the to a set of SID supported film modes.