US 20090067508A1 (19) United States (12) Patent Application Publication (10) Pub. No.: US 2009/0067508 A1 Wals (43) Pub. Date: Mar. 12, 2009

(54) SYSTEMAND METHOD FOR BLOCK-BASED Related U.S. Application Data PER-PXEL CORRECTION FOR FILMI-BASED SOURCES (60) Provisional application No. 60/971,662, filed on Sep. 12, 2007. (75) Inventor: Edrichters als Publication Classification (51) Int. Cl. Correspondence Address: H04N II/02 (2006.01) LAW OFFICE OF OUANES. KOBAYASH P.O. Box 4160 (52) U.S. Cl...... 375/240.24; 375/E07.076 Leesburg, VA 20177 (US) (57) ABSTRACT (73) Assignee: Broadcom Corporation, Irvine, A system and method for block-based per-pixel correction for CA (US) film-based sources. The appearance of mixed film/video can be improved through an adaptive selection of normal deinter (21) Appl. No.: 12/105,664 laced video relative to inverse telecine video. This adaptive selection process is based on pixel difference measures of (22)22) Filed: Apr.pr. 18,18, 2008 sub-blocks within defined blocks of ppixels.

SOURCE FILM FRAMES Frame 1 Frame 2 Fram Frame 4 Frame 5 Frame 6

INTERLACED 3:2 VIDEO (BE) 9.(BE) (BEXNCBE)( \(BE).

FIELD PHASE

DEINTERLACED FRAMES USING REVERSE 3:2

SOURCE OF f BWD WD AWG B WD B FWD AWG B) FWD BD MISSING FIELD Patent Application Publication Mar. 12, 2009 Sheet 1 of 4 US 2009/0067508 A1

I'61)

CIE?OV/THE_LNIEC] €)NISTSEIN\/H-] Z.8ESHE/\EH Patent Application Publication Mar. 12, 2009 Sheet 2 of 4 US 2009/0067508 A1

W9.

s

W9.

US 2009/0067508 A1 Mar. 12, 2009

SYSTEMAND METHOD FOR BLOCK-BASED 0011. Unfortunately, many films that have undergone tele PER-PXEL CORRECTION FOR cine have been Subsequently edited using video equipment FILM-BASED SOURCES that ignores the original film cadence. The result is “mixed content where normal video is overlaid on top of film that has undergone a telecine process (e.g., 3:2). If inverse telecine is 0001. This application claims priority to provisional appli performed on Such video, the video-mode parts appear to be cation No. 60/971,662, filed Sep. 12, 2007, which is incorpo “weaved as well, which is visually unappealing. rated by reference herein, in its entirety, for all purposes. 0012. One solution to this problem is to turn off telecine for certain portions of the screen and use a normal video BACKGROUND mode deinterlacer for those cases. What is needed therefore is an efficient mechanism for detecting the regions that contain 0002 1. Field of the Invention Video-mode content and appropriately Switching to video 0003. The present invention relates generally to television mode for those regions. signal decoding and, more particularly, to a system and method for block-based per-pixel correction for film-based SUMMARY SOUCS. 0004 2. Introduction 0013. A system and/or method for block-based per-pixel 0005 Modern high-resolution and high-definition televi correction for film-based sources, Substantially as shown in sions require a method for displaying lower-resolution field and/or described in connection with at least one of the figures, based content (e.g., SDTV. 1080i) in a high quality way. Some as set forth more completely in the claims. types of field-based content can be created through the con version of a progressive source (such as film) using a process BRIEF DESCRIPTION OF THE DRAWINGS commonly referred to as telecine. 0014. In order to describe the manner in which the above 0006 To illustrate the telecine process, consider for recited and other advantages and features of the invention can example the process of converting 24 frame/s film to 59.94 Hz be obtained, a more particular description of the invention NTSC video. NTSC is made up of fields, which are made up briefly described above will be rendered by reference to spe of every other horizontal scanline. One field contains all the cific embodiments thereof which are illustrated in the odd numbered scanlines, while the next field contains all the appended drawings. Understanding that these drawings even numbered Scanlines. A frame is then the combination of depict only typical embodiments of the invention and are not two consecutive fields viewed at the same time. This process therefore to be considered limiting of its scope, the invention is called interlacing. will be described and explained with additional specificity 0007. The conversion of 24 frame/s file to 29.97 frame/s and detail through the use of the accompanying drawings in NTSC interlaced video is known as 3:2 pulldown. In this which: conversion, there are approximately four frames of film for 0015 FIG. 1 illustrates an example of a telecine process. every five frames of NTSC video. These four frames are 0016 FIG. 2 illustrates an example of sub-blocks within a “stretched into five by exploiting the interlaced nature of block of pixels. NTSC video. To illustrate this relation, consider the 3:2 pull 0017 FIG. 3 illustrates an embodiment of a processing down of FIG. 1. block in a motion-adaptive deinterlacer (MAD) that weaves 0008. As illustrated, forevery NTSC frame, there are actu fields together. ally two fields, one for the odd-numbered lines of the image, 0018 FIG. 4 illustrates a flowchart of a process of the designated as the top field (TF), and one for the even-num present invention. bered lines, designated as the bottom field (BF). There are, therefore, ten NTSC fields for every four film frames. To DETAILED DESCRIPTION accommodate this “stretching the telecine process alter nately places the first film frame across two fields, the second 0019 Various embodiments of the invention are discussed frame across three fields, the third frame across two fields, in detail below. While specific implementations are dis and the fourth frame across three fields. The cycle repeats cussed, it should be understood that this is done for illustra itself completely after four film frames have been converted. tion purposes only. A person skilled in the relevant art will As illustrated, the TF of the second film frame and the BF of recognize that other components and configurations may be the fourth film frame are duplicate copies. used without parting from the spirit and scope of the inven 0009. The particular pulldown method that is applied is tion. based on the type of film and the type of display format (e.g., 0020. When normal video is overlaid on top of film that NTSC, PAL, etc.). As such, various other pulldown methods has undergone a telecine process (e.g., 3:2), objectionable Such as 2:2, 5:5, 6:4, 8:7, etc. can be used for a given appli artifacts can occur when an inverse telecine process is per cation formed. For example, titles (or tickers) that move or fade may 0010. To display field-based content that is generated by a be laid over film material. When this mixed content is passed particular telecine process on a modern television display, it is to an intelligent MPEG encoder, the encoder can sometimes therefore necessary to recover the original progressive source pickup the strong 3:2 cadence of the underlying film material, content Such as film. This source can be reconstructed from resulting in a poor display of the moving text. the field-based content using an inverse telecine process. In 0021. In accordance with the present invention, an adap this inverse telecine process, it is critical that the correct tive block-based per-pixel mechanism is provided that can cadence (e.g., 2:2, 3:2, 5:5, 6:4, 8:7, etc.) of the field-based turn off the inverse telecine process for certain portions of the content be identified so that the correct fields can be weaved screen and use a normal video-mode deinterlacer instead. In together to produce the original progressive source. this manner, the normal video-mode content would not US 2009/0067508 A1 Mar. 12, 2009 undergo the inverse telecine process, which can produce 0032. The sbdiff1 for each of the vertically-oriented sub objectionable display of the video-mode content. blocks and the sbdiff2 values for each of the horizontally 0022. One of the difficulties of per-pixel correction is the oriented sub-blocks are compared to a threshold to determine determination of which regions contain video-mode content. if per-pixel correction should be enabled for that particular In the present invention, it is recognized that the differences block. If any of the sbdiff1 or sbdiff2 values exceed the between the repeat fields can be used to determine the areas threshold, then a flag rf motion(bx,by) is set for that block on the screen that are suspected to contain video-mode con indicating the presence of significant repeat field motion. tent. 0033. As would be appreciated, the specific number of 0023. In the example of a 3:2 cadence, such as that illus vertically-oriented and horizontally-oriented sub-blocks cho trated in FIG. 1, the repeat field occurs when the same field is Sen can be implementation dependent. Also, the same number utilized twice. This happens during the '3' part of the 3:2 of sub-blocks need not be used. If that is the case, different cadence. In the example of FIG. 1, the repeat fields occur in thresholds may be used in the analysis vertical and horizontal the top fields (TFs) of Frame 2 and the bottom fields (BFs) of analysis. Frame 4. 0034. In one embodiment, a histogram of values of sbdiff1 0024. In the present invention, the detection of those andsbdiff2 are created. For example, a histogram for sbdiff1 regions that are suspected to contain video mode content is can be created that has 64bins, each with different thresholds. enabled by dividing the source field into blocks of pixels. In The rf motion(bX, by) could then be set based upon the one embodiment, the source field is divided into 16x8 blocks. largest inhabited bin, or some linear combination of the bin Each block is further subdivided into four 8x4 sub-blocks and values. Likewise, the histogram could also be used for four 16x2 sub-blocks. FIG. 2 illustrates the vertically-ori sbdiff2. ented and horizontally-oriented sub-blocks within 16x8 0035 FIG. 3 illustrates an embodiment of a processing blocks. The advantage of using horizontal and Vertical rect block in a motion-adaptive deinterlacer (MAD) that weaves angular Sub-blocks is to provide sensitivity to video moving fields together. As each output frame is being generated by the in both the horizontal and vertical directions. The vertical MAD, this processing block is generally designed to Switch rectangle is wider because it provides better immunity to between using either the normal deinterlaced video or inverse comb filter artifacts. telecine video based on the flags generated during detection. 0025. For each of these sub-blocks a repeat field difference In areas where a flag indicates that video-mode content has (RFD) function can be defined. In one embodiment, the RFD been detected, the normal deinterlaced video is selected by function is the signed luma difference between the two repeat selector 310. Otherwise, the inverse telecine video is used for fields. In an alternative embodiment, the RFD function can be that block of pixels. based on luma and chroma components. 0036. The control of the switching by selector 310 is based 0026. In a luma-only implementation, the function RFD on the input received from blend control module 320. In (x,y) can be defined as the signed luma difference between the general, blend control module 320 can be designed to forward two repeat fields, or Y1(x,y)-Y2(x, y), where Y1 and Y2 are an indication of repeat field motion that is determined using the luma values for the source frame. The signed RFD(x,y) is functions such assbdifflandsbdif2. In one embodiment, the then Summed over each Sub-block, taking the absolute value “Per pixel Unexpected Motion value is set to be a maximum of the result. If the upper left corner of each 16x8 block is (bx, value when a 16x8 block is flagged as containing significant by) then the following difference function for the sub-block repeat field motion, and is set to a minimum value when the can be defined as: 16x8 block is not flagged. 0037. In one embodiment, a set of flags is stored for each of the possible phases for a particular telecine cadence. In the example of a 3:2 pulldown illustrated in FIG. 1, five phases Sbdiff1(sby, sby) = are present. A set of flags can therefore be stored for each of the five possible 3:2 pulldown phases. In this embodiment, since the flags are only updated on the repeat field, the effect where (sbX, sby) corresponds to the X and y coordinates of of the flags is the same for five output frames (provided the each sub-block, or more formally: MAD stays locked to the same phase in the 3:2 pattern). The flags are then reset and regenerated every time a repeat field is 0027 sbx=bx, bx +4, bx +8, bx +12 encountered in order to track moving video. If the MAD (0028 sby-by otherwise detects a lock loss, the flags can be reset. 0029. As noted, each 16x8 block is also divided into four 0038. In one embodiment, one or more previous sets of 16x2 blocks. Again, the signed RFD(x, y) is Summed over flags are stored for each phase. In this embodiment, the cur each sub-block, taking the absolute value of the result. More rent flag and the previous sets of flags are OR'd together for a formally, block to determine whether or not to use the normal deinter laced video. In this manner, the identification of significant repeat field motion in a block would impact other blocks that Sbdiff2(sby, sby) = are temporally proximate to that block. It should be noted that if a scene change is detected, the one or more previous sets of flags can be deemed to be invalid and therefore cleared. 0039. In another embodiment, the identification of signifi where (sbX, shy) corresponds to the X and y coordinates of cant repeat field motion in a block would impact other blocks each sub-block, or more formally: that are spatially proximate to that block. Here, the identifi 0030 sbx=bx cation of significant repeat field motion in a block can lead to 0031 sby-by, by--2, by-4, by +6 the use of normal deinterlaced video for one or more adjacent US 2009/0067508 A1 Mar. 12, 2009

blocks. For example, all blocks diagonally, horizontally, and/ the invention is capable of other embodiments and of being or vertically adjacent to a flagged block could be corrected as practiced and carried out in various ways that would be appar well. In this manner, the likelihood that normal video is dis ent to one of ordinary skill in the art after reading the dis played using an inverse telecine process is reduced. closed invention, therefore the above description should not 0040. In general, the impact of a flagged block on other be considered to be exclusive of these other embodiments. proximate blocks serves to create an expanded more uniform Also, it is to be understood that the phraseology and termi correction Zone. Also, as the correction Zone is enlarged/ nology employed herein are for the purposes of description reduced based on the detection/non-detection of significant and should not be regarded as limiting. repeat field differences, the process of the present invention is adaptive to the per-pixel detection analysis. What is claimed is: 0041. To further illustrate the features of the present inven 1. A method for detecting a presence of normal interlaced tion, reference is now made to the flowchart of FIG. 4. As Video, comprising: illustrated, the process begins at step 402 where a pixel block identifying a cadence of video that has undergone a tele is selected for processing. In one embodiment, the screen is cine process; divided into a plurality of pixel blocks (e.g., 16x8 pixel selecting a block of pixels in a pair of repeat fields in said block). Here, it should be noted that the plurality of pixel identified cadence; blocks can be distinct or overlapping. In general, using over calculating a difference measure for each of a plurality of lapped blocks provides another approach for consistency dur vertically-oriented sub-blocks in said block of pixels; ing detection. calculating a difference measure for each of a plurality of 0042. For the selected block, a blend control signal is horizontally-oriented sub-blocks in said block of pixels; generated at step 404. In one embodiment, this blend control and signal is based on whether or not the pixel block has been signaling a presence of normal interlaced video if any of flagged as containing significant repeat field motion. In one said calculated difference measures for said vertically embodiment, the significant repeat field motion is identified oriented sub-blocks or horizontally-oriented sub-blocks based on a difference measure analysis for a plurality of exceeds a threshold value. sub-blocks within the pixel block. As would be appreciated, 2. The method of claim 1, wherein said identified cadence the number and size of the sub-blocks would be implemen is a 3:2 cadence. tation dependent. In one embodiment, vertical and horizontal 3. The method of claim 1, wherein said calculating com rectangular sub-blocks are used to provide sensitivity to video prises calculating a difference measure using luma values. moving in both the horizontal and vertical directions. 4. The method of claim 1, wherein said calculating com 0043. In another embodiment, the blend control signal can prises calculating a difference measure using luma and also be based on the existence of other flagged blocks that are chroma values. proximate to the selected block. In various embodiments, 5. The method of claim 1, wherein said block of pixels is a these other flagged blocks can be temporally or spatially 16x8 block that can be divided in to four equal sub-blocks. proximate to the selected block. An advantage of generating a 6. A system for displaying a block of normal interlaced blend control signal based on the existence of other flagged Video, comprising: blocks that are proximate to the selected block is the creation a blend control module that calculates, for a block of pixels of larger more uniform correction Zones. in a pair of fields of video, a difference measure for each 0044. After the blend control signal is generated at step of a plurality of vertically-oriented sub-blocks in said 404, the video output is selected at step 406 based on the blend block of pixels and each of a plurality of horizontally control signal. Here, the blend control signal can be used by a oriented sub-blocks in said block of pixels, said calcu selector module in determining whether normal deinterlaced lation generating an output signal that has a first value if video or inverse telecine video should be selected for output. any of said difference measures exceeds a threshold and If unexpected motion is signaled by the blend control signal that has a second value if none of said difference mea for the selected block, then the normal deinterlaced video Sures exceed said threshold; and output is selected. Otherwise, the inverse telecine video out a selector module that receives said output signal from said put is selected. blend control module, said selector module being opera 0045. As has been described, the block-based per-pixel tive to select a normal deinterlaced video output for said correction process reduces undesirable artifacts because it block of pixels if said output signal has said first value differentiates the video areas of the screen from film areas of and to select an inverse telecine video output for said the screen. In this adaptive approach, only those blocks that block of pixels if said output signal has said second have the potential of including Such undesirable artifacts are value. corrected through the selection of normal deinterlaced video. 7. The system of claim 6, wherein said inverse telecine 0046. As would be appreciated, the principles of the video is a reverse 3:2 weave. present invention can be employed for doing per-pixel cor 8. The system of claim 6, wherein said block of pixels has rection on other types of cadences that are found in test a size that can be divided into a first set of equal vertically material (such as 3:2:3:2:2, 5:5, 6:4, 8:7, etc.). The present oriented Sub-blocks and a second set of equal horizontally invention can also be useful for motion judder cancellation, oriented sub-blocks. which also needs to detect cadences and reliably detect 9. The system of claim 8, wherein said first set and second whether a particular part of the screen is film or video. set are the same size. 0047. These and other aspects of the present invention will 10. The system of claim 6, wherein said block of pixels is become apparent to those skilled in the art by a review of the a 16x8 block that can be divided in to four equal sub-blocks. preceding detailed description. Although a number of salient 11. A method for displaying normal interlaced video along features of the present invention have been described above, with inverse telecine video, comprising: US 2009/0067508 A1 Mar. 12, 2009

determining a difference measure of a first block of pixels 15. The method of claim 11, wherein said second block of in a pair of repeat fields in a telecine cadence; pixels is a plurality of frames into the future. if said difference measure does not indicate unexpected 16. The method of claim 11, wherein said difference mea motion, then selecting an inverse telecine video output Sure is based on pixel calculations in a plurality of vertically oriented and horizontally-oriented sub-blocks of said first for said first block of pixels; and block of pixels. if said difference measure does indicate unexpected 17. The method of claim 11, wherein said block of pixels is motion, then selecting a normal deinterlaced video out a 16x8 block that can be divided in to four equal sub-blocks. put for said first block of pixels and a second block of 18. The method of claim 11, wherein said difference mea pixels that is proximate to said block of pixels. Sure uses luma values. 12. The method of claim 11, wherein said second block of 19. The method of claim 11, wherein said difference mea pixels is spatially adjacent to said first block of pixels. Sure uses luma and chroma values. 13. The method of claim 11, wherein said second block of 20. The method of claim 11, wherein said inverse telecine pixels is temporally adjacent to said first block of pixels. video is a reverse 3:2 weave. 14. The method of claim 13, wherein said second block of pixels is in the previous frame or next frame. c c c c c