Noname manuscript No. (will be inserted by the editor)

Color timing echo artifact: detection and restoration Do you have a subtitle? If so, write it here

First Author · Second Author

Received: date / Accepted: date

Abstract Insert your abstract here. Include keywords, PACS and mathemat- ical subject classification numbers as needed. Keywords First keyword · Second keyword · More

1 Introduction

In the film industry, release print (RP) is one of the final products to be deliv- ered to cinemas for projection. The post-production phase of creating release print includes creation of intermediate elements, i.e. interpositive (IP) and in- ternegative (IN) (see Fig.1)[20]. These intermediate elements allow preserving the original negative (ON) from being reused and avoid it from the risk of damage. The principle of optical printing is to form an image onto a virgin film (called raw stock) from an original one by exposing it to a light source (see Fig.2). Note that the original and the raw stock move both at the same speed in front of the exposed light. The exposed film which now contains latent images will be passed through a chemical process to form visible images. If the original film is negative, the developed one will be positive, if the original film is positive, the developed one will be negative. Most printers have an additive light head, in which a system of dichroics separates the white light into red, green and blue components, each of which is passed through an adjustable light valve to control the intensity before being recombined at the printing gate.

F. Author first address Tel.: +123-45-678910 Fax: +123-45-678910 E-mail: [email protected] S. Author second address 2 First Author, Second Author

Original Negative Interpositive Internegative Release Print (ON) (IP) (IN) (RP) Original Negative Interpositive Internegative Release Print (ON) (IP) (IN) (RP) Fig. 1 Original negative passes through several intermediate elements (interpositive, in- ternegative) to obtain release print

Exposed film

Optical printer (R,G,B)

Raw stock Original film (virgin film)

Fig. 2 Optical printing

Note that while printing step fromFigure IP 1: Negative/Inter-Positive to IN and from impression IN to RP can be done with only one-light (process using only a single color setting for all shots), the printing from ON to IP is carried out with color-timing light (each shot is projected by a different color setting). The use of color-timing light has

several goals[7,21]: (i) variations in exposure and lighting between different shots must be evened out, to provide a continuity of colour throughout a scene, (ii) the colour of some objects may need to be reproduced exactly, (iii) the appropriate mood must be created, for example, arosy or warm balance for romantic scenes, darker for stormy effects and so on. That is why this IP is also

called master positive in the sense that it is ready to shown in theaters. The RP is just duplicate of this IP. In IP-IN and IN-RP steps, since the exposed light is unchanged for all shots then printing speed is very high between 500 to 2000 feet/minute[7]. On the other hand, in ON-IP step, this speed is much

slower (no more than 180 feet/minute[21]) due to adaptation of color-timing light. Although the printing speed is slow in ON-IP step, the IPs may suffer from

a particularly undesirable artifact producing objectionable effect, mostly visi- ble at shot boundary due to the mismatching of color-timing light. In fact, the light vanes change automatically for each shot as the negative passes through the printer, triggered by a list of Frame Count Cues (FCCs) marking each shot change. Mechanical light vanes take a few milliseconds to move form one setting to the next. A big light change may cause a coloured flash on the first frame of the next shot. Look at Fig.3(a) where a correct ON-IP printing is shown. Suppose that the roll direction is bottom-up then shot A is followed by shot B. Each shot has its own color timing light configuration: R1G1B1 for shot A and R2G2B2 for shot B. In the case wrong printing (Fig.3(b)), the light change and the shot change are mismatching. Thus the top of the first frame b1 of shot B and/or the bottom of the last frame an of shot A are exposed by Color timing echo artifact: detection and restoration 3

a wrong light RtGtBt which is a transition from R1G1B1 to R2G2B2. Conse- quently, there is a straight band with color and contrast varied on the top the

first frame of shot B and/or on the bottom of the last frame of shot A in the

IP. Indeed, this default can effect only the bottom of frame an or the top of the frame b or both of them. 1

roll direction roll direction frame an-1 frame a n-1 R1G1B1 Shot A R1G1B1 Shot A

frame an frame a n

RtGtBt frame b1 frame b 1

R2G2B2 Shot B Shot B R2G2B2 frame b2 frame b 2

(a) A correct printing example (b) A mismatching printing example Figure xxx : A correct master-IP print when each shot in the master is exposed by it Figure xxx : A correct master-IP print when each shot in the master is exposed by it own color timing light own color timing light

Fig. 3 Correct and wrong printing

Nowadays, since digital techniques are highly developed, they are also ex- ploited in the film industry. A new schema of post-production is shown in

Fig.4 in which the IP is scanned to form a digital film. From this digital film, filmmakers can explore many digital softwares to create many special effects or restore defaults such as color-timing echo. In the next step, the restored film is shot back to film pellicle to form the IN. Our work is in the digital film restoration step.

Analog elements

Original Negative Interpositive Internegative Release Print (ON) (IP) (IN) (RP)

scan shot

Digital Film Restored Film

Digital elements

Fig. 4 Film post-production with digital intermediate elements

Till now, the detection and the correction of the echo artifact have been handled manually with some existing restoration software but at the expense oftime and lack of stability and flexibility. The development of a method which allows restoring automatically the artifact is thus a challenging task.

In this paper, a new method is proposed to automatically detect and restore this default. The paper is organized as follows: analysis and model of default is presented in section??, the detection and restoration is shown in section3, the experimental results are reported in section4, the conclusions are given in section5

4 First Author, Second Author Analog elements

Original Negative Interpositive Internegative Release Print

2 Image(ON) Acquisition Model(IP) (IN) (RP)

scan shot In this section, we will model all steps to produce the digital film from the ON. Indeed, there areDigital three Film phases (see Fig.5:Restored Film (i) from ON to exposed film

which contains latent images, (ii) chemical development process to make im- Digital elements ages visible in IP, (iii) scan the IP to form digital film.

Analog elements

Original Negative InterpositiveOptical InternegativeChemical Release Print printer Development Scan (ON) Original Negative(IP) Impressed Film (IN) Interpositive (RP) Digital Film (ON) (latent images) (IP)

scan shot Fig. 5 From the original negative to the digital film

Digital Film Restored Film

To model these steps, it is important to know the film structure. For the Digital elements research purpose, we consider in Fig.6 simple model of tripacks film [22,20,17]. There are three layers which are sensitive to blue, green and red color respec-

tively. Unlike additive color system which is widely used in projectors, display materials, Optical a subtractiveChemical color system is usually used in the film photography. printer Development Scan Original Negative To this end,Impressed if the Film blue sensible layerInterpositive is exposed, after chemicalDigital Film development (ON) (latent images) (IP) this layer will form a yellow dye. For the green and red sensitive layer, magenta and cyan dye will be formed respectively. That is why these layers are called also yellow forming, magenta forming and cyan forming layer, respectively.

Blue sensitive layer Green sensitive layer

Red sensitive layer

Fig. 6 Film structure

z

αrEr(λ)

αgEg(λ) X X E( λ) cn mn yn ET( λ) Xr g b αbEb(λ)

Original Negative Exposed film

Fig. 7 Original Negative - Exposed Film Model

In the first step (see Fig.7), from ON to exposed film, the spectral energy

of the grading light E(λ) is composed of three beams: red Er(λ), green Eg(λ), blue Eb(λ) and each one is controlled by a valve which adjusts the force αr, αg, αb. Therefore, we have:

E(λ) = αrEr(λ) + αgEg(λ) + αbEb(λ) (1)

Color timing echo artifact: detection and restoration 5

The transmitted light ET (λ) is estimated by using Beer-Lambert law [28,30, 10]: ET (λ) = E(λ)10−(cnDc(λ)+mnDm(λ)+ynDy (λ))z (2)

where cn,mn and yn are dye density of cyan, magenta and yellow layer respec- tively (the subscript ’n’ stands for negative), Dc(λ), Dm(λ) andz Dy(λ) are spectral density of cyan, magenta and yellow layer respectively, z is the thick- αrEr(λ) ness of an emulsion. The raw stock (virgin film) is exposed by this transmitted αgEg(λ) light and makes latent images. The exposure level X , X , X of each layer is X X b E(g λ) r cn mn yn ET( λ) Xr g b αbEb(λ) estimated as follows: Z Original Negative Exposed film Xr = ∆t ET (λ)Lr(λ)dλ , (3a) ω Z Xg = ∆t ET (λ)Lg(λ)dλ , (3b) ω z Z αrEr(λ) Xb = ∆t ET (λ) Lb(λ)dλ , (3c) αgEg(λ) ω X X X E( λ) c n m n y n ET( λ) r g b αbEb(λ) H-D curve where ∆t is exposure time, ω is wavelengthXr Xg Xb of visible light,cp Lmrp( λy),p Lg(λ) and Chemical Lb(λ) are spectral sensitivity (film speed) of blue,development green, red sensitive layer respectively. Original Negative Exposed film Exposed film Interpositive

H-D curve

Xr Xg Xb cp mp yp Density Chemical development

Exposed film Interpositive Log -exposure

(a) From exposedfilm to interpositive (b) H-D curve

Fig. 8 Exposed film - Interpositive model

In the second step, the exposed film is passed through a chemical develop- ment to make images visible (see Fig.8a). The H-D curve (see Fig.8b) [5,29,

24] (first published by F.Hurter and V.C. Drifield) shows the relation between

dye density cp,mp (the subscript ’p’ stands for ’positive’) and yp of the devel- oped film (interpositive) and exposure level. Indeed, the dye density is linear function of log-exposure level [21,20,28] as follows:

cp = kc + γc log10(Xr) , (4a)

mp = km + γm log10(Xg) , (4b) yp = ky + γy log10(Xb) , (4c)

z

αrEr(λ)

αgEg(λ) X X E( λ) cn mn yn ET( λ) Xr g b αbEb(λ)

Original Negative Exposed film

H-D curve Xr Xg Xb cp mp yp Chemical development Exposed film Interpositive

Density

6 First Author, Second Author

Log -exposure where k , k , k are constants, γ , γ , γ are the slopes of the film H-D curve c m y c m y of cyan, magenta, yellow dye respectively, and c , m , y are dye density of p p p cyan, magenta, yellow of developed film respectively.

Digital Sensor Gamma Image S( λ) c m y ST( λ) p p p correction (R, G, B ) [.] γ

Interpositive

Fig. 9 Interpositive - Digital Film Model

In the last step, the digital film is obtained by scanning this IP. The model is shown in Fig.9. Similar as previous step, the transmitted light ST (λ) is obtained from Beer-Lambert law:

ST (λ) = S(λ)10−(cpDc(λ)+mpDm(λ)+ypDy (λ))z (5)

This transmitted light passes through a sensor and a gamma correction step hence form digital image:

 Z γ R = ∆t ST (λ)Qr(λ)dλ , (6a) ω  Z γ G = ∆t ST (λ)Qg(λ)dλ , (6b) ω  Z γ B = ∆t ST (λ)Qb(λ)dλ , (6c) ω

, where R G, B are digital values, Qr(λ), Qg(λ) and Qb(λ) are red, green and blue sensor sensitivity respectively, γ is gamma correction coefficient. In order to simplify this equation and remove the integral, we can suppose that the sensor is sensitive to narrow band. It means that sensor sensitivities behave similar to a Dirac delta function:

Qr(λ) = δ(λ − λr) , (7a)

Qg(λ) = δ(λ − λg) , (7b)

Qb(λ) = δ(λ − λb) . (7c)

In practice, Dirac delta function is, or can be made to be, a tolerable ap- proximation for most real sensors [33,11,13]. Henceforth, the equations 6 are Color timing echo artifact: detection and restoration 7 rewritten as follows:  Z γ γ R = ∆t ST (λ)δ(λ − λr)dλ = [ST (λr)] , (8a) ω  Z γ γ G = ∆t ST (λ)δ(λ − λg)dλ = [ST (λg)] , (8b) ω  Z γ γ B = ∆t ST (λ)δ(λ − λb)dλ = [ST (λb)] . (8c) ω

To conclude, the model to describe all process from the original negative to digital film is given in formulas (1, 2, 3, 4, 5, 8).

3 Color-timing correction

3.1 Related Works

To our best knowledge, there is no method to cope with color timing echo artifact since it is a particular issue in the film domain. However, some ex- isting algorithms which deal with flicker could be used in our case. Flicker refers to fluctuations of image brightness in archived films which has similar phenomenon as in our case. Initial efforts on deflickering reported in the litera- ture assumed that the entire degraded frame was globally affected in a similar way [REFERENCE]. Other approaches which deal with local fluctuation are also presented [9,8]. Cac method ve deflickering Liet ke.... Van de khi lam motion, tat ca deu gia thiet la linear !? Ko lam occlusion =¿ anh huong den ket qua cua histogram specification Inpainting: (coi nhu* vung do’ la black hole, no information) - copy thong tin tu frame truoc: nhung vung qua to the nay thi ko duoc - motion prediction: su dung cac frame truoc, uoc luong chuyen dong, tu do’ suy ra chuyen dong.... Tuy nhien o* day ko fai mat hoan toan thong tin, ma thong tin chi bi thay doi di ma thoi.

3.2 Solution Overview

The proposed solution has five steps (see Fig.10). In the first one, as discussed above, the color-timing echo artifact always appears at the shot change, hence the first step is shot change detection. Thus a motion estimation is carried out to align degraded frame and reference one. This step has threefold: (i) this improves the performance of degraded region detection (the third step), (ii) this serves to better detect occlusion region (the forth step), (iii) and finally, thanks to this motion flow, the unoccluded pixels in degraded frame can be restored by warped corresponding pixels from reference frame. The main contribution is in this step in which based on image acquisition model in section Degraded fil m 8 First Author, Second Author Shot change detection 2, we propose a new data term invariant to brightness-color change for optical flow framework. InDetection theDegraded third zone step, detection we focus on degraded frame and warped-

reference one to detect degraded region. Note that since there is movement Motion estimation in sequence, there are always occlusions. Obviously, the degraded pixels in these occlusion zones can not find the matching points in the reference frame. Therefore, to cope withOcclusion this issue, detection an occlusion detection step is considered. The degraded pixelsRestoration in the occlusion regions will be restored by using an inter/extrapolation functionInter/Extrapolation (the last step) which is derived from matching pairs (the degraded pixels which are not in the occlusion regions and their Restored film corresponding references).

Degraded fil m

Shot change detection

Motion estimation

Degraded zone detection

Occlusion detection

Inter/Extrapolation

Restored film

Fig. 10 Method overview

3.3 Shot Change Detection

Shot changes may occur in two ways: abrupt cuts, where a frame from one shot

is followed by a frame from a different shot, or gradual transitions such as cross-

dissolves, fade-ins, fade-outs... Here, we deal with the first category. There are many propositions [35] in the literature. Each one has a characteristic but all of them bases on same observation: there is small difference between two consecutive frames in the same shot whereas there is a lot of difference between frames from two different shots. In [35], the authors measure the visual-content discontinuity by comparing the corresponding pixels between two frames. To avoid high motion in film, global measures based on intensity histogram, color histogram are also proposed [2,32]. Other complex features such as edge or motion vectors [16] are taken into account to improve the performance. In this paper, we propose to use the Mean of Absolute Difference ((MAD)) between two consecutive frames for tracking and estimating these changes. The MAD is defined as follows:

Color timing echo artifact: detection and restoration 9

H K 1 X X MAD(i) = ( |Li+1 − Li |) (9) HK h,k h,k h=1 k=1 i where Lh,k is the luminance of frame i of size H × K at coordinate (h, k). An example of MAD profile (scaled in [0..255] interval) of 85 frames is given in Figure 11:

FigureFig. 5: Mean 11 MeanAbsolute of AbsoluteDifference Difference (MAD) Profile (MAD) of profile85 of 85 consecutive frames. There are consecutivetwo high frames picks whichin which correspond there are to2 shot two change shot changes.s.

As can be seen, there are two high peaks which correspond to shot changes in this sequence. In order to detect these shot changes, we need a threshold to locate the peaks. To this end, instead of using an empirical threshold which could make the method image dependant, we propose to use Otsu’s thresh- olding method [27] which is based on discriminating analysis. The goal of this algorithm is to threshold a gray image to obtain a binary one. It assumes that the image to be thresholded contains two classes of pixels (e.g. foreground and background) then calculates the optimum threshold separating those two classes so that their combined spread (Within Class Variance - WCV ) is mini- mal. It is easy to see that Otsu’s method can be used to hold our issue since in our case, there are also two classes of value: small one (difference between two consecutive frames in same shot) and high one (difference between two frames in different shots). In practice, Otsu’s algorithm exhaustively estimates WCV for 256 possible threshold values (from 0 to 255). The optimum threshold is the one which gives the smallest WCV value. More detail about this method can be found in [27].

3.4 Motion Estimation

The aim this step is to find corresponding reference pixels for degraded ones. There are two main branches of motion estimation, i.e. block matching and optical flow. The first approach is usually used in video compression domain. Its principle is to find the most similar block for a given one. Indeed, it does not care about the smoothness of motion flow. This issue is clear in homoge- nous regions where motion vector is very chaotic. On the contrary, optical flow yields smooth flow thanks to smoothness regularization term. In our context,

10 First Author, Second Author

(a) (b)

Fig. 12 Figure(a) Histogram 6: (a) Histogram of scaled of MAD scaled profile, MAD (b)profile, Within-Class Variance (WCV ) against threshold.(b) Within-Class Variance (WCV) against threshold.

since we need occlusion detection, method which gives smooth flow is appreci- ated. Therefore, in this paper, optical flow approach is chosen. The first total variation framework to estimate optical flow is proposed in [15] as follows:

E(u, v) = EData(u, v) + αESmooth(u, v) (10)

where

Z 2 EData(u, v) = (fiu + fjv + ft) didj, (11a) Ω Z 2 2
ESmooth(u, v) = |∇u| + |∇v| didj (11b) Ω

α is regularization parameter, subscript (i, j, t) denote partial derivatives, T T ∇u = [ui, uj] , ∇v = [vi, vj] , f is image data. It is well known that optical flow algorithm is based on two hypotheses (i) small motion and (ii) bright- ness constancy. However, these two hypotheses are usually violated. To deal with large motion, several methods based on pyramidal decomposition are in- vestigated [23,6,4]. Two input images are decomposed into several images by using pyramidal decomposition [1]. At the coarsest level, the motion becomes smaller, therefore it is firstly estimated in this level then propagated to finer one. To deal with brightness variation, several methods propose to modify the data term in equation 11. For example, instead of using quadratic difference metric, the authors in [14,26] propose to use adaptive normalized cross corre- lation measure. Others [12,25] suggest to use a transformation which results two new images whose values are free from brightness variation. The motion flow is then estimated from these new data. In this paper, based on image acquisition model for film in section 2, we derive that the derivative of the logarithm of pixel values is invariant to lighting change. Let denote R(i,j),deg is red value of pixel (i, j) in the degraded frame; R(i+δi,j+δj),ref is the red value of the corresponding pixel in the reference frame. Here the motion vector is (δi, δj). Note that due to mismatching pro- jected light, the brightness constancy assumption is violated, i.e. R(i,j),deg 6= R(i+δi,j+δj),ref . Color timing echo artifact: detection and restoration 11

Let us estimate the horizontal derivative of the logarithm of red value (logR)i: (i,j) (i+1,j) (logR)i = log(R ) − log(R ) (12) By using equations 8 and 5 this value becomes:

 (i+1,j) (i,j)  (i+1,j) (i,j)  (i+1,j) (i,j)  (logR)i = γz cp − cp Dc(λr) + mp − mp Dm(λr) + yp − yp Dy(λr) (13) (i+1,j) (i,j) Let consider now the term C = cp − cp . Based on equation 4a we have: ! X(i+1,j) C = γ log r (14) c 10 (i,j) Xr

Using definition of the exposure level Xr in equation 3 and the transmitted light ET (λ) in equation 2, we obtain:

(i+1,j) (i+1,j) (i+1,j) ! R (i+1,j) −(cn Dc(λ)+mn Dm(λ)+yn Dy (λ))z ω E (λ)10 Lr(λ)dλ C = γclog10 (i,j) (i,j) (i,j) R (i,j) −(cn Dc(λ)+mn Dm(λ)+yn Dy (λ))z ω E (λ)10 Lr(λ)dλ (15) Note that E(i+1,j)(λ) is spectral energy (see equation ??) which is a function (i+1,j) (i+1,j) (i+1,j) of the force αr , αg , αb :

(i+1,j) (i+1,j) (i+1,j) (i+1,j) E (λ) = αr Er(λ) + αg Eg(λ) + αb Eb(λ) (16) These forces are just function of line j, not column i. Therefore, we can say (i+1,j) (i,j) (i+1,j) (i,j) (i+1,j) (i,j) (i+1,j) (i,j) αr = αr , αg = αg , αb = αb and E (λ) = E (λ). Moreover, since red-sensitive layer Lr(λ) of the interpositive queasily does not absorb blue and green components in E(λ), we can approximate equation 15 as follows:

(i+1,j) (i+1,j) (i+1,j) R (i+1,j) −(c Dc(λ)+m Dm(λ)+y Dy (λ))z ! αr Er(λ)10 n n n Lr(λ)dλ C = γ log ω c 10 (i,j) (i,j) (i,j) (i,j) R −(cn Dc(λ)+mn Dm(λ)+yn Dy (λ))z ω αr Er(λ)10 Lr(λ)dλ (17) (i+1,j) (i,j) Since αr = αr , we can simplify them and obtain:

(i+1,j) (i+1,j) (i+1,j) ! R −(cn Dc(λ)+mn Dm(λ)+yn Dy (λ))z ω Er(λ)10 Lr(λ)dλ C = γclog10 (i,j) (i,j) (i,j) R −(cn Dc(λ)+mn Dm(λ)+yn Dy (λ))z ω Er(λ)10 Lr(λ)dλ (18) Note that in this equation, C does not depend on the forces αr, αb, αg. Similar (i+1,j) (i,j) (i+1,j) remark can be easily obtained for M = mp − mp and Y = yp − (i,j) yp . Therefore horizontal derivative of the logarithm of red value (logR)i is invariant to the lighting change. The same procedure can be carried for blue and green channels, hence (logG)i and (logB)i are invariant to the lighting change too. Let us consider now the vertical derivative of the logarithm (i,j) (i,j) (i,j) (logR)j,(logG)j,(logB)j. By using assumption that the force αr ,αg ,αb 12 First Author, Second Author

(i,j+1) (i,j) (i,j+1) (i,j) are similar in two consecutive lines, i.e. αr ≈ αr , αg ≈ αg , (i,j+1) (i,j) αb ≈ αb , we can derive that the vertical derivative of the logarithm (logR)j,(logG)j,(logB)j are invariant to the lighting change too. Based on these analyses, the proposed data term for optical flow framework is as follows: Z  2 EData(u, v) = Ψ (log() − log()) didj (19) Ω ======DO NOT READ THIS PART ======It depends only on the dye density of the original negative, since the spectral energy of three subbands Er(λ) are separated Let denote Rx is red value at coordinate x = (i, j) in the degraded frame. Let denote Rx+δx is . Note that, due to mismatching light, Rx+δx 6= Rx. CONG THUC FIDELITY CUA HORN LA LINEARISATION ROI, minh dung cong thuc goc thoi =¿ However, this linearisation is only valid under the assumption that the image changes linearly along the displacement, which is in general not the case, especially for large displacements. Therefore, our model will use the original, non-linearised grey value constancy assumption Since with quadratic penalisers, outliers get too much influence on the estimation, an increasing concave function (s2) is applied, leading to a robust energy [7, 16]: TAI SAO LAI DUNG sqrt o* fidelity term [16] E. Memin and P. Perez. Dense estimation and objectbased segmentation of the optical flow with robust techniques. IEEE Transactions on Image Processing, 7(5):703 719, May 1998. [7] M. J. Black and P. Anandan. The robust estimation of multiple motions: parametric and piecewise smooth flow fields. Computer Vision and Image Understanding, 63(1):75104, Jan. 1996. Bau dau citer cac pp lam invariant voi brightness change o* day !!!!!!!!! Since in the next step, occlusion detection is estimated which depend The main drawback of this approach However, motion estimation when the brightness varies from a frame to another is difficult task. In the literature, there are two ways to deal with issue. In the first one, a fidelity measure which is invariant to brightness change is proposed []. In the second one, However this measure fails when the brightness varies from a frame to another. The most challenging However, motion estimation in Once degraded region is localized, a Them hinh forward motion, them warped frame. ======Indeed, the warped frame can be used as restored version for degraded frame. However, since there is movement, there is always occlusion. For ex- ample, in Fig.17, the pixels in the left and bottom border of degraded frame do not have corresponding pixels in reference one due to global motion (cam- era movement). The pixels around the man has no longer matching pixels in the reference frame due to local motion. Therefore, to restore these occluded pixels, the warping way is not enough. To cope with this issue, it is necessary Color timing echo artifact: detection and restoration 13

isolate them and restore by other way. This work is described in section 3.6 and 3.7.

3.5 Degraded Zone Detection

Based on reliable motion flow, we can entirely restore the haft of degraded frame. However, it should be better to detect degraded region and do not alter uneffected region. As the roll speed is fixed, the affected region is usually rectangular. Since the artifact never exceeds the middle of the frame, to save the computational time, the forward motion estimation (in considering the degraded frame is anchor and the reference one is target) is carried out on only a haft of the frame. In the default is on the top, the top haft is used, otherwise, the bottom haft is concerned. To detect the extent of the degraded region, we propose to estimate the Mean Intensity of each Line (MIL) as follows: K 1 X MILc(j) = ( Ii,j) (20) K c i=1

j th where Ii,j is the intensity value of channel c (red, green or blue) at j line and ith column. Note that, due to motion, the MIL of the degraded and the refer- ence frame are usually mismatching (see Fig.20) which leads detection trigger fail. In this paper, we propose to compare the MIL of the degraded frame and warped-reference frame. Thanks to reliable motion flows, the warped-reference frame now is aligned with the degraded one. Hence, their MIL profiles are well aligned too (see Fig.21). Note that in the warped-frame, there is some red re- gions on the borders due to global motion (camera movement). This color region indicates that it does not exist corresponding pixels in the reference frame for the pixels in this region of the degraded one. Therefore, the pix- els in this red region are not taken into account to estimate the MIL. Now, to detect the extent of the degraded region, an absolute difference between MIL of these profiles for each channel (see Fig.22). If the degradation is on the top of the frame, we seek from left to right the first point for which the absolute difference is smaller than a threshold Tline. The spatial extent of the distortion is then determined from this crossover. When the distortion occurs in the bottom of frame, a similar searching procedure is done from right to left. Note that the extent of degraded region can be different from one channel to another. Note that the extent of degraded region can be different from one channel to another.

3.6 Occlusion Detection

When there is movement, there is obviously occlusion. After motion estimation step, each occluded pixel has anyway an motion vector due to to forced, but unreliable, data matching. These pixels evidently do not have corresponding 14 First Author, Second Author

pixels in the reference frame. Therefore, we can not restore these pixels by using the estimated motion flow and reference frame. To cope with this issue, we propose to distinguish these pixels from unoccluded ones and restore them by other way (see section 3.7). In the literature, there are two main approaches to deal with this task. In the first one, [34] propose to directly integrate an occlusion-penalty term in the total variation framework. This method yields a motion flow for unoccluded pixels and indicates occlusion region in which there is no motion. In the second one, the occlusion is detected based on the mismatch between forward and backward flows [18,3,19,31]. Note that these motions are normally estimated without making an explicit occlusion-penalty term as in the first approach. The main idea behind this case is that the forward motion (the degraded frame is anchor, the reference is target) of an unoccluded pixel in the degraded frame is exactly opposite to the backward motion of its correspondence in the reference frame, whereas for an occluded pixel, this property is no longer true. Therefore, by checking the consistency of forward and backward flow, we can detect occluded region. Let denote (i,j) (i,j) uf , vf are forward motion of pixel (i, j) in the degraded frame (the sub- (i,j) (i,j) script f stands for forward). Let denote ub , vb are backward motion of pixel (i, j) in the reference frame (the subscript b stands for backward). A pixel (i, j) in the degraded frame is considered unoccluded if the matching error vec- (i,j) (i,j) tor [δu , δv ] is very small. Otherwise, this pixel is in occluded region. The matching error vector is defined as follows:   (i,j) (i,j) (i,j) (i,j) δu = uf + ub x + uf , y + vf , (21a)   (i,j) (i,j) (i,j) (i,j) δv = vf + vb x + uf , y + vf (21b) The occlusion map OCM (see Fig.26) can be estimated by thresholding the (i,j) norm nδ of this error vector:

( (i,j) (i,j) 1 if nδ > Tocc OCM = (i,j) (22) 0 if nδ ≤ Tocc q (i,j) (i,j) 2 (i,j) 2 where nδ = (δu ) + (δv ) (see Fig.25). Indeed, in the literature, it does not exist an algorithm which can perfectly determinate occlusion region. Therefore, in our work, a supplemental dilation operator is used to enlarge this region (see Fig.27). Of course, some unoccluded pixels are effected by this dilation and are considered occluded ones. But it is not trouble when they can be perfectly restored by inter-extrapolation function.

3.7 Inter-Extrapolation

The aim of this step is to restore the occluded pixels (pixels under red mask in Fig.28). To this end, a fit function which is derived from unoccluded pixels in Color timing echo artifact: detection and restoration 15 the degraded frame and its correspondences in the warped-reference frame is used. Note that when the default effects differently from one line to other, this function is then estimated for each line. Moreover, as can be seen in equation ??, log(Degraded)/log() = constant, it means that the function is linear. An example of this function is given in Fig.29 where the red line is linear function which is best fitted with distribution points. The occluded pixels are then restored based on this function.

4 Experimental Results and Discussion

The proposed method is evaluated with both simulated and real images in term of objective and subjective measure. In the case of simulated image,

4.1 Simulated Images

4.2 Real Images

The proposed method is evaluated also with HD images.

4.3 Discussion

It is important to note that, the performance of the proposed method depends heavily on the quality of motion estimation step. Note that performance of this step depends heavily on the quality of mo- tion flow. If motion vectors is not well estimated, the occlusion detection is consequently broken down. It should be noted that, in the literature, even there are many evolution in optical flow algorithms, 16 First Author, Second Author

Fig. 13 (left) Degraded frame where the artifact is on the top, (right) zooms in degraded region

Fig. 14 (left) Reference frame, (right) zooms of this frame

Fig. 15 color coding of the flow: hue indicates orientation and saturation indicates magni- tude

Color timing echo artifact: detection and restoration 17

Fig. 16 Forward motion estimated from the proposed method (the degraded frame is an- chor, the reference frame is target)

Fig. 17 Warped-frame estimated from reference frame and the forward motion in Fig.16

Fig. 18 Forward motion estimated directly from the image intensity

Fig. 19 Warped-frame estimated from reference frame and the forward motion in Fig.18

18 First Author, Second Author

Fig. 20 The MIL profiles of the degraded and reference frame for the 3 channels (red, green, blue). It is easy to see that these profiles are not well aligned due to motion

(a) (b)

Fig. 21 The MIL profiles of the degraded and warped-reference frame for the 3 channels (red, green, blue). These profiles are now well aligned thanks to reliable motion flows

Fig. 22 The absolute difference between MIL of degraded and warped frames for 3 channels (red, green, blue) Color timing echo artifact: detection and restoration 19

Fig. 23 Backward motion estimated from the proposed method (the reference frame is anchor, the degraded frame is target)

Fig. 24 Backward motion estimated directly from the image intensity (the reference frame is anchor, the degraded frame is target)

Fig. 25 The norm of the matching error vector (see equation 21) calculated from forward and backward motions in Fig.16 and Fig.23

Fig. 26 Occlusion map: yellow indicates unoccluded region whereas red presents occluded one. This map is obtained by thresholding the norm of the matching error vector in Fig.25

20 First Author, Second Author

Fig. 27 Dilated occlusion map. This map is obtained by dilating the red region in the occlusion one

Fig. 28 The region under red mask is considered in occluded region and it is restored by using inter/extrapolation function (see section 3.7)

Fig. 29 XYZ A linear function (red line) fits with degraded and warped-reference pixels

Fig. 30 (left) Restored frame, (right) zooms of this frame

Color timing echo artifact: detection and restoration 21

(a) (b) (c) (d)

Fig. 31 Zooms: (from left to right) (a) the degraded frame, (b) the reference frame, (c) the warped-reference frame, (d) the restored frame

(a) (b) (c) (d)

(a) (b) (c)

(a) (b) (c)

Fig. 32 Zooms: (from left to right) (a) the degraded frame, (b) the reference frame, (c) the restored frame

5 Conclusions and Perspectives

One mishandling in the current variational model is trying to minimize the squared intensity error or data energy for every pixel regardless if the pixel is occluded or not. As a result, the warped-reference image, ..., has to perform incorrect defromation to fill the occluded area of the degraded frame even though no corresponding pixel in reference frame can match the occluded pixels of the degraded frame.

First, a color timed copy called interpositive (IP) is made from an original negative after editing and color correction have been achieved and validated. From the IP, an one-light (process using only a single color setting for all shots) copy called the internegative (IN) is then created from which several copies of final release print can be made. Although this advantage, the IPs may suffer from a particularly undesirable artifact producing objectionable effect, mostly visible at shot boundary. This artifact that we referred as color timing echo behaves as an echo and takes the form of straight band with color and contrast varied either on the bottom of the last frame of a shot or on the top of the subsequent frame of the next shot. This undesirable degradation through the IN affects hence the quality of the release prints. - noi context film anh: nega=¿interpo... - noi tai sao phai can color-timing (sua lai, create motion,..) - noi color-timing chi co tu nega=¿interpo. Toc do cham (180feet/minute) con doan sau thi one-light rat nhanh (¿2000feet/minute) 22 First Author, Second Author

- noi exposed light duoc qua dioriche, thanh 3 thanh fan red,green,blue, moi thanh fan lai co’ 1 cai valve de dieu chinh truoc khi cho hoi tu lai o dau ra. Firstly, a shot change detection is carried out followed by a degraded zone detection. Next, a motion estimation step is done to Moreover, to avoid bad interpolation in occlusion region, an occlusion detection step is also carried out. Pixels in this region is restored by using a inter-extrapolation function which is derived from Indeed, these distortions are more serious when shown on a large display in cinemas. Furthermore, before registering and validating the quality, the films are evaluated by golden eyes who are known to be very sensitive to any changes or mismatches. j th where Ih,k is the intensity value of channel c (red, green or blue) at h line th and k column. For example, in Fig.??, the two MIL profiles of the degraded frame and the reference one are computed for red channel. It is easy to see that there is an important difference on the left of the two profiles. It means that, on the top of the degraded frame, the red channel is affected by the echo artifact. It is important to note that the difference gradually reduces from left to right. It means that the default is more serious on the top border and gentle reduces inside the frame. According the above observations, some

Fig. 33 MIL profiles of the red channel of the degraded and reference frames

characteristics about the color-timing echo artifacts can be outlined as follows:

– It appears at boundary of shot changes. Precisely, it is present on the bottom of the last frame of the first shot or on the top of the first frame

of the following shot. – The default is more serious on the top border (or bottom border) and gradually reduces inside the frame Moreover, according our experiences – The echo artifact present only in the case of abrupt cuts shot change not gradual transition one. – The degraded region never exceeds the middle of the frame Color timing echo artifact: detection and restoration 23

References

1. Adelson, E., Simoncelli, E., Freeman, W.: Pyramids and multiscale representations. Proc European Conf on Visual Perception (1990) 2. Ahanger, G., Little, T.: A survey of technologies for parsing and indexing digital video. Journal of visual Communication and image representation 7, 28–43 (1996) 3. Alvarez, L., Deriche, R., Papadopoulo, T., Sanchez, J.: Symmetrical dense optical flow estimation with occlusion detection. European Conference on Computer Vision pp. 721–735 (2002) 4. Alvarez, L., Weickert, J., S´anchez, J.: Reliable Estimation of Dense Optical Flow Fields with Large Displacements. International Journal of Computer Vision 39(1), 41–56 (2000) 5. Attridge, G.: The characteristic curve (1991). The Journal of Photographic Science 6. Black, M.J., Anandan, P.: The robust estimation of multiple motions: parametric and piecewise-smooth flow fields. Comput. Vis. Image Underst. 63(1), 75–104 (1996) 7. Case, D.: Film technology in post production (2001). 2th edition, Focal Press 8. Delon, J.: Midway image equalization. Journal of Mathematical Imaging and Vision 21 (2004) 9. Delon, J., Desolneux, A.: Un problme de restauration des vieux films : la correction du flicker. MATAPLI (2009) 10. Edelman, A., Wang, F., Rao, A.: A physically based numerical color cali- bration algorithm for film (2003). Available at http://math.mit.edu/ edel- man/Edelman/publications.htm 11. Finlayson, G., Hordley, S.: Color constancy at a pixel. Journal of the Optical Society of America A: Optics, Image Science, and Vision,Volume 18, Issue 2 pp. 253–264 (2001) 12. Finlayson, G., Xu, R.: Illuminant and gamma comprehensive normalisation in log rgb space. Pattern Recogn. Lett. 24, 1679–1690 (2003) 13. Heo, Y., Lee, K., Lee, S.: Illumination and camera invariant stereo matching. in CVPR (2008) 14. Heo, Y., Lee, K., Lee, S.: Illumination and camera invariant stereo matching. Computer Vision and Pattern Recognition, IEEE Computer Society Conference on 0, 1–8 (2008) 15. Horn, B., Schunck, B.: Determining optical flow. Artificial Intelligence 17, 185–203 (1981) 16. Huang, C.L., Liao, B.Y.: A robust scene-change detection method for video segmenta- tion. IEEE Trans. Circuits Syst. Video Techn. 11(12), 1281–1288 (2001) 17. Hunt, R.: The reproduction of colour (2004). 6th edition, John Wiley and Sons Ltd 18. Ince, S., Konrad, J.: Geometry-based estimation of occlusions from video frame pairs. IEEE International Conference on Acoustics, Speech, and Signal Processing 2, 933–936 (2005) 19. Izquierdo, M.: Stereo matching for enhanced telepresence in 3-dimensional videocom- munications 7, 629–643 (1997) 20. Jacobson, R., Ray, S., Attridge, G., Axford, N.: The manual of photography: photo- graphic and digital imaging (2000). 9th edition, Focal Press 21. Kennel, G.: Color and mastering for digital cinema (2007). Focal Press 22. Kodak: The essential reference guide for filmmakers (2010). Avaible on http://motion.kodak.com 23. Memin, E., Perez, P.: Dense estimation and object-based segmentation of the optical flow with robust techniques. Image Processing, IEEE Transactions on 7(5), 703–719 (1998) 24. Mess, C.: The theory of the photographic process (1954). New York : McMilian 25. Mileva, Y., Bruhn, A., Weickert, J.: Illumination-robust variational optical flow with photometric invariants. In: Proceedings of the 29th DAGM con- ference on Pattern recognition, pp. 152–162. Springer-Verlag (2007). URL http://portal.acm.org/citation.cfm?id=1771530.1771548 26. Moln´ar,J., Chetverikov, D., Fazekas, S.: Illumination-robust variational optical flow using cross-correlation. Comput. Vis. Image Underst. 114, 1104–1114 (2010) 27. Otsu, N.: A threshold selection from gray level histograms. IEEE Trans. Systems, Man and Cybernetics 9 (1979) 24 First Author, Second Author

28. Pratt, W.: Digital image processing (1991). 1st edition, John Wiley and Sons Ltd 29. Silberstein, L., Trivelli, A.: Relations between the sensitometric and the size-frequency characteristics of photographic emulsions (1938). Journal of the Optical Society of America 30. Smith, W.: Modern optical engineering: The design of opitical systems (2007). 4th edition, McGraw-Hill Professional 31. Thoma, R., Bierling, M.: Motion compensating interpolation considering covered and uncovered background. Signal Processing: Image Communication 1(2), 191–212 (1989) 32. Tsekeridou, S., Pitas, I.: Content-based video parsing and indexing based on audio- visual interaction (2001) 33. Worthey, J., Brill, M.: Heuristic analysis of von kries color constancy. J. Opt. Soc. Am. A 3 pp. 1708–1712 (1986) 34. Xiao, J., Cheng, H., Sawhney, H., Rao, C., Isnardi, M., Corporation, S.: Bilateral filtering-based optical flow estimation with occlusion detection. European Conference on Computer Vision pp. 211–224 (2006) 35. Yeo, B., Liu, B.: Rapid scene analysis on compressed video. IEEE Transactions on Circuits and Systems for Video Technology 5, 533–544 (1995)