Overview: motion-compensated coding n Motion-compensated prediction n Motion-compensated hybrid coding n Power spectral density of the motion-compensated prediction error n Rate-distortion analysis n Loop filter n Motion compensation with sub-pel accuracy Bernd Girod: EE368b Image and Video Compression Motion Compensation no. 1 Motion-compensated prediction previous frame stationary Dt background current frame x time t y moving object æ dx ö „Displacement vector“ ç d ÷ shifted è y ø object Prediction for the luminance signal S(x,y,t) within the moving object: ˆ S (x, y,t) = S(x - dx , y - dy ,t - Dt) Bernd Girod: EE368b Image and Video Compression Motion Compensation no. 2 1 Motion-compensated prediction: example Previous frame Current frame Current frame with Motion-compensated displacement vectors Prediction error Bernd Girod: EE368b Image and Video Compression Motion Compensation no. 3 Motion-compensated hybrid coder Control „side information“ Video Channel signal Intraframe S e DCT coder - Intraframe Decoder Motion compensated predictor Decoder Bernd Girod: EE368b Image and Video Compression Motion Compensation no. 4 2 Motion-compensated hybrid decoder Channel reconstructed Intraframe video signal Decoder Motion compensated predictor side information Bernd Girod: EE368b Image and Video Compression Motion Compensation no. 5 Model for performance analysis of an MCP hybrid coder luminance signal S e R-D optimal intraframe - encoder intraframe decoder e‘ motion s‘ compensated predictor displacement estimate æ dx ö æ D xö true displacement ç ÷ + ç ÷ displacement error è dyø è D yø Bernd Girod: EE368b Image and Video Compression Motion Compensation no. 6 3 Analysis of the motion-compensated prediction error Motion-compensated signal c(x) = s(x - Dx )- n(x) Prediction error Previous e(x) = s(x)- c(x) frame = s(x)- s(x - Dx )+ n(x) x c(x) s(x) Current frame Displacement dx Displacement error Dx x Bernd Girod: EE368b Image and Video Compression Motion Compensation no. 7 Analysis of m.c. prediction error (cont.) n Motion-compensated prediction error e(x)= s(x)-c(x)=s(x)-s(x-Dx)+n(x)=(d(x)-d(x-Dx ))*s(x)+n(x) n Power spectrum of prediction error, assuming constant displacement error D x, statistical independence of s and n - jwD x jwD x F ee(w) = F ss(w)(1 - e )(1 - e )+ Fnn (w ) = 2F (w ) 1- Re e- jwD x + F (w ) ss ( { }) nn n Random displacement error D x, statistically independent from s, n F ( ) = E 2F ( ) 1 - Re e- jwD x + F ( ) ee w { ss w ( { }) nn w } = 2F ( ) 1- Re E e- jwD x + F ( ) ss w ( { { }}) nn w = 2Fss (w )(1- Re{P(w )})+ Fnn (w) Bernd Girod: EE368b Image and Video Compression Motion Compensation no. 8 4 Analysis of m.c. prediction error (cont.) - jw D n What is P(w)? P(w ) = E{e x } ¥ - jw D = p (D )e dD = F p D x ò D x { D x ( )} -¥ Fourier transform of the displacement error pdf!! n Same as characteristic function of displacement error, except for sign n Extension to 2-d F ee (w x ,w y ) = 2F ss (w x ,w y )(1- Re{P(w x ,w y )})+ F nn (w x ,w y ) Fourier transform of the noise spectrum displacement error pdf power spectrum of p(D , D ) luminance signal x y Bernd Girod: EE368b Image and Video Compression Motion Compensation no. 9 Power spectrum of motion-compensated prediction error 0 p frequency w x Bernd Girod: EE368b Image and Video Compression Motion Compensation no. 10 5 R-D function for MCP with integer-pixel accuracy Minimum bit-rate for given SNR T n (D x , D y ) assumed uniformly distributed between 1 D = ± pel x 2 1 D = ± line ~0.7 bpp y 2 n Gaussian signal model 3 2 2 - 2 æ w x +w y ö Fss (w x ,w y ) = Aç 1+ 2 ÷ è w 0 ø n Typical parameters for CIF resolution (352 x 288 pixels) Bernd Girod: EE368b Image and Video Compression Motion Compensation no. 11 Required accuracy of motion compensation n p(D x , D y ) isotropic Gaussian pdf with variance s 2 3 - 2 2 2 æ wx +w y ö n Fss (wx ,wy ) = Aç 1+ 2 ÷ è w0 ø n Typical parameters for CIF resolution (352 x 288 pixels) n Minimum bit-rate for SNR = 30 dB Bernd Girod: EE368b Image and Video Compression Motion Compensation no. 12 6 Model of MCP hybrid coder with loop filter e R-D optimal intraframe luminance - encoder signal S intraframe decoder spatial filter F motion compensated predictor displacement estimate æ dx ö æ D x ö true displacement ç ÷ + ç ÷ displacement error è d y ø è D y ø Bernd Girod: EE368b Image and Video Compression Motion Compensation no. 13 Motion-compensated prediction error with loop filter Motion-compensated signal Prediction error c(x) = s(x - Dx )- n(x) e(x) = s(x)- f(x)*c(x) = s(x)- f(x)*s(x - Dx)+ f(x)*n(x) Previous frame Impulse response of loop filter x c(x) s(x) Current frame Displacement dx Displacement error Dx x Bernd Girod: EE368b Image and Video Compression Motion Compensation no. 14 7 Spatial power spectrum of m.c. prediction error with loop filter 2 2 Fee(L) = Fss(L)(1+| F(L)| -2Re{F(L)P(L)})+Fnn(L)| F(L)| P(L) 2 -D Fourier transform of displacement error pdf F(L) 2 -D Fourier transform of f (x, y) Fuu spatial spectral power density of signal u L vector of spatial frequencies (w x ,w y ) n(x, y) noise Bernd Girod: EE368b Image and Video Compression Motion Compensation no. 15 Optimum loop filter n Wiener filter minimizes prediction error variance F (L) F (L) = P*(L) × ss opt 12 3 F L + F L 1ss4 (4 4) 2 4 nn4 (43 ) accounts for accuracy of accounts for noise motion estimation n To determine Wiener filter from measurements: cross spectrum between s(x,y) and the motion-compensated signal F (L) ˆ ˆ sc c(x, y) = r(x - d x , y - d y ) Fopt (L) = F cc (L) Bernd Girod: EE368b Image and Video Compression Motion Compensation no. 16 8 Required accuracy of motion compensation with loop filter 2 n p(D x , D y ) isotropic Gaussian pdf with variance s n Minimum bit-rate for SNR = 30 dB ~0.8 bpp Bernd Girod: EE368b Image and Video Compression Motion Compensation no. 17 Experimental evaluation of sub-pixel motion compensation n ITU-R 601 TV signals, 13.5 MHz sampling rate, interlaced, blockwise motion compensation with blocksize16x16 Zoom Voiture Bernd Girod: EE368b Image and Video Compression Motion Compensation no. 18 9 Influence of noise on the performance of MCP Bernd Girod: EE368b Image and Video Compression Motion Compensation no. 19 Motion Compensation Performance in H.263 500 400 Intra mode only 300 Integer-pel accuracy Bit Rate [kbps] 200 1/2-pel 100 accuracy 0 24 30 36 Simulation details: PSNR [dB] Foreman, QCIF, SKIP=2 Q=4,5,7,10,15,25 Bernd Girod: EE368b Image and Video Compression Motion Compensation no. 20 10 Summary: motion-compensated coding n Motion-compensated prediction exploits similarity of successive frames of a video sequence. n Hybrid coder combines motion compensation and spatial 2D coding. n Power spectral density of motion-compensated prediction error is flat. n Loop filter improves the prediction. n Maximum gain by motion compensation with integer-pel accuracy ~0.8 bit/sample. n Sub-pixel accuracy of motion compensation can improve prediction n “Noise” in the image signal limits the accuracy of motion compensation. Bernd Girod: EE368b Image and Video Compression Motion Compensation no. 21 11.