Transform Coding III s1 u1 cos φ sin φ − sin φ cos φ s0 u0 Signal-Independent Unitary Transforms Optimal Transforms & Adaptation Optimal Unitary Transform Stationary Gaussian sources: KLT General sources: Not straightforward to determine Signal dependent (may change due to signal instationarities) Backward Adaptive Transform Estimate transform in encoder and decoder based on reconstructed samples Difficult, not often used in practice Forward Adaptive Transform Determine transform in encoder, include transform specification in bitstream Increased side information Simple variant: Switched transforms Signal-Independent Transforms Choose transform that provides good performance for variety of signals Not optimal, but often close to optimal for typical signals T. Wiegand (TU Berlin) — Image and Video Coding: Transform Coding III 2 / 38 Signal-Independent Unitary Transforms / Walsh-Hadamard Transform Walsh-Hadamard Transform For transform sizes N that are positive integer powers of 2 1 AN=2 AN=2 h i AN = p with A1 = 1 : (1) 2 AN=2 −AN=2 Example: Transform matrix for N = 8 2 1 1 1 1 1 1 1 1 3 6 1 −1 1 −1 1 −1 1 −1 7 6 7 6 1 1 −1 −1 1 1 −1 −1 7 6 7 1 6 1 −1 −1 1 1 −1 −1 1 7 A = p · 6 7 (2) 8 8 6 1 1 1 1 −1 −1 −1 −1 7 6 7 6 1 −1 1 −1 −1 1 −1 1 7 6 7 4 1 1 −1 −1 −1 −1 1 1 5 1 −1 −1 1 −1 1 1 −1 Very simple orthogonal transform (only additions & final scaling) Image & video coding: Piece-wise constant basis vectors yield subjectively disturbing artifacts when combined with strong quantization T. Wiegand (TU Berlin) — Image and Video Coding: Transform Coding III 3 / 38 Signal-Independent Unitary Transforms / Discrete Fourier Transform Discrete Fourier Transform (DFT) Fourier Transform Integral transform representing a signal as integral of frequency components Forward and inverse transforms are given by 1 1 Z Z X (f ) = x(t) · e−2πift dt () x(t) = X (f ) · e2πift df (3) −∞ −∞ Discrete Fourier Transform Unitary transform (specified by unitary matrix) Discrete version of Fourier transform Input: Discrete signal with finite number N of samples Output: N Fourier coefficients (complex values) Can be derived from Fourier transform using sampling (see illustrations on following slides) T. Wiegand (TU Berlin) — Image and Video Coding: Transform Coding III 4 / 38 Signal-Independent Unitary Transforms / Discrete Fourier Transform Discrete Fourier Transform (DFT) T. Wiegand (TU Berlin) — Image and Video Coding: Transform Coding III 5 / 38 Signal-Independent Unitary Transforms / Discrete Fourier Transform Discrete Fourier Transform (DFT) T. Wiegand (TU Berlin) — Image and Video Coding: Transform Coding III 6 / 38 Signal-Independent Unitary Transforms / Discrete Fourier Transform Discrete Fourier Transform (DFT) T. Wiegand (TU Berlin) — Image and Video Coding: Transform Coding III 7 / 38 Signal-Independent Unitary Transforms / Discrete Fourier Transform Discrete Fourier Transform (DFT) Discrete Fourier Transform Forward Transform N−1 1 X −i 2πkn u[k] = p s[n] · e N (4) N n=0 Inverse Transform N−1 1 X i 2πkn s[n] = p u[k] · e N (5) N k=0 Properties Unitary transform that produces complex transform coefficients For real inputs, it obeys the symmetry u[k] = u∗[N − k] N real samples are mapped onto N real values Fast algorithm: Fast Fourier transform (FFT) T. Wiegand (TU Berlin) — Image and Video Coding: Transform Coding III 8 / 38 Signal-Independent Unitary Transforms / Discrete Fourier Transform Disadvantage of DFT for Transform Coding Sampling of frequency spectrum causes implicit periodic signal extension Will often observe rather large difference between left and right signal boundary Large difference reduces rate of convergence of Fourier series Strong quantization yields significant high-frequency artifacts T. Wiegand (TU Berlin) — Image and Video Coding: Transform Coding III 9 / 38 Signal-Independent Unitary Transforms / Discrete Trigonometric Transforms Overcome DFT Disadvantage: Discrete Cosine Transform DFT: implicit signal replica signal implicit signal replica DCT-II: DFT implicit signal replica signal mirrored signal implicit signal replica Idea of Discrete Cosine Transform (DCT) Introduce mirror symmetry (different possibilities, example: DCT-II) Apply DFT of approximately double size (or four times the size) No discontinuities in periodic signal extension Ensure symmetry around zero: Only cosine terms T. Wiegand (TU Berlin) — Image and Video Coding: Transform Coding III 10 / 38 Signal-Independent Unitary Transforms / Discrete Trigonometric Transforms Discrete Trigonometric Transforms (DTTs) Discrete Cosine Transforms (DCTs) Introduce mirror symmetry around zero and apply DFT of larger size Imaginary sine terms get eliminated Only cosine terms remain 8 possibilities: DCT-I to DCT-VIII 2 cases for left side: Whole sample or half-sample symmetry 4 cases for right side: Whole sample or half-sample symmetry or whole sample or half-sample anti-symmetry Most relevant case: DCT-II (half-sample symmetry at both sides) Discrete Sine Transforms (DSTs) Introduce anti-symmetry around zero and apply DFT of larger size Real cosine terms get eliminated Only imaginary sine terms remain Similarly as for DCT: 8 possibilities (DST-I to DST-VIII) T. Wiegand (TU Berlin) — Image and Video Coding: Transform Coding III 11 / 38 Signal-Independent Unitary Transforms / Discrete Trigonometric Transforms The Discrete Cosine Transform (DCT) Family DCT-I DFT of size 2N − 2 DCT-V DFT of size 2N − 1 DCT-II DFT of size 2N DCT-VI DFT of size 2N − 1 DCT-III DFT of size 4N DCT-VII DFT of size 4N − 2 DCT-IV DFT of size 4N DCT-VIII DFT of size 4N + 2 T. Wiegand (TU Berlin) — Image and Video Coding: Transform Coding III 12 / 38 Signal-Independent Unitary Transforms / Discrete Trigonometric Transforms The Discrete Sine Transform (DST) Family DST-I DFT of size 2N + 2 DST-V DFT of size 2N + 1 DST-II DFT of size 2N DST-VI DFT of size 2N + 1 DST-III DFT of size 4N DST-VII DFT of size 4N + 2 DST-IV DFT of size 4N DST-VIII DFT of size 4N − 2 T. Wiegand (TU Berlin) — Image and Video Coding: Transform Coding III 13 / 38 Signal-Independent Unitary Transforms / Discrete Cosine Transform of Type II Derivation of Discrete Cosine Transform Type II (DCT-II) DFT implicit signal replica signal mirrored signal implicit signal replica Specification signal for applying DFT Given: Discrete signal s[n] of size N (i.e., 0 ≤ n < N) Mirror signal with sample repetition at both sides (size 2N) s[n]: 0 ≤ n < N sm[n] = s[2N − n − 1]: N ≤ n < 2N Ensure symmetry around zero by adding half-sample shift s[n − 1=2]: 0 ≤ n < N s+[n] = sm[n − 1=2] = s[2N − n − 3=2]: N ≤ n < 2N Apply DFT of size 2N to new signal s+[n] T. Wiegand (TU Berlin) — Image and Video Coding: Transform Coding III 14 / 38 Signal-Independent Unitary Transforms / Discrete Cosine Transform of Type II Derivation of Discrete Cosine Transform Type II (DCT-II) s[n − 1=2]: 0 ≤ n < N s+[n] = s[2N − n − 3=2]: N ≤ n < 2N DFT of size 2N for extended signal s+[n] ( N)− 2 1 + 1 X −i 2πkn s only known at half-sample u+[k] = s+[n] · e (2N) p(2 ) positions ! use m = n − 1=2 N n=0 2N−1 1 X + 1 −i πk m+ 1 = p s m + · e N ( 2 ) 2 2 N m=0 N−1 2N−1 ! 1 X −i πk n+ 1 X −i πk m+ 1 = p s[n] · e N ( 2 ) + s[2N − m − 1] · e N ( 2 ) 2 N n=0 m=N ? yn=2N−m−1 N−1 N−1 ! 1 X −i πk n+ 1 X −i πk N−n− 1 = p s[n] · e N ( 2 ) + s[n] · e N (2 2 ) 2 N n=0 n=0 T. Wiegand (TU Berlin) — Image and Video Coding: Transform Coding III 15 / 38 Signal-Independent Unitary Transforms / Discrete Cosine Transform of Type II Derivation of Discrete Cosine Transform Type II (DCT-II) Continue derivation N−1 N−1 ! + 1 X −i πk n+ 1 X −i πk N−n− 1 u [k] = p s[n] · e N ( 2 ) + s[n] · e N (2 2 ) 2 N n=0 n=0 0 1 N−1 N−1 1 X −i πk (n+ 1 ) X −i2πk i πk (n+ 1 ) = p @ s[n] · e N 2 + s[n] · e · e N 2 A 2N | {z } n=0 n=0 1 N−1 1 X −i πk n+ 1 i πk n+ 1 = p s[n] · e N ( 2 ) + e N ( 2 ) 2N n=0 | {z } πk 1 2 cos( N (n+ 2 )) DFT of extended signal r N−1 2 X π 1 u+[k] = · s[n] · cos k n + (6) N N 2 n=0 T. Wiegand (TU Berlin) — Image and Video Coding: Transform Coding III 16 / 38 Signal-Independent Unitary Transforms / Discrete Cosine Transform of Type II Derivation of Discrete Cosine Transform Type II (DCT-II) DFT of extended signal (2N samples) has 2N transform coefficients r N−1 2 X π 1 k = 0;:::; 2N − 1 : u+[k] = · s[n] · cos k n + N N 2 n=0 1 Signal s[n] is completely described by first N transform coefficients r N−1 2 X π 1 k = 0;:::; N − 1 : u+[k] = · s[n] · cos k n + N N 2 n=0 2 Basis functions of derived transform are orthogonal to each other, but they don’t have the same norm Introduce factors αk so that transform matrix becomes orthogonal N−1 X π 1 k = 0;:::; N − 1 : u[k] = αk · s[n] · cos k n + N 2 n=0 T.