An Introduction to Video Compression Typical Camera An Introduction to Video Compression Image Image Video Sensor Processor Interface 4 AGENDA Image Sensors (CCD/CMOS) • Video Basics • Each cell is a monochrome – Analog Video detector – Digital Video • A filter array is used to make – Scanning Formats them sensitive to different color • Video Compression frequencies – Intra-Frame Coding • More green cells are used than red and blue cells because – Inter-Frame Coding human vision is more sensitive • Video Quality to green frequencies • Video Coding Standards • This is called a Bayer pattern • Audio Compression 2 5 Image Processing • Image processing internal to the camera converts the Bayer pattern to the desired Video Basics camera output format • Demosaicing and Interpolating algorithms are used to generate the output pixels • Typical outputs include RGB and YCrCb 3 6 Video Interface Analog Composite Video Formats • National Television Standard Committee (NTSC) – Interlaced: 2 fields/frame • Converts the camera’s pixel array data to an – 525 lines/frame, 262.5 lines/field output signal – 29.97 frames/second, 59.94 fields/second – 63.5 uSeconds/line • Horizontal/Vertical scanning of image array – Color burst: 3.58 MHz • Parallel to Serial conversion of pixel data • Phase Alternate Line (PAL) – Interlaced: 2 fields/frame • Synchronization, timing and status insertion – 625 lines/frame, 312.5 lines/field – 25 frames/second, 50 fields/second • Electrical signal generation – 64 uSeconds/line – Color burst: 4.43 MHz • Séquentiel couleur à mémoire (SECAM) 7 10 Analog Composite Video Waveform Digitized Composite Video Formats • NTSC – 720 horizontal x 480 vertical pixels – 2:1 Interlaced – 29.97 Frames per Second (59.94 Fields per Second) – 165.888 M Bits per Second • PAL – 720 horizontal x 576 vertical pixels – 2:1 Interlaced – 25 Frames per Second (50 Fields per Second) – 165.888 M Bits per Second 8 11 Analog Video Image Other Analog Formats • S-Video – Y – Luminance Signal – C – Chrominance Signal – Eliminates need for color separation filtering – Improved quality over Composite Video • YPrPb Component – Y – Luminance Signal – Pr – Red-Green Color Difference Signal – PB – Blue-Green Color Difference Signal – Supports High Definition Resolutions • RGB Component – Red, Green, and Blue Signals – Separate Sync signal – Sync on Green – Primarily a computer monitor format 9 12 Digital Broadcast Interfaces New Digital Broadcast Interfaces • Serial Digital Interface • New Serial Digital Interfaces – SMPTE 259 SDI (270Mbps) • 480i30 (720x480) – SMPTE 2081 6G-SDI (5.94Gbps) • 576i25 (720x576) • 4Kp30 (3840x2160) – SMPTE 292 HD-SDI (1.485Gbps, 1.485/1.001Gbps) – SMPTE 2082 12G-SDI (11.88Gbps) • 720p60 (1080x720) • 4Kp60 (3840x2160) • 1080i30 (1920x1080) • 1080p30 (1920x1080) – SMPTE 2083 24G-SDI (23.76Gbps) – SMPTE 424 3G-SDI (2.970Gbps, 2.970/1.001Gbps) • 8Kp30 (7680x4320) • 1080p60 (1920x1080) 13 16 Serial Digital Interface Line Broadcast Interface Machine Vision Interface • Fixed resolutions and • Random resolutions and frame rates frame rates • Unidirectional interface • Bidirectional interface • No control channel • Control channel • Fire and forget • Often requires handshake • Format indicated within • Format configured in the signal control channel • Coax cable • Most use complex cables • No trigger capability • Triggers • No camera power • Camera power 14 17 Serial Digital Interface Frame Machine Vision Interfaces • GigE Vision – Uncompressed video over UDP on 1-Gigabit and 10-Gigabit Ethernet networks – Cat5 cabling, Ethernet infrastructure – 125MB/s (1250MB/s) transfer rate • USB3 Vision – Same API as GigE Vision over USB 3.0 connections 15 18 Machine Vision Interfaces (Cont) Interlaced Scanning Format 1 • CoaXPress 264 2 – 6.25 Gbps over coax cable Retrace 265 – Group multiple cables (4 cables = 25 Gbps) 3 – Future expansion to 10 and 12.5 Gbps Odd field scan line • CameraLink – 26-pin Mini-D ribbon connector 524 – LVDS signal pairs, 2.04 Gbps 262 – Group multiple cables (2 cables = 4.08 Gbps) 525 – Video data, discrete signals, control channel 263 Even field scan line 19 22 Machine Vision Interfaces (Cont) Progressive Scanning Format 1 • IEEE-1394 FireWire 2 3 – Tree bus topology, bus master negotiations Retrace and self identification 4 5 – 400 and 800 Mbps (1600 and 3200 Mbps defined but not widely supported). Scan line – 4, 6 and 9 conductor connectors • DVI/HDMI 522 – 19 pin connector 523 524 – 48 Gbps data rate 525 – I2C Control channel – Copy Protection negotiation 20 23 Video Resolutions Interlace vs Progressive 21 24 Interlace vs Progressive Video Compression 25 28 Color Spaces The need for Video Compression • Digital Transmission. • Monochrome – Need to match video data rate to digital • Red, Green, Blue (RGB) communications system bandwidth. – Separate sync, Sync on Green • YIQ (NTSC, PAL) • Digital Storage. •YUV – Need to match video data rate to digital storage • YPbPr (Analog), YCbCr (Digital) system bandwidth. – Pb/Cb = Blue - Luma – Need to reduce storage capacity or increase storage – Pb/Cr = Red - Luma time. • HSV, HSB, HSL • Multiplexing •CMYK – Send more programs or other data over the same channel. 26 29 Other Characteristics Video Data Rates • Pixel Bit Depth • Uncompressed SD Video – 720x480, 30 fps, 16 bpp – 8, 10, 12, 14, 16 bits per component • 166 Mbps • Chroma Sampling • Uncompressed HD Video – 4:4:4, 4:2:2, 4:2:0, 4:0:0 – 1920x1080, 60 fps, 16 bpp • 1,990 Mbps • Uncompressed UHD Video – 7680x4320, 120 fps, 16 bpp • 63,701 Mbps 27 30 Digital Transmission Rates Types of Compression • Lossless • Transmission System Data Rates – Output image is numerically identical to the original image on a pixel-by-pixel basis. – Ethernet: 10/100 Mbps, 1/10 Gbps – Only statistical redundancy is reduced – OC12: 622 Mbps – Compression ratio is usually low – 2:1 to 4:1 – OC3: 155 Mbps – Reversible (infinite compress/decompress cycles) – DS3: 45 Mbps • Lossy – T1: 1.544 Mbps – Output image is numerically degraded relative to the original. – DSO: 64 Kbps – Statistical and perceptual redundancy is reduced – Modem: 33.6 Kbps – High compression due to reduction of perceptual – Cellular: 9600 redundancy – Can be visually lossless – Irreversible (compress/decompress degrades images) 31 34 Digital Storage Capacity Video Compression Standards • Uncompressed SD Video: 166 Mbps PRE-PROCESS POST- BIT- PROCESS -Noise Filter STREAM - Signal – 1 Min. Clip = 1.24 G Bytes -Scaling ENCODE DECODE Enhancement - Filter - Error – 30 Min. TV Show = 37.3 G Bytes Concealment – 2 Hour Movie = 149.3 G Bytes • Uncompressed HD Video: 1,990 Mbps • Functions Standardized • Functions Not Standardized – 1 Min. Clip = 14.9 G Bytes – Bit stream syntax – Pre-processing – 30 Min. TV Show = 447.8 G Bytes – How the decoder interprets the bit • Filtering, scaling, noise reduction stream – 2 Hour Movie = 1.79 T Bytes – Encoding strategy • Uncompressed UHD Video: 63,701 Mbps • Mode Selection • Quantizer Selection – 1 Min. Clip = 477.8 G Bytes • Block Pattern Selection – 30 Min. TV Show = 14.3 T Bytes • Motion Estimation – 2 Hour Movie = 57.3 T Bytes – Post-filter • Scaling, block filtering, error 32 35 concealment Digital Storage Devices Preprocessing (1/5) • Noise Reduction •CD – 185 to 870 M Bytes (various formats) – Filter out high frequency information and improves compression efficiency •DVD – 1.46 to 17.08 G Bytes (SS, SD, SL, DL) – Spatial Noise Reduction • BluRay Disk • Reduces high frequency information within a picture – 25 G Bytes (50 GB for dual layer disks) – Temporal Noise Reduction • Disk Drives • Reduces high frequency information between –> 1 T Byte frames 33 36 Preprocessing (2/5) Preprocessing (5/5) • Color representation • Resolution Reduction – Reduces the amount of spatial information – Reduces the amount of that needs to be compressed spatial information that needs to be 4:4:4 4:2:2 4:2:0 compressed – Scaling • Maintains field of view 24 16 12 Bits/Pixel Bits/Pixel Bits/Pixel 37 40 Y Cr Cb Preprocessing (3/5) Intra-Frame Coding • Resolution Reduction • Removes the redundancies within a frame – Reduces the amount of • Each frame is encoded separately without spatial information that regard to adjacent frames needs to be compressed • Similar to JPEG – Cropping • Maintains pixel resolution 38 41 Preprocessing (4/5) Encoding Stages • Frame Rate Reduction • Signal Analysis – Reduces the amount of temporal information that needs to be compressed – Transform from spatial domain to another domain that is easier to compress 12 3456 • Quantization Video Sequence – 30 Frames per Second – Discard less important information 135XXX • Variable Length Coding – Last stage of efficient coding Video Sequence – 15 Frames per Second 39 42 Video Compression System Transform Coding by Discrete Cosine Transform (DCT) Signal Analysis Variable length coding Segmentation of a frame INTRAFRAME Quantization (entropy coding) - Predictive into blocks of pixels + 1D, 2D predictor - Variable precision -Fixed + Fixed, adaptive + Huffman - Block transform + B + DCT - Visibility matrix + Comma + Karhunen Loeve Uncompressed - Prediction + 2 dimensional COMPRESSED + Hadamard input error (run length/COEFF) OUTPUT + Fourier e.g. 8 bits/pixel - Fixed, adaptive + Haar - Transform - Adaptive - Sub-band filtering coefficients + Conditional + Unconstrained 8x8 - Lossy, lossless + Arithmetic rectangular blocks - Filter energy + One, two passes Coefficients + QMF, other filters levels 8x8 Pixels + Wavelets - Scalar, vector - Lossless + Pyramidal - Block pattern matching (VQ) - Fractals