Multimedia Information Systems
Total Page：16
File Type：pdf, Size：1020Kb
Multimedia Information Systems Multimedia Compression-MPEG Multimedia Compression ❚ Fundamentals of Data Compression ❚ Image Compression & JPEG ❚ Video Compression & MPEG-Video ❚ References ❙ V. O. K. Li & W. J. Liao, Distributed Multimedia Systems, Proceedings of IEEE, Vol. 85, No. 7, 1997. (1083-1089) ❙ D. L. Gall, MPEG: A Video Compression Standard for Multimedia Applications, CACM, Vol. 34, No. 4, 1991. ❙ http://www.cis.ohio- state.edu/hypertext/faq/usenet/compression-faq ❙ http://bmrc.berkeley.edu/frame/research/mpeg/mpegfaq.thml Standards ❚ Important for communications ❚ Customers prefer standards (freedom to choose) ❚ Increases volumes and bring down cost of service and SW/HW ❚ Reduce risk of deploying new technology ❚ Major players often participate ❚ Facilitate development on a common background ❚ Provide research opportunity Types of Standards ❚ Industrial/Commercial standards ❙ mutual agreement among companies ❙ may become de facto standards ❚ Voluntary standards ❙ By volunteers in open committee ❙ Based on consensus ❙ Market driven ❙ Stay ahead of technology Global Standards ❚ International ❙ ITU: International Telecommunications Union (UN) ◆ ITU-T: ITU Telecommunication Standardization Sector (CCITT) ◆ ITU-R: ITU Radio Communication Sector (CCIR) ❙ ISO: International Standards Organization ❙ IEC: International Electrotechnical Commission ❙ JTC1: Joint Technical Committee on Information Technology ❚ National ❙ ANSI: American National Standards Institute Organizations of an ISO Standard Body ❚ Group: WG1 (JPEG), WG11(MPEG) ❚ Convenor: Danial Lee(JPEG), Leonardo Chiariglione (MPEG) ❚ Sub-Group: Video, System, Audio and Conformance ❚ Ad Hoc Group: Coding Efficiency, Encoder Optimization ❚ NB: National Body Delegates ❚ HoD: Head of Delegation ❚ Observer ISO/IEC JTC1 SC29 ❚ Study Committee (SC) 29 ❙ Working Group (WG) 1 ◆ Joint Bi-level Image Group(JBIP) ◆ Joint Photographic Experts Group (JPEG) ❙ WG11 ◆ Moving Picture Experts Group (MPEG) ❙ WG12 ◆ Multimedia Hypermedia Experts Group (MHEG) How does Standards Work? ❚ Schedule ❙ 3 to 5 one week meeting in different nations each year ❙ 300 to 400 delegate from around the world ❙ 200 companies from over 50 nations ❙ A final standard in about 4-5 years Proposal Review Process ❚ Call for proposal ❚ CE: Core Experiments Process ❙ Complete descriptions with at least one independent verification ❙ One functionality one tool as reviewed by peer ❙ Consensus based decision process at AHG, SG level ❚ CE: Core Experiment Description ❙ Proposal that is relevant and is supported by two companies ❚ VM: Verification Model ❙ The best proposal is admitted to VM for everyone to implement ❙ The new reference for the best performance ❙ The proposal needs to be challenged by incoming proposal Proposal Review Process (Cont.) ❚ WD: Working Draft ❚ CD: Committee Draft ❙ All the doors will be frozen ❙ First round vote by National Bodies with comment ❚ FCD: Final Committee draft ❚ DIS: Draft International Standard ❙ Second round vote by National Bodies without comment ❚ IS:International standard Why Does Company Work in the Standards ❚ Interoperability: war of formats (VHS vs. Beta) ❚ Patent Royalties ❙ licensing fee for MPEG-2 box: US$4 ❙ Total licensing fee for DVD US$10 ❙ Big companies can avoid being taxed by other companies ❙ $250 Millions per year for RCA patent profiles ❚ Create new market ❙ VCD: Video Compact Disk ❙ DVD: Digital Versatile Disk ❙ DBS: Direct Broadcast System ❙ HDTV or DVB (digital video broadcast) MPEG ❚ Motion Picture Expert Group ISO/IEC JTC1/SC29/WG11 ❚ MPEG Standards ❙ MPEG-1 (ISO/IEC 11172, Nov. 92) ❙ MPEG-2 (ISO/IEC 13814, Nov. 94) ❙ MPEG-4 (ISO/IEC 14496, Oct, 98) ❙ MPEG-7 (ongoing) ❙ MPEG-21(ongoing ) ❚ Only bit stream syntax & decoding are specified History of MPEG ❚ MPEG-1 ❙ started in 1988 ❙ Coding of moving pictures and associated audio for digital storage media at up to about 1.5 Mb/s ❙ compression standards for progressive frame-based video in SIF (360x240) –source input format ❙ Applications; VCD ❚ MPEG-2 ❙ Generic coding of moving pictures and associated audio ❙ compression standard for interlaced frame-based video in CCIR-601(720x480) and HDTV (1920x1088) ❙ Applications: DVD, SVCD, Direct TV, DVB, HDTV History of MPEG (Cont.) ❚ MPEG-4 ❙ Very low bit rate audio-visual coding ❙ Multimedia Standard for object-based video from natural or synthetic source ❙ Applications: Internet, cable TV, virtual studio, home LAN ❚ MPEG-7 ❙ Multimedia content description interface ❙ Applications: Internet, video search engine, digital library Applications of MPEG-1 & 2 ❚ Digital storage media: 1~1.5 Mb/s ❚ Asymmetric applications ❙ Electronic publishing ◆ education & training ◆ travel guidance ◆ videotext ◆ point of sale ❙ Games ❙ Entertainment ❚ Symmetric Applications ❙ Electronic publishing ❙ Video mail ❙ Video-telephone ❙ Video conferencing JPEG H.261 MPEG-1 MPEG-2 Application multilevel still video-based multimedia and Digital NTSC images tele-communica broadcast TV and video- tions on-demand Resolution 352x288 352x288 720x480 Data rate 64kbps ~ 1.5Mbps 4 ~ 10Mbps 2.048Mbps Full Motion No Yes Yes Yes Picture Rate 30 24 ~ 30 30 Compression 15:1 100:1 ~ 200:1 200:1 100:1 ratios Features of Video Compression Algorithm ❚ Random access (latency: 0.5 seconds) ❚ Fast Forward/Reverse Searches ❚ Reverse playback ❚ Audio-Visual Synchronization ❚ Robustness to errors ❚ Coding/decoding delay (150ms of videotelephone) ❚ Editability ❚ Format Flexibility: raster size & frame rate ❚ Cost tradeoffs: implementable Principles of MPEG ❚ Spatial redundancy (Intra-frame) ❙ DCT ❚ Temporal redundancy(Inter-frame) ❙ block based motion compensation ❙ MB: 16x16 macro block ❙ Prediction, Interpolation Forward Prediction 1 2 3 4 5 6 7 8 9 10 11 12 13 I B B B P B B B P B B B I Bidirectional Prediction Motion estimation and compensation ❚ Motion estimation: estimate motion parameters of moving objects in an image sequence ❙ At the encoder ❚ Motion compenstion: replace a picture or portion thereof, based on displaced pels of a previously transmitted frame in an image sequence ❙ At the decoder ❚ Why motion compensation ❙ Reduce interframe correlation ❙ Block motion compensation is adopted by H/261/H.263, MPEG1/2 Motion Estimation ❚ Predict current frame from previous frame ❚ Motion Estimation approach ❙ Block matching method ◆ Pel based ◆ Block based ◆ Object based ❙ Differential (gradient) method – optical flow ❙ Fourier method Motion estimation problem ❚ Moving object: a group of contiguous perls that share the same set of motion parameters – not necessarily match the ordinary meaning of object ❚ Assumptions: ❙ Objects are rigid body: object deformation can be neglected for at least a few nearby frames ❙ Objects move only in translational movement for at least a few frames Motion estimation problem (cont) ❚ Assumptions: (cont) ❙ Illumination is spatially and temporally uniform; the observed object intensities are unchanged under movement ❙ Occlusion of one object by another and uncovered background are neglected Block matching motion estimation ❚ Concept: correlation technique that searches for the best match between the current image block and candidates in a confined area of previous frame ❚ Assumptions: images are partitioned into non-overlapped rectangular blocks ❙ Each block is viewed as an independent object ❙ The motion of pels within ghe same block is uniform Motion Estimation (x,y) (x,y) Search position (4) Motion (1) Current Vector Macroblock (2) Defined Search Window (3) Best matched Predicted Reference macroblock within Picture Picture the search window Factors Affect Block-based Matching Algorithm •Searching algorithm, order •Matching criteria •Searching range Full search W W b Current frame Reference frame Search position (2W +1) 2 Logarithmic search Cross search algorithm 2 2 1 1 3 3 2 2 3 3 3 3 1 1 Search position: 5+ 4 log2 w Matching function ❚ Mean squared error (MSE): (min) ∑∑ MSE(d1, d2) = 1 / N1N2 [f(n1, n2, t) -f(n1-d1, n2-d2, t-1) ] Mean absolute difference (MAD): (min) ∑∑ MSE(d1, d2) = 1 / N1N2 |f(n1, n2, t) -f(n1-d1, n2-d2, t-1) | d1, d2 are distance away MPEG 1/ 2 A Motion Compensated interpolation Previous frame B current MC interpolation modes: C 1.Block B = Block A 2. Block B= Block C 3. Block B = (A + C)/2 Future frame MPEGF-Decoder Coded Data VLC-1 Scan-1 Q-1 DCT-1 I frame Output Motion + Compensation P/B frame Framestore Memory MPEG Encoder Rate Input Controller Pictures Inter - DCT Q Scan VLC Buffer Output Intra Q-1 DCT-1 Motion Motion Estimator Compensation + Coding Mode Decision Framestore Memory VLC : Varialbe-length Coding Q : Quantization DCT : DCT Transform Parts of MPEG-1 ❚ ISO/IEC 11172-1: Systems ❚ ISO/IEC 11172-2: Video ❚ ISO/IEC 11172-3: Audio ❚ ISO/IEC 11172-4: Conformance Testing ❚ ISO/IEC 11172-5: Software Parameters of MPEG-1 ❚ Picture size: up to 4096x4096,normally 360x240 ❚ Pel aspect ratio: choices ❚ Picture rates: 23.976, 24,25, 29.97, 30, 50, 59.94, 60 ❚ 4:2:0 format MPEG Layers Layers of the syntax Function Sequence layer Random access unit : context Group of pictures layer Random access unit : video Picture Layer Primary coding unit Slice layer Resynchronization unit Macroblock layer Motion compensation unit Block layer DCT unit Video stream structure Sequence Sequence GOP1 …… GOP i …… GOP n Sequence layer header end GOP Picture 1 ….. Picture j …. Picture m GOP header layer Picture Slice 1 ….. Slice k …. Slice p Picture header layer Slice Macroblock 1 ….. Macroblock l …. Macroblock q Slice header layer Macroblock Macroblock block 1 block 2