INF SERV – Media Storage and Distribution Systems: DataData FormatsFormats andand CodecsCodecs 1/9 – 2003 Why codecs and formats? ! Codecs (coders/decoders) " Determine how information is represented " Important for servers and distribution systems # Required sending speed # Amount of loss allowed # Buffers required # … ! Formats " Determine how data is stored " Important for servers and distribution systems # Where is the data? # Where is the data about the data? INF 5070 – media servers and distribution systems 2003 Carsten Griwodz & Pål Halvorsen Media data Media data ! Medium: "Thing in the middle“ " here: means to distribute and present information ! Media affect human computer interaction ! The mantra of multimedia users " Speaking is faster than writing " Listening is easier than reading " Showing is easier than describing INF 5070 – media servers and distribution systems 2003 Carsten Griwodz & Pål Halvorsen Dependence of Media ! Time-independent media " Text " Graphics " Discrete media ! Time-dependent media " Audio " Video ! "Continuous" refers to the " Animation user’s impression of the data, " Continuous media not necessarily to its representation ! Combined video and audio is ! Interdependant media multimedia - relations must " Multimedia be specified INF 5070 – media servers and distribution systems 2003 Carsten Griwodz & Pål Halvorsen Properties of a Multimedia System ! Flexibility " Provide mechanisms to handle all kinds of media, in particular, discrete and continuous media " A VCR and a desktop publishing system for text and graphics are no multimedia systems " An editor with voice annotation is a multimedia system ! Integration " Independent media storage " Computer-controlled media combination ! Definition A multimedia system is characterized by the integrated computer-controlled handling of independent discrete and continuous media INF 5070 – media servers and distribution systems 2003 Carsten Griwodz & Pål Halvorsen Multimedia: Not Your Ordinary Data ! Multimedia is different from traditional digital data: " High data volume " Continuous streaming " Several related streams " Quality of services INF 5070 – media servers and distribution systems 2003 Carsten Griwodz & Pål Halvorsen High Data Volume ! Throughput: " Higher volume than for traditional data " Longer transactions than for traditional data " Requires # Performance and bandwidth # Resource management techniques # Compression ! Typical values " Uncompressed video: 140 – 216 Mbit/s " Uncompressed audio (CD): 1.4 Mbit/s " Uncompressed speech: 64 Kbit/s " Compressed audio & video (VoD): down to 1.2 – 4 Mbit/s " Compressed audio & video (Conf.): down to 128 Kbit/s " Compressed speech: down to 6.2 Kbit/s INF 5070 – media servers and distribution systems 2003 Carsten Griwodz & Pål Halvorsen Coding for distribution Compression - Necessity ! E.g., video sequence " 25 images/sec. # PAL standard " 3 byte/pixel # YUV (luminance + 2 chrominance values) # RGB (red-green-blue values) " Image resolution 640 * 480 pixel # Data rate = 640 * 480 * 3 Byte * 25/s = 23040000 byte/s ~ 22 MByte/s o Approx. 1/100 stream over ADSL o Approx. 1/16 stream over Ethernet o Approx. 1/2 stream over Fast Ethernet # Compression is necessary INF 5070 – media servers and distribution systems 2003 Carsten Griwodz & Pål Halvorsen Compression – General Requirements ! Dependence on application type: " Dialogue mode " Retrieval mode INF 5070 – media servers and distribution systems 2003 Carsten Griwodz & Pål Halvorsen Compression – Mode Dependent Requirements ! Dialogue and retrieval mode requirements: " Synchronization of audio, video, and other media ! Dialogue mode requirements: " End-to-end delay < 150ms " Compression and decompression in real-time " Symmetric ! Retrieval mode requirements: " Fast forward and backward data retrieval " Random access within 1/2 s " Asymmetric ! We look mainly at retrieval mode! INF 5070 – media servers and distribution systems 2003 Carsten Griwodz & Pål Halvorsen Compression Categories INF 5070 – media servers and distribution systems 2003 Carsten Griwodz & Pål Halvorsen Basic Encoding Steps INF 5070 – media servers and distribution systems 2003 Carsten Griwodz & Pål Halvorsen Run-Length Coding ! Assumption " Long sequences of identical symbols ! Example INF 5070 – media servers and distribution systems 2003 Carsten Griwodz & Pål Halvorsen Bit-Plane Coding ! Assumption " Even longer sequences of identical bits ! Example 10,0,6,0,0,3,0,2,2,0,0,2,0,0,1,0, … ,0,0 (absolute) 0,x,1,x,x,1,x,0,0,x,x,1,x,x,0,x, … ,x,x (sign bits) 1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, … ,0,0 (MSB) 0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0, … ,0,0 (MSB-1) 1,0,1,0,0,1,0,1,1,0,0,1,0,0,0,0, … ,0,0 (MSB-2) 0,0,0,0,0,1,0,0,0,0,0,0,0,0,1,0, … ,0,0 (MSB-3) (0,1) (2,1) (0,0)(1,0)(2,0)(1,0)(0,0)(2,1) (5,0)(8,1) " Up to 20% savings over run-length coding can be achieved INF 5070 – media servers and distribution systems 2003 Carsten Griwodz & Pål Halvorsen Huffman Coding ! Assumption " Some symbols occur more often than others " E.g., character frequencies of the English language ! Fundamental principle " Frequently occurring symbols are coded with shorter bit strings INF 5070 – media servers and distribution systems 2003 Carsten Griwodz & Pål Halvorsen Huffman Coding ! Example " Characters to be encoded: # A, B, C, D, E " Probability to occur: # p(A)=0.3, p(B)=0.3, p(C)=0.1, p(D)=0.15, p(E)=0.15 INF 5070 – media servers and distribution systems 2003 Carsten Griwodz & Pål Halvorsen Huffman ! Table and example of application to data stream INF 5070 – media servers and distribution systems 2003 Carsten Griwodz & Pål Halvorsen JPEG ! “JPEG”: Joint Photographic Expert Group ! International Standard: " For digital compression and coding of continuous-tone still images: # Gray-scale # Color " Since 1992 ! Joint effort of: " ISO/IEC JTC1/SC2/WG10 " Commission Q.16 of CCITT SGVIII ! Compression rate of 1:10 yields reasonable results INF 5070 – media servers and distribution systems 2003 Carsten Griwodz & Pål Halvorsen JPEG ! Very general compression scheme ! Independence of " Image resolution " Image and pixel aspect ratio " Color representation " Image complexity and statistical characteristics ! Well-defined interchange format of encoded data ! Implementation in " Software only " Software and hardware ! “Motion JPEG” for video compression " Sequence of JPEG-encoded images INF 5070 – media servers and distribution systems 2003 Carsten Griwodz & Pål Halvorsen JPEG ! Sequence of compression steps " Different resolutions possible " Lossy or lossless mode # lossless compression factor ~1,6:1 " Symmetrical codec INF 5070 – media servers and distribution systems 2003 Carsten Griwodz & Pål Halvorsen JPEG – Baseline Mode: Quantization ! Use of quantization tables for the DCT-coefficients " Map interval of real numbers to one integer number " Allows to use different granularity for each coefficient INF 5070 – media servers and distribution systems 2003 Carsten Griwodz & Pål Halvorsen JPEG – 4 Modes of Compression INF 5070 – media servers and distribution systems 2003 Carsten Griwodz & Pål Halvorsen Motion JPEG ! Use series of JPEG frames to encode video ! Pro " Lossless mode – editing advantage " Frame-accurate seeking – editing advantage " Arbitrary frame rates – playback advantage " Arbitrary frame skipping – playback advantage " Scaling through progressive mode – distribution advantage " Min transmission delay = 1/framerate – conferencing advantage " Supported by popular frame grabbers ! Contra " Series of JPEG-compressed images " No standard, no specification # Worse, several competing quasi-standards " No relation to audio " No inter-frame compression INF 5070 – media servers and distribution systems 2003 Carsten Griwodz & Pål Halvorsen H.261 (px64) ! International Standard " Video codec for video conferences at p x 64kbit/s (ISDN): # Real-time encoding/decoding, max. signal delay of 150ms # Constant data rate ! Intraframe coding " DCT as in JPEG baseline mode ! Interframe coding, motion estimation " Search of similar macroblock in previous image and compare # Position of this macroblock defines motion vector # Difference between similar macroblocks INF 5070 – media servers and distribution systems 2003 Carsten Griwodz & Pål Halvorsen MPEG (Moving Pictures Expert Group) ! International Standard: " Compression of audio and video for playback (1.5 Mbit/s): " Real-time decoding ! Sequence of I-, P-, and B-Frames: ! Random access " at I-frames " at P-frames: i.e. decode previous I-frame first " at B-frame: i.e. decode I and P-frames first INF 5070 – media servers and distribution systems 2003 Carsten Griwodz & Pål Halvorsen MPEG-2 ! From MPEG-1 to MPEG-2 " Improvement in quality # From VCR to TV to HDTV " No CD-ROM based constraints # Higher data rates o MPEG-1: about 1.5 MBit/s o MPEG-2: 2-100 MBit/s ! Evolution " 1994: International Standard " Also later known as H.262 " Prominent role for digital TV in DVB (digital video broadcasting) and DVD (digital video disk) # Commercial MPEG-2 realizations available INF 5070 – media servers and distribution systems 2003 Carsten Griwodz & Pål Halvorsen MPEG-2 ! Beyond MPEG-1: " Higher quality encoding " Higher data rates " Interleaved modes ! Use cases " Broadcast quality production # DVB-T: Terrestrial # DVB-S: Satellite # DVB-C: Cable " Program Stream # for post-processing, storage, and DVD distribution " Transport Stream # for broadcasting, error resilience ! Scaling: " Signal to Noise Ration (SNR) scaling - progressive compression error correcting codes " Spatial scaling