Compressing Moving Images
Moving versus still images: – Temporal as well as spatial compression Compression and File Formats – Data transfer rates are critical updated to include HTML5 video tag. • How much data is needed to maintain 24 frames/sec?
Methods: – Motion JPEG, Cinepak, Indeo, MPEG – Known as CODECs • compression / decompression algorithms • hardware and software implementations • symmetrical / asymmetrical
Autumn 2013 ITNP80: Multimedia 2
Why and where to compress The DV standard
Consider video as displayed on a television screen Digital video (DV) standard includes compression on camera – Raw data rate is huge: • 768 (across) by 576 (vertical) by 25 (frames/sec) CCIR601 standard specifies signal sampling – 31Mbytes/sec or 1.85Gbytes/minute – Subsamples chrominance – Eyes are less sensitive to variations in colour than to variations Can we compress in hardware and/or in software to provide in brightness reasonable data rates and storage requirements? – Uses luminance, Y (=0.2125R + 0.7154G + 0.0721B), B-Y, and R-Y – Luminance is sent each pixel, but others are subsampled e.g. colour (B-Y & R-Y) sent every 4th pixel
Data rate is 166Mbits (just over 20Mbytes)/second
Autumn 2013 ITNP80: Multimedia 3 Autumn 2013 ITNP80: Multimedia 4 The DV standard Spatial Compression
166Mbits/second is inconveniently high Compress each frame (still image) independently
DV standard uses further compression to bring the data rate Animated GIFs down to 25Mbits/second (just over 3Mbytes/sec) – single file containing multiple images compressed using LZW – Compression is on-camera compression – Widely used – only suitable for simple, cartoon-like animations Still too high a data rate for storage Motion JPEG (MJPEG) – 25Mbits/sec = 210Mbytes/min = about 12Gbytes per hour – compresses each frame using JPEG – Too high for transmission, or for storage on a DVD – suitable for videos of real-world action – implemented in hardware for real-time capture and compression of video – no standard file format! (MJPEG-A emerging)
Autumn 2013 ITNP80: Multimedia 5 Autumn 2013 ITNP80: Multimedia 6
Temporal Compression Difference Frames
Relies on only small changes between frames: Naïve approach subtracts each pixel in current frame from corresponding pixel in previous frame – result is 0 for pixels that do not change between frames – difference frames will contain large numbers of zeros and so will compress well, using e.g. run-length encoding (RLE).
-! =!
Autumn 2013 ITNP80: Multimedia 7 Autumn 2013 ITNP80: Multimedia 8 Motion Compensation Temporal Compression
Can possibly do better when there is movement between frames – whole objects (e.g. a walking person) move between frames Basic idea is this: – or camera moves – Certain frames designated as key frames Difference frame would be better if we could match pixels • e.g. every sixth frame (may be user-settable) from the moving object between frames – Key frames either are left uncompressed or only spatially compressed – Each frame between key frames is replaced by a difference frame • records only differences between frame and preceding frame or preceding key frame ! ! – Difference frames readily compressed with RLE or dictionary- - = Displacement based schemes vector ! !
Autumn 2013 ITNP80: Multimedia 9 Autumn 2013 ITNP80: Multimedia 10
Software CODECs for Multimedia Cinepak and Indeo
Multimedia is often delivered on CD/DVD or over the Internet Similar methods that use vector quantization. Each frame – low to very low data transfer rates divided into small rectangular blocks of pixels (the vectors) Use a code book consisting of typical vectors High compression required for video – areas of flat colour Software implementations of a variety of codecs – sharp or soft edges – Cinepak – different textures – Intel Indeo – (the dictionary of still image compression) – MPEG-1, MPEG-2, MPEG-4. Processing power needed for compression and (real-time) Quantization is the process of matching each image vector with decompression its closest vector in the code book. Each image vector is – limits frame size and rate replaced by its index in the code book. Note there may not be an exact match resulting in loss of quality.
Autumn 2013 ITNP80: Multimedia 11 Autumn 2013 ITNP80: Multimedia 12 Cinepak and Indeo (2) MPEG-1
Highly asymmetric: compression takes much longer than Motion Picture Experts Group decompression -> 150:1 for Cinepak – Audiovisual compression why does this not matter? – See http://www.mpeg.org
Also use temporal compression based on key and difference Defines a data stream and a decompressor frames – Compression is left to the software developer – Automatic keyframe selection • “competitive advantage in the marketplace” – Good enough for 320x240 pixels at 12 frames/sec from CD- – Standard does define compression implicitly ROM on a basic PC – not great viewing... – Spatial compression similar to JPEG – Temporal compression via difference frames and motion compensation.
Autumn 2013 ITNP80: Multimedia 13 Autumn 2013 ITNP80: Multimedia 14
Motion Compensation Frame Types in MPEG
Motion compensation in MPEG does not attempt to identify Key frames called I-pictures (I for intra) objects in a scene – Difference frames based on previous frames called P-pictures (predictive pictures) – A hard computer vision / artificial intelligence problem – MPEG also allows difference frames based on later frames: B- pictures (bi-directional prediction) Divides each frame into 16 by 16 pixel blocks, known as macroblocks Display order: – Different from the smaller blocks used in spatial compression
– Attempts to predict where the corresponding macroblock is in IBBPBBIBBPBBI the next frame via simple searching in a limited area around the Bitstream order: IPBBIBBPBBIBB macroblock – decoder needs to reconstruct display order from bitstream – Difference frame constructed by subtracting matching macroblocks Group of Pictures (GOP): IBBPBBPBBPBBIBBPBBPB – BUT also need to store displacement vectors
Autumn 2013 ITNP80: Multimedia 15 Autumn 2013 ITNP80: Multimedia 16 MPEG Variations MPEG-4
MPEG-1: Delivers around 2Mbps of video and 250Kbps of 2- Newer standard offering multiple data rates channel audio from CD-ROM – H.264 – 352x240 pixels at 30 fps on standard PC – Based on Profiles – 40 minutes of video on CD-ROM (9 hours on DVD) • 5Kbps to 4Mbyte/s, with interactive multimedia as well as video – best for multimedia applications • Low rate profiles for video-phones. • Simple profile, Advanced simple profile MPEG-2: Needs higher data rates of 3-15Mbps, but delivers – Allow different compression techniques for different better picture quality, interlaced video formats and rectangular areas multichannel audio – Patent-encumbered – designed for broadcast digital video (e.g. DVD) Other profiles can be object-based.
Autumn 2013 ITNP80: Multimedia 17 Autumn 2013 ITNP80: Multimedia 18
New and Future CODECs File Formats for Video
The challenge of sending video over the Internet has led to Like file formats for still images, they provide a packaging that research into new compression methods includes details of the contents of the file, such as the codec – very low bitrates required (e.g. 28kbps over a phoneline) used. Various formats, such as...
– More sophisticated motion estimation – Wavelets, fractals etc – Microsoft s AVI & WMV – H.261 and H.263
• designed for video conferencing – QuickTime • use multiple compression techniques – MPEG • better than nothing performance over a phoneline
– VC-1 You need appropriate player and CODECs on your computer to • Windows Media Video 9 • HD and Blu-Ray play files.
– Windows Media Player, QuickTime, HTML5
Autumn 2013 ITNP80: Multimedia 19 Autumn 2013 ITNP80: Multimedia 20 Video for Windows QuickTime
AVI: Audio and Video Interleaved QuickTime movie ( .mov ) is a framework for organizing and – Video and synchronized audio in a small window at around 15 manipulating the data that make up the actual video frames frames-per-second without specialized hardware – many video codecs supported – Video is interleaved with the audio – can contain media other than video
– Variety of codecs (identified by 4-byte code, FOURCC): – media not necessarily stored in the .mov file • Cinepak (CVID) • Intel Indeo A movie plays according to a time base • Microsoft Video – frames may be dropped to keep in time • Microsoft RLE (MRLE) – enables constant playback speed on any system • others – maintains synchronization between video and audio
Autumn 2013 ITNP80: Multimedia 21 Autumn 2013 ITNP80: Multimedia 22
QuickTime Streamed Video
Component-based architecture Playback as it arrives over the Internet – Plug in new components to deal with new formats or provide new Progressive download operations – download movie to hard disk – Components include: – start playing back when playback time of downloaded content • Compressors (e.g. Cinepak, Indeo, Sorenson) equals download time of remainder of movie
• Sequence grabbers: digitize video – still a large delay before playback begins and not suitable for • Movie player: will handle AVI and MPEG live video • Transcoders: translate between data formats True streaming – never stored on user s disk – must arrive at playback rate – different movie formats depending on connection speed – Streaming QuickTime and RealVideo
Autumn 2013 ITNP80: Multimedia 23 Autumn 2013 ITNP80: Multimedia 24 File Formats for Animation Shockwave Flash (SWF)
Adobe s Shockwave Flash File contains two broad classes of items… – Native file format for Flash movies – Definitions: dictionary of symbols that appear in the animation – Vector images and animations – Control items: instructions to place, remove or move a symbol – Metafile: can incorporate bitmap images and sound • movement specified using transformation matrices – Was De facto standard for animation on the Web – Browser plugins freely available Rather like a program (cf. FIG, POVRay, SVG, X3D) SVG / SMIL / X3D • stored in compressed binary format: SMALL! – XML languages for use on web pages Microsoft s Silverlight – Programmable web browser plugin – .NET languages and development tools
Autumn 2013 ITNP80: Multimedia 25 Autumn 2013 ITNP80: Multimedia 26
HTML5 and video Supported in most browsers: but …
•
H264:&much&used,&good&compression,&but&patented&(so&costs&money&to& use)& Ogg&Theora:&Not&as&good&as&H264,&but¬&patented& VP8:&also&good:&new&kid&on&the&block& WebM:&towards&an&open&video&codec.& &&See&http://www.webmproject.org/about/& Autumn 2013 ITNP80: Multimedia 29