Compressing Moving Images" Moving versus still images: – Temporal as well as spatial compression Compression and File Formats! – Data transfer rates are critical updated to include HTML5 video tag." • How much data is needed to maintain 24 frames/sec? Methods: – Motion JPEG, Cinepak, Indeo, MPEG – Known as CODECs • compression / decompression algorithms • hardware and software implementations • symmetrical / asymmetrical Autumn 2013" ITNP80: Multimedia" 2" Why and where to compress" The DV standard" Consider video as displayed on a television screen Digital video (DV) standard includes compression on camera – Raw data rate is huge: • 768 (across) by 576 (vertical) by 25 (frames/sec) CCIR601 standard specifies signal sampling – 31Mbytes/sec or 1.85Gbytes/minute – Subsamples chrominance – Eyes are less sensitive to variations in colour than to variations Can we compress in hardware and/or in software to provide in brightness reasonable data rates and storage requirements? – Uses luminance, Y (=0.2125R + 0.7154G + 0.0721B), B-Y, and R-Y – Luminance is sent each pixel, but others are subsampled e.g. colour (B-Y & R-Y) sent every 4th pixel Data rate is 166Mbits (just over 20Mbytes)/second Autumn 2013" ITNP80: Multimedia" 3" Autumn 2013" ITNP80: Multimedia" 4" The DV standard" Spatial Compression" 166Mbits/second is inconveniently high Compress each frame (still image) independently DV standard uses further compression to bring the data rate Animated GIFs down to 25Mbits/second (just over 3Mbytes/sec) – single file containing multiple images compressed using LZW – Compression is on-camera compression – Widely used – only suitable for simple, cartoon-like animations Still too high a data rate for storage Motion JPEG (MJPEG) – 25Mbits/sec = 210Mbytes/min = about 12Gbytes per hour – compresses each frame using JPEG – Too high for transmission, or for storage on a DVD – suitable for videos of real-world action – implemented in hardware for real-time capture and compression of video – no standard file format! (MJPEG-A emerging) Autumn 2013" ITNP80: Multimedia" 5" Autumn 2013" ITNP80: Multimedia" 6" Temporal Compression" Difference Frames" Relies on only small changes between frames: Naïve approach subtracts each pixel in current frame from corresponding pixel in previous frame – result is 0 for pixels that do not change between frames – difference frames will contain large numbers of zeros and so will compress well, using e.g. run-length encoding (RLE). -! =! Autumn 2013" ITNP80: Multimedia" 7" Autumn 2013" ITNP80: Multimedia" 8" Motion Compensation" Temporal Compression" Can possibly do better when there is movement between frames – whole objects (e.g. a walking person) move between frames Basic idea is this: – or camera moves – Certain frames designated as key frames Difference frame would be better if we could match pixels • e.g. every sixth frame (may be user-settable) from the moving object between frames – Key frames either are left uncompressed or only spatially compressed – Each frame between key frames is replaced by a difference frame • records only differences between frame and preceding frame or preceding key frame ! ! – Difference frames readily compressed with RLE or dictionary- - = Displacement based schemes ! vector ! Autumn 2013" ITNP80: Multimedia" 9" Autumn 2013" ITNP80: Multimedia" 10" Software CODECs for Multimedia" Cinepak and Indeo" Multimedia is often delivered on CD/DVD or over the Internet Similar methods that use vector quantization. Each frame – low to very low data transfer rates divided into small rectangular blocks of pixels (the vectors) Use a code book consisting of typical vectors High compression required for video – areas of flat colour Software implementations of a variety of codecs – sharp or soft edges – Cinepak – different textures – Intel Indeo – (the dictionary of still image compression) – MPEG-1, MPEG-2, MPEG-4. Processing power needed for compression and (real-time) Quantization is the process of matching each image vector with decompression its closest vector in the code book. Each image vector is – limits frame size and rate replaced by its index in the code book. Note there may not be an exact match resulting in loss of quality. Autumn 2013" ITNP80: Multimedia" 11 " Autumn 2013" ITNP80: Multimedia" 12" Cinepak and Indeo (2)" MPEG-1" Highly asymmetric: compression takes much longer than Motion Picture Experts Group decompression -> 150:1 for Cinepak – Audiovisual compression why does this not matter? – See http://www.mpeg.org Also use temporal compression based on key and difference Defines a data stream and a decompressor frames – Compression is left to the software developer – Automatic keyframe selection • “competitive advantage in the marketplace” – Good enough for 320x240 pixels at 12 frames/sec from CD- – Standard does define compression implicitly ROM on a basic PC – not great viewing... – Spatial compression similar to JPEG – Temporal compression via difference frames and motion compensation. Autumn 2013" ITNP80: Multimedia" 13" Autumn 2013" ITNP80: Multimedia" 14" Motion Compensation" Frame Types in MPEG" Motion compensation in MPEG does not attempt to identify Key frames called I-pictures (I for intra) objects in a scene – Difference frames based on previous frames called P-pictures (predictive pictures) – A hard computer vision / artificial intelligence problem – MPEG also allows difference frames based on later frames: B- pictures (bi-directional prediction) Divides each frame into 16 by 16 pixel blocks, known as macroblocks Display order: – Different from the smaller blocks used in spatial compression – Attempts to predict where the corresponding macroblock is in IBBPBBIBBPBBI the next frame via simple searching in a limited area around the Bitstream order: IPBBIBBPBBIBB macroblock – decoder needs to reconstruct display order from bitstream – Difference frame constructed by subtracting matching macroblocks Group of Pictures (GOP): IBBPBBPBBPBBIBBPBBPB – BUT also need to store displacement vectors Autumn 2013" ITNP80: Multimedia" 15" Autumn 2013" ITNP80: Multimedia" 16" MPEG Variations" MPEG-4" MPEG-1: Delivers around 2Mbps of video and 250Kbps of 2- Newer standard offering multiple data rates channel audio from CD-ROM – H.264 – 352x240 pixels at 30 fps on standard PC – Based on Profiles – 40 minutes of video on CD-ROM (9 hours on DVD) • 5Kbps to 4Mbyte/s, with interactive multimedia as well as video – best for multimedia applications • Low rate profiles for video-phones. • Simple profile, Advanced simple profile MPEG-2: Needs higher data rates of 3-15Mbps, but delivers – Allow different compression techniques for different better picture quality, interlaced video formats and rectangular areas multichannel audio – Patent-encumbered – designed for broadcast digital video (e.g. DVD) Other profiles can be object-based. Autumn 2013" ITNP80: Multimedia" 17" Autumn 2013" ITNP80: Multimedia" 18" New and Future CODECs" File Formats for Video" The challenge of sending video over the Internet has led to Like file formats for still images, they provide a packaging that research into new compression methods includes details of the contents of the file, such as the codec – very low bitrates required (e.g. 28kbps over a phoneline) used. Various formats, such as... – More sophisticated motion estimation – Wavelets, fractals etc – Microsofts AVI & WMV – H.261 and H.263 • designed for video conferencing – QuickTime • use multiple compression techniques – MPEG • better than nothing performance over a phoneline – VC-1 You need appropriate player and CODECs on your computer to • Windows Media Video 9 • HD and Blu-Ray play files. – Windows Media Player, QuickTime, HTML5 <video> Autumn 2013" ITNP80: Multimedia" 19" Autumn 2013" ITNP80: Multimedia" 20" Video for Windows QuickTime AVI: Audio and Video Interleaved QuickTime movie (.mov) is a framework for organizing and – Video and synchronized audio in a small window at around 15 manipulating the data that make up the actual video frames frames-per-second without specialized hardware – many video codecs supported – Video is interleaved with the audio – can contain media other than video – Variety of codecs (identified by 4-byte code, FOURCC): – media not necessarily stored in the .mov file • Cinepak (CVID) • Intel Indeo A movie plays according to a time base • Microsoft Video – frames may be dropped to keep in time • Microsoft RLE (MRLE) – enables constant playback speed on any system • others – maintains synchronization between video and audio Autumn 2013 ITNP80: Multimedia 21 Autumn 2013 ITNP80: Multimedia 22 QuickTime Streamed Video Component-based architecture Playback as it arrives over the Internet – Plug in new components to deal with new formats or provide new Progressive download operations – download movie to hard disk – Components include: – start playing back when playback time of downloaded content • Compressors (e.g. Cinepak, Indeo, Sorenson) equals download time of remainder of movie • Sequence grabbers: digitize video – still a large delay before playback begins and not suitable for • Movie player: will handle AVI and MPEG live video • Transcoders: translate between data formats True streaming – never stored on users disk – must arrive at playback rate – different movie formats depending on connection speed – Streaming QuickTime and RealVideo Autumn 2013 ITNP80: Multimedia 23 Autumn 2013 ITNP80: Multimedia 24 File Formats for Animation Shockwave Flash (SWF) Adobes Shockwave Flash File contains two broad classes of items… – Native