Compressing Moving Images Methods: • Motion JPEG, Cinepak, Indeo, MPEG • Known as CODECs – compression / decompression algorithms – hardware and software implementations – symmetrical / asymmetrical Compression and File Formats Moving versus still images: • Temporal as well as spatial compression • Data transfer rates are critical – How much data is needed to maintain 24 frames/sec? Chapman & Chapman, Chapter 10 1 2 Spatial Compression Temporal Compression Also, known as ‘inter-frame’ compression - relies on • Compress each frame (still image) independently - also known as ‘intra-frame’ compression only small changes between frames: • Animated GIFs – single file containing multiple images compressed using LZW compression – only suitable for simple, cartoon-like animations • Motion JPEG (MJPEG) – compresses each frame using JPEG – suitable for videos of real-world action – implemented in hardware for real-time capture and compression of video – no standard file format! (MJPEG-A emerging) 3 4 Temporal Compression Software CODECs for Multimedia Also Basic idea is this: • Multimedia is likely delivered on CD-ROM or over the • Certain frames designated as key frames Internet – e.g. every sixth frame (may be user-settable) • Low to very low data transfer rates • Key frames either are left uncompressed or only • High compression required spatially compressed • Software implementations of a variety of codecs • Each frame between key frames is replaced by a – Cinepak difference frame – Intel Indeo – records only differences between frame and preceding frame – MPEG-1 or preceding key frame • Difference frames readily compressed with RLE or dictionary-based schemes 5 6 1 Cinepak and Indeo Cinepak and Indeo (2) • Similar codec methods that use vector quantization • Each frame divided into small rectangular blocks of pixels • Highly asymmetric: compression takes much longer (the vectors) than decompression • Use a code book consisting of typical vectors (representing – 150:1 for Cinepak typical patterns that might occur in a image, e.g.) • Also use temporal compression based on key and – areas of flat colour difference frames – sharp or soft edges – different textures • full-motion full screen playback cannot be achieved – (the dictionary of still image compression) with these codecs • Vector Quantization is the process of matching each image • Good enough for 320x240 pixels at 12 frames/sec vector with its closest vector in the code book – not great viewing! • Each image vector is replaced by its index in the code book • New Sorenson codec does a bit better (can • Image can be similarly re-constructed from these indexes compress video to give a frame-rate of 50 kbytes/s) and the codebook - de-compression efficient 7 8 MPEG-1 Temporal Compression in MPEG-1 • Several international standards for compressed digital video are • Naïve approach subtracts each pixel in current frame from grouped together under the name, MPEG corresponding pixel in previous frame • Motion Picture Experts Group – result is 0 for pixels that do not change between frames – difference frames will contain large numbers of zeros and so will • Defines a data stream syntax and a decompressor compress well, using e.g. RLE • Compression is left to the software developer • BUT can possibly do better when there is large movement – “competitive advantage in the marketplace” between frames • MPEG-1 Standard does define compression implicitly – pictures usually composed of whole objects (e.g. a walking person) • So we can describe MPEG-1 compression which that move between frames combines: • Difference frame would be better if we could match pixels • Spatial compression similar to JPEG, and from the moving object between frames • Temporal compression via difference frames and motion • Motion compensation (if we can identify coherent area of compensation. change corresponding to object, then only need to record object’s movement together with changed pixels in the 9 area) 10 Motion Compensation Frame Types in MPEG • Motion compensation in MPEG does not attempt to • Temporal compression has to start somewhere (e.g. Keyframe) identify objects in a scene • Key frames called I-pictures (I for intra) – hard computer vision / artificial intelligence problem!! • Difference frames based on previous frames called P- • Divides each frame into 16 by 16 pixel blocks, known as pictures (predictive pictures) macroblocks • MPEG also allows difference frames (motion compensation) – different from the smaller blocks used in spatial compression based on later frames: B-pictures (bi-directional prediction) • Attempts to predict where the corresponding macroblock • A video clip can be encoded in compressed form as a is in the next frame sequence of I-, P- and B- pictures – simple searching in a limited area around the macroblock (all possible displacements within a limited range are tried and best • MPEG Display order: IBBPBBIBBPBBI best match chosen) • Encoders normally use a repeating sequence, called Group • Difference frame is then constructed by subtracting of Pictures (GOP): IBBPBB (always starts with Key frame) matching macroblocks (i.e. each macroblock from its predicted • Display order is converted into a Bitstream order (for counterpart in next frame) transmission): e.g. IPBBIBBPBBIBB • BUT also need to store displacement vectors (describing – Decoder needs to reconstruct display order from bitstream predicted displacement of macroblocks between frames)! 11 12 2 MPEG Variations New and Future CODECs MPEG-1: Delivers 1.2Mbps of video and 250Kbps of 2- • The challenge of sending video over the Internet has channel audio from CD-ROM led to much research into new compression methods – best for multimedia applications – very low bitrates required (e.g. 28kbps over a phoneline) – However, MPEG-1 cannot handle interlacing or HDTV formats, • More sophisticated motion estimation hence need for MPEG-2 for broadcasting & studio work • Wavelets, fractals etc MPEG-2: Needs higher data rates of 3-15Mbps, but • H.261 and H.263 delivers better picture quality, interlaced video formats and multichannel audio – designed for video conferencing – use multiple compression techniques – designed for broadcast digital video (e.g. DVD) – “better than nothing” performance over a phoneline MPEG-4: emerging standard offering data rates from 5Kbps to 4Mbytesps, with interactive multimedia as well as video 13 14 File Formats for Video Video for Windows • Like file formats for still images, file formats for video • AVI: Audio and Video Interleaved provide a packaging that includes details of the • Video and synchronized audio in a small window at contents of the file, such as the codec used around 15 frames-per-second without specialized • Various formats, such as... hardware • Microsoft’s AVI • Video is interleaved with the audio • QuickTime • Variety of codecs (identified by 4-byte code, FOURCC): • MPEG – Cinepak (CVID) – not really a file format, just defines a bit (data) stream (like – Intel Indeo JPEG) – Microsoft Video – Microsoft RLE (MRLE) – others 15 16 QuickTime QuickTime • De facto standard (as of version 3.0 in 1998) • Success of QT is due to its Component-based • QuickTime (QT) movie (“.mov”) is a framework for architecture organizing and manipulating the data that make up • Plug in new components (small programs conforming to the actual video frames certain interface conventions) to deal with new formats or – many video codecs supported provide new operations – can contain media other than video • Components include: – media not necessarily stored in the .mov file – Compressors (for implementing e.g. Cinepak, Indeo) • A movie plays according to a time base (which records – Sequence grabbers: digitize video rate at which it should be played back & its current position) – Movie player component is used to provide a user-interface – frames may be dropped to keep in time for playing movies: will handle AVI and MPEG – enables constant playback speed on any system – Transcoders: translate between data formats that use the – maintains synchronization between video and audio same compression algorithm 17 18 3 Streamed Video File Formats for Animation • Playback as it arrives over the Internet • Macromedia’s Shockwave Flash • Current method of delivery on WWW: – Native file format for Flash movies (and Director) Progressive download or HTTP streaming – Vector images and animations – download movie to hard disk – Metafile: can incorporate bitmap images and sound – start playing back when playback time of downloaded – De facto standard for simple animation on the Web content equals download time of remainder of movie – Browser plugins freely available – still a large delay before playback begins and not suitable for • VRML “live” video – Virtual Reality Modelling Language • True streaming – 3D interactive worlds – never stored on user’s disk – Browser plugins freely available – must arrive at playback rate (not possible on dial-up internet connections using V90 modems or basic rate ISDN) – different movie formats depending on connection speed – Streaming QuickTime and RealVideo 19 20 Shockwave Flash (SWF) VRML • Code directly, or output from 3D modelling package • File contains two broad classes of items… • Interactive 3D worlds: • Definitions: dictionary of symbols that appear in the animation • Control items: instructions to place, remove or move a symbol – movement specified using transformation matrices • Rather like a program (cf. FIG, POVRay, VRML) – stored in compressed binary format: SMALL! • Flash creates simple animations • Director creates more complicated movies 21 22 VRML Example #VRML V1.0 ascii Separator { DirectionalLight { End of Lecture direction 0 0 -1 } PerspectiveCamera { position -8.6 2.1 5.6 orientation -0.1352 -0.9831 -0.1233 1.1417 focalDistance 10.84 Exam revision/hints for my part of the course } during tutorials next week! Separator { # The red sphere Material { diffuseColor 1 0 0 # Red Separator { # The blue cube } Material { Good luck with all your exams! Translation { translation 3 0 1 } diffuseColor 0 0 1 # Blue Sphere { radius 2.3 } } } Transform { translation -2.4 .2 1 rotation 0 1 1 .9 } Cube {} } 23 24 4.