Compression and File Formats Compressing Moving Images

Compression and File Formats

Compressing Moving Images

Methods: • Motion JPEG, Cinepak, Indeo, MPEG • Known as CODECs – compression / decompression algorithms – hardware and software implementations – symmetrical / asymmetrical

Moving versus still images: • Temporal as well as spatial compression • Data transfer rates are critical – How much data is needed to maintain 24 frames/sec?

Chapman & Chapman, Chapter 10

1 Spatial Compression

• Compress each frame (still image) independently - also known as ‘intra-frame’ compression • Animated GIFs – single file containing multiple images compressed using LZW compression – only suitable for simple, cartoon-like animations • Motion JPEG (MJPEG) – compresses each frame using JPEG – suitable for videos of real-world action – implemented in hardware for real-time capture and compression of video – no standard file format! (MJPEG-A emerging)

Temporal Compression

Also, known as ‘inter-frame’ compression - relies on only small changes between frames:

2 Temporal Compression

Also Basic idea is this: • Certain frames designated as key frames – e.g. every sixth frame (may be user-settable) • Key frames either are left uncompressed or only spatially compressed • Each frame between key frames is replaced by a difference frame – records only differences between frame and preceding frame or preceding key frame • Difference frames readily compressed with RLE or dictionary-based schemes

Software CODECs for Multimedia

• Multimedia is likely delivered on CD-ROM or over the Internet • Low to very low data transfer rates • High compression required • Software implementations of a variety of codecs – Cinepak – Intel Indeo – MPEG-1

3 Cinepak and Indeo • Similar codec methods that use vector quantization • Each frame divided into small rectangular blocks of pixels (the vectors) • Use a code book consisting of typical vectors (representing typical patterns that might occur in a image, e.g.) – areas of flat colour – sharp or soft edges – different textures – (the dictionary of still image compression) • Vector Quantization is the process of matching each image vector with its closest vector in the code book • Each image vector is replaced by its index in the code book • Image can be similarly re-constructed from these indexes and the codebook - de-compression efficient 7

Cinepak and Indeo (2)

• Highly asymmetric: compression takes much longer than decompression – 150:1 for Cinepak • Also use temporal compression based on key and difference frames • full-motion full screen playback cannot be achieved with these codecs • Good enough for 320x240 pixels at 12 frames/sec – not great viewing! • New Sorenson codec does a bit better (can compress video to give a frame-rate of 50 kbytes/s)

4 MPEG-1

• Several international standards for compressed digital video are grouped together under the name, MPEG • Motion Picture Experts Group • Defines a data stream syntax and a decompressor • Compression is left to the software developer – “competitive advantage in the marketplace” • MPEG-1 Standard does define compression implicitly • So we can describe MPEG-1 compression which combines: • Spatial compression similar to JPEG, and • Temporal compression via difference frames and motion compensation.

Temporal Compression in MPEG-1 • Naïve approach subtracts each pixel in current frame from corresponding pixel in previous frame – result is 0 for pixels that do not change between frames – difference frames will contain large numbers of zeros and so will compress well, using e.g. RLE • BUT can possibly do better when there is large movement between frames – pictures usually composed of whole objects (e.g. a walking person) that move between frames • Difference frame would be better if we could match pixels from the moving object between frames • Motion compensation (if we can identify coherent area of change corresponding to object, then only need to record object’s movement together with changed pixels in the area) 10

5 Motion Compensation • Motion compensation in MPEG does not attempt to identify objects in a scene – hard computer vision / artificial intelligence problem!! • Divides each frame into 16 by 16 pixel blocks, known as macroblocks – different from the smaller blocks used in spatial compression • Attempts to predict where the corresponding macroblock is in the next frame – simple searching in a limited area around the macroblock (all possible displacements within a limited range are tried and best best match chosen) • Difference frame is then constructed by subtracting matching macroblocks (i.e. each macroblock from its predicted counterpart in next frame) • BUT also need to store displacement vectors (describing predicted displacement of macroblocks between frames)! 11

Frame Types in MPEG • Temporal compression has to start somewhere (e.g. Keyframe) • Key frames called I-pictures (I for intra) • Difference frames based on previous frames called P- pictures (predictive pictures) • MPEG also allows difference frames (motion compensation) based on later frames: B-pictures (bi-directional prediction) • A video clip can be encoded in compressed form as a sequence of I-, P- and B- pictures • MPEG Display order: IBBPBBIBBPBBI • Encoders normally use a repeating sequence, called Group of Pictures (GOP): IBBPBB (always starts with Key frame) • Display order is converted into a Bitstream order (for transmission): e.g. IPBBIBBPBBIBB – Decoder needs to reconstruct display order from bitstream 12

6 MPEG Variations

MPEG-1: Delivers 1.2Mbps of video and 250Kbps of 2- channel audio from CD-ROM – best for multimedia applications – However, MPEG-1 cannot handle interlacing or HDTV formats, hence need for MPEG-2 for broadcasting & studio work MPEG-2: Needs higher data rates of 3-15Mbps, but delivers better picture quality, interlaced video formats and multichannel audio – designed for broadcast digital video (e.g. DVD) MPEG-4: emerging standard offering data rates from 5Kbps to 4Mbytesps, with interactive multimedia as well as video

New and Future CODECs

• The challenge of sending video over the Internet has led to much research into new compression methods – very low bitrates required (e.g. 28kbps over a phoneline) • More sophisticated motion estimation • Wavelets, fractals etc • H.261 and H.263 – designed for video conferencing – use multiple compression techniques – “better than nothing” performance over a phoneline

7 File Formats for Video

• Like file formats for still images, file formats for video provide a packaging that includes details of the contents of the file, such as the codec used • Various formats, such as... • Microsoft’s AVI • QuickTime • MPEG – not really a file format, just defines a bit (data) stream (like JPEG)

Video for Windows

• AVI: Audio and Video Interleaved • Video and synchronized audio in a small window at around 15 frames-per-second without specialized hardware • Video is interleaved with the audio • Variety of codecs (identified by 4-byte code, FOURCC): – Cinepak (CVID) – Intel Indeo – Microsoft Video – Microsoft RLE (MRLE) – others

8 QuickTime

• De facto standard (as of version 3.0 in 1998) • QuickTime (QT) movie (“.mov”) is a framework for organizing and manipulating the data that make up the actual video frames – many video codecs supported – can contain media other than video – media not necessarily stored in the .mov file • A movie plays according to a time base (which records rate at which it should be played back & its current position) – frames may be dropped to keep in time – enables constant playback speed on any system – maintains synchronization between video and audio

QuickTime

• Success of QT is due to its Component-based architecture • Plug in new components (small programs conforming to certain interface conventions) to deal with new formats or provide new operations • Components include: – Compressors (for implementing e.g. Cinepak, Indeo) – Sequence grabbers: digitize video – Movie player component is used to provide a user-interface for playing movies: will handle AVI and MPEG – Transcoders: translate between data formats that use the same compression algorithm

9 Streamed Video

• Playback as it arrives over the Internet • Current method of delivery on WWW: Progressive download or HTTP streaming – download movie to hard disk – start playing back when playback time of downloaded content equals download time of remainder of movie – still a large delay before playback begins and not suitable for “live” video • True streaming – never stored on user’s disk – must arrive at playback rate (not possible on dial-up internet connections using V90 modems or basic rate ISDN) – different movie formats depending on connection speed – Streaming QuickTime and RealVideo 19

File Formats for Animation

• Macromedia’s Shockwave Flash – Native file format for Flash movies (and Director) – Vector images and animations – Metafile: can incorporate bitmap images and sound – De facto standard for simple animation on the Web – Browser plugins freely available • VRML – Virtual Reality Modelling Language – 3D interactive worlds – Browser plugins freely available

10 Shockwave Flash (SWF)

• File contains two broad classes of items… • Definitions: dictionary of symbols that appear in the animation • Control items: instructions to place, remove or move a symbol – movement specified using transformation matrices • Rather like a program (cf. FIG, POVRay, VRML) – stored in compressed binary format: SMALL! • Flash creates simple animations • Director creates more complicated movies

VRML • Code directly, or output from 3D modelling package • Interactive 3D worlds:

11 VRML Example

#VRML V1.0 ascii Separator { DirectionalLight { direction 0 0 -1 } PerspectiveCamera { position -8.6 2.1 5.6 orientation -0.1352 -0.9831 -0.1233 1.1417 focalDistance 10.84 } Separator { # The red sphere Material { diffuseColor 1 0 0 # Red Separator { # The blue cube } Material { Translation { translation 3 0 1 } diffuseColor 0 0 1 # Blue Sphere { radius 2.3 } } } Transform { translation -2.4 .2 1 rotation 0 1 1 .9 } Cube {} } 23

End of Lecture

Exam revision/hints for my part of the course during tutorials next week!

Good luck with all your exams!