MPEG-2 in a Stream Programming Language
Total Page:16
File Type:pdf, Size:1020Kb
MPEG-2 in a Stream Programming Language Matthew Drake, Hank Hoffmann, Rodric Rabbah, and Saman Amarasinghe MIT Computer Science and Artificial Intelligence Laboratory Abstract tations are typically not portable as they are tuned to spe- cific architectures. And while image and video compression Image and video codecs are prevalent in multimedia appli- is needed on embedded systems, desktop PCs, and high end cations, ranging from embedded systems, to desktop comput- servers, writing a separate implementation for every archi- ers, to high-end servers such as HDTV editing consoles. It is tecture is not cost effective. Furthermore, compression stan- not uncommon however that developers create (from scratch) dards are also continuously evolving, and thus compression and customize their codec implementations for each of the ar- programs must be easy to modify and update. chitecture targets they intend their coders and decoders to run A typical compression algorithm involves three types of on. This practice is time consuming and error prone, leading operations: data representation, lossy compression, and loss- to code that is not malleable or portable. In this paper we less compression. These operations are semi-autonomous, describe an implementation of the MPEG-2 codec using the exhibit data and pipeline parallelism, and easily fit into a se- StreamIt programming language. StreamIt is an architecture- quence of distinct processing kernels. As such, image and independent stream language that aims to improve program- video compression is a good match for the streaming model mer productivity, while concomitantly exposing the inherent of computation, which affords certain advantages in terms parallelism and communication topology of the application. of programmability, robustness, and achieving high perfor- We describe why MPEG is a good match for the streaming mance. Our goal is to implement well-known still image programming model, and illustrate the malleability of the im- and motion picture compression standards—such as JPEG plementation using a simple modification to the decoder to and MPEG-2—in StreamIt [36], a high-level architecture- support alternate color compression formats. StreamIt allows independent language for the streaming domain. This will for modular application development, which also reduces the result in clean, malleable, and portable codes. In addition, complexity of the debugging process since stream components using the stream-aware StreamIt compiler, we can produce can be verified independently. This in turn leads to greater highly optimized codes that are competitive with hand-tuned programmer productivity. We implement a fully functional implementations. Our architecture targets include conven- MPEG-2 decoder in StreamIt. The decoder was developed tional processors, as well as new and emerging wire exposed in eight weeks by a single student programmer who did not and multi-core architectures [12, 24, 34, 35, 5]. have any prior experience with MPEG or other video codecs. This work is in the context of the StreamIt programming Many of the MPEG-2 components were subsequently reused language [36], an architecture-independent stream language to assemble a JPEG codec. that aims to improve programmer productivity within the streaming domain. StreamIt provides an intuitive program- ming model, allowing the programmer to build an applica- 1. Introduction tion by connecting components together into a stream graph, where the nodes represent actors that carry out the compu- Image compression, whether for still pictures or motion tation, and edges represent FIFO communication channels pictures (e.g., video), plays an important role in Internet and between actors. As a result, the parallelism and commu- multimedia applications, digital appliances such as HDTV, nication topology of the application are exposed, empower- and handheld devices such as digital cameras and mobile ing the compiler to perform many stream-aware optimiza- phones. Compression allows one to represent images and tions [1, 11, 19, 30] that elude other languages. video with a much smaller amount of data and negligible quality loss. The reduction in data decreases storage require- 2. MPEG-2 Video Coding and Decoding ments (important for embedded devices) and provides higher effective transmission rates (important for Internet enabled MPEG-2 [14] is a popular coding and decoding standard devices). for digital video data. The scheme is a subset of both the Unfortunately, implementing a compression scheme can DVD-Video [33] standard for storing movies, and the Digi- be especially difficult. For performance reasons, implemen- tal Video Broadcasting specifications for transmitting HDTV and SDTV [10]. The scheme is used by a wide variety of the picture into parts with varying visual importance. The in- multimedia applications and appliances such as the Tivo Dig- put to the DCT is one block. The output of the DCT is an ital Video Recorder [38], and the DirecTV satellite broadcast 8x8 matrix of frequency coefficients. The upper left corner service [6]. of the matrix represents low frequencies, whereas the lower MPEG-2 encoding uses both lossy compression and loss- right corner represents higher frequencies. The latter are of- less compression. Lossy compression permanently elimi- ten small and can be neglected without sacrificing human vi- nates information from a video based on a human percep- sual perception. tion model. Humans are much better at discerning changes in The DCT coefficients are quantized to reduce the num- color intensity (luminance information) than changes in color ber of bits needed to represent them. Following quantization, (chrominance information). Humans are also much more sen- many coefficients are effectively reduced to zero. The DCT sitive to low frequency image components, such as a blue sky, matrix is then run-length encoded by emitting each non-zero than to high frequency image components, such as a plaid coefficient, followed by the number of zeros that precede it, shirt. Details which humans are likely to miss can be thrown along with the number of bits needed to represent the coeffi- away without affecting the perceived video quality. cient, and its value. The run-length encoder scans the DCT Lossless compression eliminates redundant information matrix in a zig-zag order (Figure 2) to consolidate the zeros while allowing for its later reconstruction. Similarities be- in the matrix. tween adjacent video pictures are encoded using motion pre- Finally, the output of the run-length encoder, motion vec- diction, and all data is Huffman compressed[13]. The amount tor data, and other information (e.g., type of picture), are of lossy and lossless compression depends on the video data. Huffman coded to further reduce the average number of bits Common compression ratios range from 10:1 to 100:1. For per data item. The compressed stream is sent to the output example, HDTV, with a resolution of 1280x720 pixels and a device. streaming rate of 59.94 frames per second, has an uncom- pressed data rate of 1.33 Gigabits per second. It is com- 2.2. MPEG Decoding pressed at an average rate of 66:1, reducing the required streaming rate to 20 Megabits per second [40]. An MPEG-2 input stream is organized as a Group of Pic- 2.1. MPEG Coding tures (GOP) which contains all the information needed to re- construct a video. The GOP contains the three kinds of pic- The encoder operates on a sequence of pictures. Each pic- tures produced by the encoder, namely I, P, and B pictures. ture is made up of pixels arranged in a 16x16 array known as I pictures are intended to assist scene cuts, random access, a macroblock. Macroblocks consist of a 2x2 array of blocks fast forward, or fast reverse playback [14, p. 14]. A typi- (each of which contains an 8x8 array of pixels). There is a cal I:P:B picture ratio in a GOP is 1:3:8, and a typical pic- separate series of macroblocks for each color channel, and the ture pattern is a repetition of the following logical sequence: macroblocks for a given channel are sometimes downsampled I1 B2 B3 P4 B5 B6 P7 B8 B9 P10 B11 B12 where the subscripts to a 2x1 or 1x1 block matrix. The compression in MPEG denote positions in the original video. However, to simplify is achieved largely via motion estimation, which detects and the decoder, the encoder reorders the pictures to produce the eliminates similarities between macroblocks across pictures. following pattern: I1 P4 B2 B3 P7 B5 B6 P10 B8 B9 B11 B12. Specifically, the motion estimator calculates a motion vector Under this configuration, if the decoder encounters a P pic- that represents the horizontal and vertical displacement of a ture, its motion prediction is with respect to the previously given macroblock (i.e., the one being encoded) from a match- decoded I or P picture; if the decoder encounters a B picture, ing macroblock-sized area in a reference picture. The match- its motion prediction is with respect to the previously two de- ing macroblock is removed (subtracted) from the current pic- coded I or P pictures. ture on a pixel by pixel basis, and a motion vector is associ- As with the encoding process, pictures are divided up ated with the macroblock describing its displacement relative into 16x16 pixel macroblocks, themselves composed of 8x8 to the reference picture. The result is a residual predictive- blocks. Macroblocks specify colors using a luminance chan- code (P) picture. It represents the difference between the cur- nel to represent saturation (color intensity), and two chromi- rent picture and the reference picture. Reference pictures en- nance channels to represent hue. MPEG-2 streams specify coded without the use of motion prediction are intra-coded (I) a chroma format which allows the chrominance data to be pictures. In addition to forward motion prediction, it is possi- sampled at a lower rate. The most common chroma format is ble to encode new pictures using motion estimation from both 4:2:0 which represents a macroblock using four blocks for the previous and subsequent pictures.