IMPLEMENTATION OF AN MPEG-1 VIDEO ENCODER AND DECODER

By John Bateman

November 1995

Submitted as partial fulfilment of the degree of Master of Engineering Science DECLARATION

I hereby declare that this submission is my own work and that, to the best of my knowledge and belief, it contains no material previously published or written by another person nor material which to a substantial extent has been accepted for the award of any other degree or diploma of a university or other institute of higher learning, except where due acknowledgement is made in the text of the thesis.

J. A. BATEMAN November 1995 ACKNOWLEDGEMENTS

I would like to thank my supervisor, Dr John Arnold, for his assistance and guidance throughout this project. I would also like to thank Dr Michael Frater for his advice on variable length coding techniques and other issues, and Mr Mark Pickering for providing advice on a range of video coding issues. ABSTRACT

The demand for multimedia applications which include digital video (sequences of images in time) as well as still images, sound and text is increasing rapidly. Processing digital video is problematic due to the high bit rates involved. For example, a standard television picture requires a bit rate of approximately 165 Mbps. To overcome this problem an international standard known as MPEG-1 has been developed to define a coded video bitstream at compressed bit rates of around 1.5 Mbps. The aim of this project was to develop a software program to encode and decode video in accordance with this standard for use in future standards research and teaching. Due to the complexity of the task, a structured approach was followed for the software design and development process. The program was implemented using the MATLAB progranmiing environment to make use of its powerful mathematical and image processing features, and a faster development time than traditional languages such as C and Pascal. The performance of the encoder and decoder was then analysed in terms of picture quality and processing speed. Picture quality was found to be good at MPEG-1 bit rates. Encoding and decoding is a computationally intensive and slow process. Some areas of the encoder and decoder were targeted for possible future optimisation. TABLE OF CONTENTS

Page No CHAPTER 1 INTRODUCTION 1.1 MPEG-1 Origins 1 1.2 Related Standards 2 1.3 Project Aim 2 1.4 Thesis Overview 3

CHAPTER 2 MPEG-1 STANDARD 2.1 Structure of the Standard Document 4 2.2 Applications 5 2.3 Features 6

CHAPTER 3 MPEG-1 COMPRESSION ALGORITHM 3.1 Overview 7 3.2 Constrained Parameters 8 3.3 Source Video 9 3.4 Picture Types 10 3.5 Motion Compensation 11 3.6 Discrete Cosine Transform (DCT) 12 3.7 Quantisation 13 3.8 Variable Length Coding (VLC) 14 3.9 Layered Syntax 15

CHAPTER 4 SOFTWARE DEVELOPMENT METHODOLOGY 4.1 Classical Life Cycle 17 4.2 Structured Analysis and Design 18 4.3 Structured Analysis 18 4.4 Acceptance Test Generation 21 4.5 Structured Design 22 4.6 Coding and Testing 24 4.7 Program Documentation 24 4.8 Quality Assurance 24 4.9 Results 25 Page No CHAPTERS MPEG-1 ENCODER 5.1 Coded Bitstream Structure 26 5.1.1 Sequence Layer 26 5.1.2 Group of Pictures Layer 26 5.1.3 Picture / Slice / Macroblock / Block Layers 28 5.2 Encoding Process 29 5.2.1 Source Video 30 5.2.2 Motion Estimation 30 5.2.3 Motion Compensation 3 2 5.2.4 Discrete Cosine Transform 32 5.2.5 Quantisation 33 5.2.6 VLC Coding 34 5.2.7 Encoder Output 34 5.2.8 Reconstructed Pictures 34 5.2.9 Rate Control 35 5.2.10 Error Reporting 35

CHAPTER 6 MPEG-1 DECODER AND VIDEO PLAYBACK 6.1 Decoding Process 36 6.1.1 Error Reporting 3 8 6.2 Video Playback 38

CHAPTER? RESULTS 7.1 Coded Bitstream Bit Rate 39 7.2 Picture Quality at Various Bit Rates 40 7.3 Numerical Accuracy 42 7.4 Processing Speed 42 7.4.1 Efficient MATLAB Code 43 7.4.2 Efficient Design 44 7.4.3 Encoder and Decoder Execution Profile 44

CHAPTER 8 CONCLUSION 8.1 Summary of Findings 47 8.2 Suggestions for Future Work 48 BIBLIOGRAPHY

APPENDIX A - SAMPLE ENCODER/DECODER OUTPUT

APPENDIX B - CODER STRUCTURE CHART

APPENDIX C - CODE I-PICTURE STRUCTURE CHART

APPENDIX D - CODE P-PICTURE STRUCTURE CHART

APPENDIX E - CODE B-PICTURE STRUCTURE CHART

APPENDIX F - MOTION ESTIMATION STRUCTURE CHART

APPENDIX G - DECODER STRUCTURE CHART

APPENDIX H - DECODE I-PICTURE STRUCTURE CHART

APPENDIX I - DECODE P-PICTURE STRUCTURE CHART

APPENDIX J - DECODE-B PICTURE STRUCTURE CHART

APPENDIX K - LIST OF MATLAB FUNCTIONS

APPENDIX L - MATLAB CODE LISTING CHAPTER 1

INTRODUCTION

l.lMPEG-1 ORIGINS

MPEG (Moving Picture Expert Group) is a group formed by the International Standards Organisation (ISO) to create standards for digital video and audio. The official title for the group is: ISO/IEC JTCl SC29 WGll.

ISO: International Standards Organisation lEC: International Electrotechnical Commission JTCl: Joint Technical Committee 1 SC29: Sub-committee 29 WGl 1: Working Group 11 (moving pictures with audio)

MPEG meets several times a year and receives input from both academic researchers and industry. One of the products of MPEG is the MPEG-1 standard for video and audio compression. The full title of the standard is:

"Information Technology - Coding of moving pictures and associated audio - For digital storage media at up to about 1.5 Mbits/s IS 11172"

Work on MPEG-1 began in 1988 in response to an increasing need for a common format for representing compressed video on digital storage media such as CD-ROM, Digital Audio Tape (DAT), computer hard disks, writable optical disks as well as LANs and telecommunications links [1]. By making the format an international standard, advantages such as compatibility between different vendor equipment, reduced cost, and encouraging consumer confidence that products based on the standard have a reasonable lifetime, are realised [4]. The target bit rate is 1.5 Mbps because CD-ROM, DAT and other digital storage media transfer data at around this rate. Video is allocated approximately 1.15 Mbps of this bandwidth, with the remainder going to audio and the synchronisation and timing functions between the video and audio bit streams [1]. Expressing the bit rate as "about 1.5 Mbps" is deliberate as the overall bit rate can vary from one application to another. 1.2 RELATED STANDARDS

MPEG-1 is primarily intended for displaying progressive video (as opposed to interiaced TV pictures) with formats up to 288 lines of 352 pels and picture rates of around 25 to 30 Hz. (A pel is an 8-bit sample of luminance or chrominance data and is a synonym for pixel. A picture is an image consisting of luminance and chrominance data, and is similar to a TV frame). The fact that MPEG-1 is optimised for these parameters differentiates it from other standards such as H.261, JPEG and MPEG-2. JPEG uses similar compression techniques, but is intended for still images only. H.261 is optimised for visual telephony at low-bit rates (multiples of 64 kbps) where motion is more Hmited. MPEG-1 shares many of the features of H.261, such as the entropy coding tables. MPEG-2 is a newer standard intended for higher quaUty video at higher bit rates than MPEG-1 to support services such as entertainment television and HDTV (high definition television). MPEG-2 is much more than simply a scaled up version of MPEG-1. It supports field based coding (to handle interlace) and adds more powerful techniques for processing video bit streams in terms of profiles and levels at a cost of much greater complexity.

1.3 PROJECT AIM

The aim of this project was to develop an MPEG-1 encoder which produced a coded bitstream which complies with the standard at bit rates of approximately 1.15 Mbps, and to develop an MPEG-1 decoder which complies with the standard. The encoder and decoder have been developed in MATLAB - a fourth generation computer language with powerful toolbox functions for image processing, graphics and other applications. MATLAB was selected as the development environment for its powerful features and ease of programming.

Computer files stored in a format known as MPEG-1 Source Input Format (SIF - see Chapter 3) were used as source video for the encoder. The encoder output bitstream was stored as a binary file. The decoder used a binary file bitstream as input, and stored output video in MPEG-1 SIF format to file. The performance of the encoder and decoder was assessed in terms of the picture quality at the target bit rate (1.15 Mbps) and the processing speed of the encoder and decoder.

There is not a great deal of literature on MPEG-1 due to the fact that it is relatively new technology. Besides professional journal articles and the standard itself, a significant amount of information is available over the Internet. Internet sites such as www.crs4.it provide excellent descriptions of the algorithm, whilst other sites such as www.cs.tu- berlin.de/'-phade provide general information such as details on MPEG-1 chips such as the MCD250 by Motorola, and on software products such as the set of tools available from the company North Valley Research. The significance of this project in the context of these other products is that the MATLAB code created is very easy to modify and adapt to different appHcations. This is important given the intended application of project as a research tool for future standards development, and as a teaching aid for video coding.

1.4 THESIS OVERVIEW

The MPEG-1 standard is described in terms of its structure, applications and features in Chapter 2. Chapter 3 develops this further by explaining how the MPEG-1 compression algorithm works. Chapter 4 discusses the approach used for developing the software for the encoder and decoder, and the guidehnes followed for developing an efficient, reliable and maintainable program. The implementation of the encoder and decoder is discussed in Chapters 5 and 6 respectively. Several test video sequences were used to test the encoder and decoder. In Chapter 7 the performance of the encoder and decoder in terms of picture quality at various bit rates was assessed, as was the processing performance of both programs. The conclusions and suggestions for future work are presented in Chapter 8. CHAPTER 2

MPEG-1 STANDARD

2.1 STRUCTURE OF THE STANDARD DOCUMENT

The MPEG-1 standard is broken into four parts: systems (part 1), video (part 2), audio (part 3) and conformance testing (part 4). Part 1 specifies how the video and audio bitstreams are synchronised and combined into a multiplexed structure for playback; Part 2 specifies the coded representation of the video bitstream and the decoding process (but not the encoding process); Part 3 specifies the coded representation of the audio bitstream; and Part 4 specifies how to test a bitstream for compliance with Parts 1, 2 and 3. This project examines Part 2 and implements an encoder and decoder in accordance with Part 2.

Part 2 begins with a general overview of the algorithm for encoding and decoding video. The remainder of the standard is divided into normative sections which explicitly define the requirements of the standard and how they are to be implemented, and informative sections which attempt to describe the normative sections and make them more understandable.

The standard is very difficult to follow and understand. The normative elements are expressed in a very terse format with a mixture of brief sentences, pseudo-C algorithms for describing bitstream syntax, tables and definitions. They represent the algorithms which proved to be the most efficient and effective according to the selection criteria during the standardisation process. The normative elements are tightly defined in a way which describes what they are and how they inter-relate with each other, but generally not in terms of why, or which option or parameter value (from a range of alternatives) is optimal for a particular case. For example, in section 2.4.4 "The Video Decoding Process" a pseudo-C procedure is defined for reconstructing DCT (discrete cosine transform) coefficients. Part of the procedure is shown below:

if ((dct_recon[m][n] & 1 == 0) dct_recon[m][n] = dct_recon[m][n] - Sign(dct_recon[m][n]);

where dct_recon[m][n] is the coefficient of the m^*^ row and n^^ column of the matrix of reconstructed DCT coefficients & is the bitwise And operator == is the Equal to relational operator Sign(x) = 1, X > 0, Sign(x) = 0, x == 0, Sign(x) = -1, x < 0

The above process tests if a coefficient is even by extracting the least significant bit. If a coefficient is even and positive it is made odd by subtracting 1, whilst if it is even and negative, it is made odd by adding 1. The point here is that the process is described by the pseudo-C syntax, but why the process is carried out is not, which makes it difficult to understand the significance of the process in relation to the algorithm as a whole. The informative sections generally help in this regard (although not in the above case). Other documents which were helpful were Simulation Models Two and Three (SM2 [2], SMS [3]), and also a number of Internet sites. The simulation models are very similar to the informative sections of the final standard, and were used during the standardisation process for testing and comparing different algorithms. The Internet sites were useful because they provided descriptions of MPEG-1, a glossary of terms more comprehensive than the standard, and because the number of sources consulted provided a number of different ways of describing the same information which aided in understanding of the standard.

2.2 APPLICATIONS

MPEG-1 applications can be divided into two main categories: symmetric applications and asymmetric applications [4]. Symmetric applications are those which use both compression and decompression. The video source is typically a video camera, or pre- recorded video which is being edited. Examples of symmetric applications include electronic publishing (production), video mail, video telephone and video conferencing [4]. Asymmetric applications use the decompression process only (the video is compressed and stored once only). Video games, movie delivery and electronic publishing (eg education and training, travel and point of sale) are examples of asymmetric applications.

In the applications where video is stored onto digital storage media, the encoder must have knowledge of the bit rate of the storage device. This is because the decoder must be matched to the storage rate of the device so that it can display pictures at the desired rate without decoder buffer underflow or overflow. For this to be achieved, the encoder must exercise rate control over its output in accordance with the device bit rate and the decoder buffer size. MPEG-1 specifies a decoder buffer size of 327680 bits to meet most applications, which are specified as meeting the "constrained parameters bitstream" [1]. Rate control is discussed in later chapters.

2.3 FEATURRS

The MPEG-1 bitstream syntax has been tailored to provide a number of features which are useful for the above applications. The syntax uses pictures which are coded independently of other pictures (ie. coded as a still image) periodically to allow frequent starting points for decoding video. These pictures, known as intra-coded or I-pictures require a considerably greater number of bits than other pictures coded in conjunction with past and/or future pictures, so there is a trade-off between the amount of compression achieved and the degree of random access to the bitstream. The pictures coded with reference to past pictures are known as predicted pictures (P-pictures) and they achieve higher compression than I-Pictures because they only code the motion differences between the current picture and the previous reference picture. This process is known as motion compensation and is described in more detail in Chapter 3. The pictures which code the motion differences between past and/or future reference pictures are known as bidirectionally-predicted pictures (B-pictures). B-pictures provide the greatest amount of compression, and unlike I and P-pictures, are not used as reference pictures for other P and B-pictures. Other MPEG-1 compression features such as the Discrete Cosine Transform (DCT), quantisation, and variable length coding are described in Chapter 3.

In addition to random access to the bitstream, other features supported by the algorithm include fast search (fast forward and reverse), reverse playback, and the ability to edit the coded bitstream. Error control in conmiunications channels is beyond the scope of this standard, however, the algorithm allows for the decoder to recover and re- synchronise if errors do occur. The decoder re-synchronises by searching the bitstream for the next valid slice start code (a slice is part of a picture), and continuing decoding from that point. The missing parts of the picture may be concealed by fiUing them with previously decoded information from the same picture or previously decoded information in the same location from a previous or future picture. The technique used for concealment is the responsibility of the decoder designer. 7

CHAPTER3 MPEG l COMPRESSION AT.GORTTRM

MPEG-1 uses a number of sophisticated techniques for achieving high compression and for supporting the features described in Chapter 2. These are grouped into a flexible, generalised format which can be customised to meet the requirements of a wide range of applications. This means that applications with very different requirements can tailor the elements of the standard to meet their needs without suffering a performance penalty. This section provides a brief overview of the key techniques and elements of MPEG-1. For a more detailed discussion refer to Le Gall, 1991 or to the standard itself.

3.1 OVERVIEW

Compression in MPEG-1 is achieved through a combination of techniques which remove redundancy within each picture (spatial redundancy) and between pictures (temporal redundancy), and through subsampling of chrominance pels, quantisation, variable length coding, and picture interpolation. Quantisation and chrominance subsampling are the only lossy processes, where some information is permanently lost through compression. Chrominance subsampling takes advantage of the reduced sensitivity of the human eye to chrominance information relative to luminance information.

Temporal redundancy works on the theory that successive pictures in a video scene contain similar information, so that previous or future pictures can be used to predict the current picture. Based on this, a decoder can reconstruct the original picture given that it has received a difference picture from the encoder (current picture - prediction picture) and can form a prediction from a previously decoded picture. Picture interpolation means that a picture can be reconstructed from average values from previously decoded past and future pictures. The prediction process, known as motion compensation, uses 16x16 pel areas known as macroblocks and is described in detail later in this chapter.

Spatial redundancy within a picture occurs when adjacent pels or groups of pels are identical or nearly identical in terms of brightness or colour. Spatial redundancy is removed by transforming pel (or pel difference) values at the block level (8x8 pels) into 8 spatial (horizontal and vertical) frequencies and keeping only the most significant frequencies. The transform used in MPEG-1 is the Discrete Cosine Transform (DCT). The low frequencies are usually the most significant as they often contain most of the spatial information for the block. The DC or zero frequency component represents the average brightness level for the block. Quantisation is used to remove the smaller high frequency coefficients.

After quantisation most of the DCT coefficients are zero, so the remainder are coded by scanning the block in a zig-zag pattern (see Figure 3.6) to determine the run of zeros between coefficient levels (run-length coding), and finally these (run, level) pairs are coded using variable length codewords (modified Huffman code). Shorter codes are used for statistically more frequent values [4].

3.2 CONSTRAINED PARAMETERS

MPEG-1 supports a wide range of picture sizes, aspect ratios and bit rates. A sub-set of these parameters, known as the Constrained Parameters, has been defined to meet the majority of applications and to minimise the cost and complexity for hardware implementations of the algorithm (see Figure 3.1). Part 4 of this standard, which specifies how to test bitstreams for conformance with Parts 1, 2 and 3, only applies to constrained parameter bitstreams.

Horizontal picture size <= 768 pels Vertical picture size <=: 576 lines Picture area <= 396 macroblocks (macroblock = 16x16 pels) Pel rate <= 396x25 macroblocks per second Picture rate <= 30 Hz Motion vector range < "*"/- 64 pels (using half-pel vectors) Decoder buffer size <= 327 680 bits Bit rate <= 1.856 Mbps (constant bit rate)

Figure 3.1 Summary of Constrained Parameters [1] 3.3 SOURCE VIDEO

The source video may come in a wide variety of formats. The international standard for digital TV, CCIR Recommendation 601, is a popular format. It specifies a colour space with three 8-bit components: Y, Cb and Cr. Y is the luminance component and Cb and Cr are chrominance colour difference signals. Y has a range from 16 (black) to 235 (white) which gives 219 levels. Cb and Cr are represented in two's complement form centred on 128 with 224 possible levels. Y, Cb and Cr are derived from the original red (E'R), green (E'G) and blue (E'B) colours as follows [12]

Y = (0.30 * ER + 0.59 * Eg + 0.11 * Er) where g = 2.2 - 2.8 Y =16 + 219*Y Cb = 128 + 112 * (0.5/0.886) * (E'B - Y) where E'B is gamma corrected blue Cr = 128 + 112 * (0.5/0.7071) * (E'R - Y) where E'R is gamma corrected red

Y is sampled at a resolution of 720x576 pels which is too large for MPEG-1 rates around 1.15 Mbps. Y, Cb and Cr are generally converted into the lower resolution Source Input Format (SIF). This conversion process can be implemented in a number of ways, but using only the odd field, decimation filtering and subsampling provides the best results (see Figure 3.2) [1]. This is a lossy process which also reduces the picture rate from 50 Hz (outside the constrained parameter range) to 25 Hz. Because the human eye is less sensitive to colour than brightness, the chrominance components are subsampled more heavily than the luminance component (2:1 subsampling in both horizontal and vertical directions with respect to the Y sampling resolution). Finally the first and last 4 columns of Y, and the first and last 2 columns of Cr/Cb are discarded to give the "significant pel area" which is divisible by 16 in both dimensions (see Figure 3.2). The result of this conversion process is known as 4:1:1 format SIF with 1 Cr sample and 1 Cb sample for every 4 (2x2 pel) Y samples:

720 720 4 352 4 288 Odd Field Only 288

Select 1 Field Horizontal Filter and Subsample Significant Pel Area

Figure 3.2. Conversion from CCIR Rec. 601 Y to SIF (showing significant pel area) 10

3.4 PICTURE TYPES

Source pictures are coded as intra (I-picture), predictive (P-picture), or bi-directionally predictive (B-picture) picture types. A fourth type of picture (D-picture) is also available, but this is not part of the constrained parameters bitstream, and is not used in conjunction with the other types of pictures. I-pictures are self contained in that they are coded using only the information contained within the picture itself. They are used to allow random access to the bitstream. All macroblocks within an I-picture are intra- coded (i.e. without prediction from any other picture).

P-pictures are coded as a prediction from the most recent previous reference picture, which is either an I or a P-picture, using forward motion compensation. P-pictures may consist of a number of different types of macroblock, which are determined by image characteristics in the current picture and the reference picture. Macroblocks may be predictive, intra or skipped (not coded). P-pictures provide a greater amount of compression than I-pictures, and act as reference pictures for a future P or B-picture.

B-pictures are coded as a prediction from a previous reference picture, future reference picture, or an interpolation (average) of the previous and future pictures. Macroblocks in B-pictures may be forward-predictive (like P-pictures), backward-predictive, interpolative, intra or skipped. As for P-pictures, the choice of macroblock type depends on image content. B-pictures provide the greatest amount of compression but cannot be used as reference pictures. B-pictures are useful for the following reasons [4]:

• Noise reduction through the averaging process and, • Deals with "uncovered" areas not predictable from a past reference • Allows decoupling between prediction and coding (no error propagation)

The main disadvantages of B-pictures are [4]:

• Picture order is changed at the encoder output. Extra picture buffers are required at the decoder to hold the reference pictures. Picture re-ordering is required after decoding. • Computational complexity is increased. • Coding delay is increased. • Makes random access points (I-pictures) further apart. • Decreases correlation between pictures (i.e. predictors not as good). n

The picture order is not defined by the standard, but subject to the trade-off between the above advantages and disadvantages with reference to the apphcation. A typical order for a constrained parameter bitstream is:

Forward Prediction Backward Prediction

ujju y

Figure 3.3 Example of Picture Arrangement in a Video Sequence

3.5 MOTION COMPENSATTON

P and B-pictures use motion compensation to remove temporal redundancy between pictures. Motion compensation is usually performed using the luminance component of the picture at the macroblock level. This means that a macroblock in the current picture is modelled by a macroblock in a reference picture (previous, future or both), which may need to be moved horizontally and/or vertically relative to the position of the macroblock in the current picture to correct for motion between the pictures. The horizontal and vertical movement is known as the motion vector (right and down is positive) and typically has a resolution of half a pel, and a range which can extend from a few pels to the entire picture boundary (see Figure 3.4). If motion between pictures is very great, then resolution can be decreased to single pel accuracy, which also doubles the search range. The simplest example of motion compensation is when there is no motion or change between the pictures. In this case the motion vector is (0,0) and the macroblock in the prediction macroblock is in the same position relative to the macroblock in the current picture. The amount and direction of motion in a picture depends on the individual video sequence. Motion compensated prediction is performed by both the encoder and decoder. Synchronisation is maintained by using decoded 12

pictures as reference pictures to form a motion compensated prediction in both the encoder and the decoder.

Determining which motion vectors are best for each macroblock is achieved by a process known as motion estimation, which is performed by the encoder only. Motion estimation takes a macroblock in the current picture and tries to match it with a macroblock in the reference picture (see Figure 3.4). The matching criteria is typically based on the minimisation of a cost function, such as the squared error between the current macroblock and the motion compensated macroblock. The choice of search range and search strategy is left up to the encoder. Larger search ranges may be required for applications with very fast moving objects such as sports videos. The only search strategy guaranteed to find the optimal match is the full search technique where all possible motion vectors within the search range are tested using the matching criterion. This technique is slow and therefore less optimal but faster techniques such as the three-step search or telescopic search are usually used [1]. Motion estimation techniques are not specified by the standard, only the way that the motion vector is sent to the decoder.

Reference Picture Current Picture

current nnacroblock j|' best match

search area

horizontal motion vector

vertical motion vector

Figure 3.4 Motion Estimation

3.6 DISCRETE COSINE TRANSFORM (DCT)

The DCT is a block based transform coding technique which transforms an image block (luminance or chrominance) into its constituent spatial frequencies. The two dimensional DCT is defined as follows [1]: 13

7 7 F(u,v) = 1/4 C(u)C(v) I I f(x,y) cos(n(2x+l)u/16) cos(n(2y+l)v/16) x=0 y=0

with: u, V, X, y = 0, 1,2, ..7 where: x, y = spatial coordinates in the pel domain u, V = coordinates in the transform domain C(u),C(v) =1/V2foru,v = 0 = 1 otherwise

Figure 3.5 Two-dimensional Discrete Cosine Transform

The effect of the DCT is that it packs most of the information within the image block into the low frequency components in the top-left region of the transform block. From the top-left coefficient of the 8x8 coefficient transform block, horizontal spatial frequencies increase to the right, and vertical spatial frequencies increase down. The DCT does not achieve compression in itself - this comes from the subsequent quantisation and coding steps [4].

The DCT was selected over other transform techniques such as the Karhunen-Loeve transform, discrete Fourier transform, and Walsh-Hadamard transform during the standardisation process due to the transform's high energy packing capability (second only to the Karhunen-Loeve transform) and the fact that it is not too complex for implementation in hardware [13]. The Karhunen-Loeve transform is the most efficient transform in terms of energy packing capability but is computationally very complex. The discrete Fourier transform is complex to implement in hardware, whilst the Walsh Hadamard transform is relatively simple to implement. The energy packing capability of both of these transforms, however, is not as good as the DCT [13].

3.7 OUANTLSATTON

Quantisation of DCT coefficients is a lossy process which results in high compression and also provides a means for rate control. Different quantisation algorithms are used for intra-coded and non-intra-coded blocks because non-intra blocks contain pel difference values (rather than actual pel values) which will be close to zero if the 14

prediction macroblock was close to the original. The non-intra-quantiser has a "deadzone" around zero so that most of these difference pels quantise to zero. If the prediction was poor, then the macroblock should be intra-coded. A "visually weighted" technique is applied where the DC coefficient is quantised by a fixed quantiser, and the higher frequencies are quantised more heavily because the eye is less sensitive to quantisation effects at higher frequencies. There is no deadzone in the intra-quantiser.

3.8 VARIABLE LENGTH CODING (VLO

Following quantisation, the remaining non-zero coefficients are scanned in a zig-zag pattern (see Figure 3.6) and run-length coded where the run of zeros between non-zero coefficients, and the coefficient level are recorded. These (run, level) pairs are coded using variable length codes which assign shorter codes to statistically more frequent combinations [4]. For example, a run of three zeros from the last non-zero coefficient to the next non-zero coefficient, which has a level of -2, would have the following VLC code:

Run = 3, Level = -2: VLC = 0010 0100 1

Variable length codes are also used for other parameters in the bitstream such as motion vectors and macroblock address increments.

1 2 6 7 15 16 28 29 3 5 8 14 17 27 30 43 4 9 13 18 26 31 42 44 10 12 19 25 32 41 45 54 11 20 24 33 40 46 53 55 21 23 34 39 47 52 56 61 22 35 38 48 51 57 60 63 36 37 49 50 58 59 63 64

Figure 3.6 Zig-Zag Scan Sequence For 8x8 Pel Block 15

3.9 LAYERED SYNTAX

The MPEG-1 bitstream syntax is arranged into six layers which each support a signal processing or system function. The layers are hierarchical in the sense that each layer is a subset of the next highest layer. The top four layers (Sequence, Group of Pictures, Picture and Slice) are marked by start codes in the bitstream. The layers, then- functions and key parameters are described below [1]:

• Sequence Layer (The video sequence) - bit rate, picture size, picture rate and constrained parameters flag.

• Group of Pictures Layer (Random access point) - time stamp used for specifying time in hours minutes and seconds. • Picture Layer (Primary coding unit) - picture type (I, P or B-picture) and the temporal reference, which identifies the proper order of pictures for playback. • Slice Layer (Re-synchronisation unit (under error conditions)) - quantiser scale ( a value from 1-31 used to scale the quantiser for rate control) • Macroblock Layer (Motion compensation unit) - macroblock address (macroblocks are numbered from 0 to 395 for SIF resolution pictures), macroblock type (intra- coded, predictive-coded, bi-directionally predictive-coded, or skipped), quantiser scale, and coded block pattern (a VLC used to describe which blocks within a macroblock have been coded). • Block Laver (DCT unit) - coded Y, Cr and Cb data. 16

CHAPTER 4

SOFTWARE DEVELOPMENT METHODOT.OOY

The two major programs developed in this project are an MPEG-1 encoder and an MPEG-1 decoder. The encoder takes digital video input from a source file and produces a binary output file which contains a bitstream defined by the MPEG-1 syntax. The decoder does the reverse process. Video playback of either encoder input or decoder output is achieved using a number of smaller programs independent of both the encoder and decoder.

"Software development methodology" refers to the process used to build the encoder and decoder. The basic aim of this process was to design and implement the encoder/decoder in accordance with the MPEG-1 standard within the timeframe available. This is a very broad aim which could have been achieved in a number of ways, not all of which would be satisfactory according to a more detailed set of criteria. Kendall (1989) suggests a set of criteria or goals for large software projects:

• Effectiveness - meets the requirements • Efficiency - uses a minimum amount of resources (eg processing time, memory requirements) to meet the requirements • Useable - "user friendly" • Reliable - defects are minimised • Maintainable - easy to modify after becoming operational (well structured and well documented)

Several different methodologies could have been used to achieve these goals. Each of these defines a "project life cycle" from start (concept) to finish (operational program), and breaks the life cycle into a number of phases. The main differences between the methodologies are the number and type of phases, and how these relate to each other. Two different methodologies were considered - the classical life cycle and structured analysis and design. 17

4.1 CLASSICAL LTFR CYCJ F.

As its name suggests, the classical life cycle is tlie traditional approach for software projects. It breaks the development life cycle into seven distinct phases. These phases are sequential, with the output of one phase being the input of the next [10]:

1. Problem Recognition - establish exacdy what the problem is, and what is to be done 2. Feasibility Study - determine whether the project is worth doing 3. Analysis - determine the requirements 4. Design - design a program which meets the requirements, based on discrete modules which perform a particular function 5. Construction - write and test the program 6. Conversion - change from existing way of doing things to the new program (if applicable) 7. Maintenance - repair and upgrade the program as required

The construction (coding and testing) phase is broken down further into a number of stages: unit test, subsystem test, system integration and finally whole system test and debug [10]. The unit test occurs after each stand-alone function or module has been independently coded and debugged. When all the modules are working, they are grouped into larger subsystems and tested (subsystem test). When this has been achieved, subsystems are grouped into the total system which is then tested and debugged.

There are several major problems with the classical approach. The first is that the program is only visible at the end of the life cycle as the full version, rather than throughout the life cycle as a series of incrementally developing versions. The danger with this is that the most difficult task (interfaces between subsystems) is left until last when there is little time left for proper testing and debugging (given a set deadline) [10]. Although the completion of each phase can be viewed as a milestone, this is not necessarily a measure of the proportion of work done since early analysis and design errors will only be detected late in the process, resulting in the majority of the effort going into the final testing, and maintenance phases. Various sources (Kendall (1989), Page-Jones (1990) and Yourdon (1989)) agree that systems developed using the classical life cycle generally do not meet the goals of reliability and maintainability described previously. 18

4.2 STRUCTURED ANALYSIS AND DESTGN

Structured analysis and design is a disciplined methodology for developing software which attempts to address the shortcomings of the classical approach. It places more emphasis on analysis and design in order to minimise the number of errors and omissions, and therefore make the overall program more reliable and easier to maintain. The main features of structured analysis and design are [10, 11]:

• It is a systematic approach for developing reliable and maintainable software • It is non-sequential and allows feedback • Large problems are partitioned into a hierarchy of smaller sub-problems • Ideas should be conveyed graphically whenever possible • Documentation is an integral part of the development process • It encourages incremental development which is a better indicator of successful progress than milestones in the classical approach • It encourages use of fourth generation computer languages for prototyping

Structured analysis and design was selected as the framework for developing this project on the basis of these features, and the inherent advantages over the classical life cycle. The aim was to perform a number of builds of the encoder and decoder, starting with very simple versions and adding complexity or added functions after each iteration. The major iterations are shown in Table 4.1. Since the decoder performs approximately the inverse function to the encoder, a critical check was to ensure that the decoder output matched the encoder input after each iteration. A schematic of the structured analysis and design process is shown in Figure 4.1. Subsequent sections will explain how each of the processes in the schematic works, and describe the key inputs and outputs for each process.

4.3 STRUCTURED ANALYSIS

The aim of the structured analysis process was to determine a set of requirements for the problem under consideration. Large problems were partitioned into smaller, more manageable sub-problems. How the requirements were expressed depended on the nature of the problem, and the simplest way of understanding the problem. Approaches used include writing requirements in point form (interpreted from the standard), decision trees (yes/no branches), and flowcharts. Decision trees and flow charts greatly 19

assisted the analysis because they helped to make unclear statements clear, and helped to identify an optimal sequence of events from the requirements and decision trees.

Figure 4.1. The project life cycle using structured analysis and design 20

ENCODER DECODER DCT Inverse DCT Quantisation Inverse Quantisation VLC Coding Inverse VLC coding Motion estimation n/a Motion Compensation Motion Compensation Code I-Picture (Y only) Decode I-Picture (Y only) Code P-Picture (Y only) Decode B-Picture (Y only) Code B-Picture (Y only) Decode P-Picture (Y only) Play Video Play Video Code Group of Pictures Decode Group of Pictures Include chrominance (Cr, Cb) Include chrominance (Cr, Cb) Optimisation Optimisation

Table 4.1 Major Iterations in Development of the Encoder and Decoder

A useful example is the selection of macroblock type within slices in B-pictures. This is based on a sequence of decisions (choosing the optimal path at each decision) and also other criteria interspersed throughout the standard. The fact that the requirements for this problem are scattered throughout the standard is also significant because it meant that other areas of the document had to be consulted to check if they had any impact on the problem under consideration. In this case, the following requirements were relevant [1]:

1. Selection of Macroblock Tvpe in B-Pictures (clause 2-D.6.5.4, p-D-53) - Specifies decision sequence for selecting macroblock type (see Figure 4.2): 1.1 Motion compensation mode - forward, backward, interpolative or skipped? 1.2 Intra or Non-intra Coding - Code the macroblock by itself (intra) or using motion compensation from 1.1? 1.3 If the macroblock is non-intra, should it be coded? 1.4 Does the quantiser scale need to be changed? 2. Skipped Macroblocks (clause 2.4.4.4, p. 46) - In B-pictures, the skipped macroblock has the same macroblock type (forward, backward or interpolated) as the previous macroblock, differential motion vectors equal to zero and no DCT coefficients. In 21

B-pictures, a skipped macroblock is not allowed to follow an intra-coded macroblock. 3. Slice (p. 47) - the first and last macroblocks of a slice shall not be skipped (p. 25) 4. Intra/Non-Intra Coding Decision (p. D-48) - Specifies how the variance of a luminance macroblock could be determined. 5. Predictive-coded Macroblocks in B-pictures (clause 2.4.4.3, p. 45) - Specifies how predictive-coded macroblocks are decoded. 6. Intra-coded Macroblocks (clause 2.4.4.1, p. 45) - Specifies how intra-coded macroblocks are decoded.

Forward Motion Compensation

Backward Motion Begin Compensation

Interpolated Motion Compensation ^

Quant Pred-*cq Coded No Quant Pred-*c Non-lntra

Not Coded Fred-* or Skipped

Quant Intra-q Intra No Quant Intra-d

* = i, f, or b

Figure 4.2 Decision Tree for Selecting Macroblock Type in B Pictures [1]

Once all the requirements had been determined, the next step was to combine them, and add any prerequisite steps for the requirements using a simple flowchart.

4.4 ACCEPTANCE TEST GENERATION

Acceptance tests describe any type of test which verifies that the program or group of program modules developed meets the requirements. These tests use the requirements as a starting point, but other common sense considerations (eg visual quality of the 22 results) were also be used to identify any unusual behaviour, or possibly erroneous results.

In the above example, the acceptance tests were simply:

• Check that first and last macroblocks of the slice are not skipped • Check that a skipped macroblock does not follow an intra-coded macroblock • Check that all types of macroblocks are present over a range of test images • Check that the visual quality of a B-picture constructed from all of the different macroblock types is satisfactory

Appendix A shows a sample coding output which was used to aid in the above tests.

4.5 STRUCTURED DESIGN

Structured design is a systematic design approach for specifying program modules in terms of the interfaces (parameter passing) between modules and the algorithms inside modules based on a problem specification. Whereas structured analysis addresses the "what" aspect of the problem, structured design addresses the "how". It seeks to reduce complexity by partitioning problems into smaller problems, and by organising these smaller problems in a hierarchical fashion. This is also a feature of top-down design. What differentiates structured design from top-down design and other techniques is that it incorporates guidelines for the partitioning process based on the concepts of coupling (degree of linkage between modules), cohesion ( degree of sameness of activities within a module) and others [9].

The tools of structured design are the structure chart and pseudo-code. The structure chart is a graphical representation of program structure. Modules are represented as boxes, and communications between modules are represented by lines and arrows. The structure chart does not show flow directly, like a flowchart, but rather it shows the program hierarchy of modules. Higher level modules generally call a number of lower level modules which, after execution, return control to the higher level module. Structure charts for the encoder are shown at Appendices B-J. Pseudocode is used to represent the algorithms within modules.

Because the choice for breaking a complex problem into simpler sub-problems is obviously application specific, structured design offers a "tool box" of guidelines to aid the process, and to provide a measure of assessing the success of a design. The success 23

is based on the degree of modularity, or the extent to which each module acts as a "black box" by hiding its internal workings. This is used as a measure because modular programs are easier to understand and change, less susceptible to error propagation, and restrict the problem scope so that errors can be tracked quickly. Page-Jones (1988) describes features of a good design as follows:

• Minimum number of parameters passed between modules • Simple, self explanatory parameter names • Local module to module communications rather than through global data • Obvious connections between modules (easy to understand why modules communicate)

Coupling

CoupHng is the measure of the interdependence of two modules. A good design minimises coupling by getting rid of unnecessary relationships between modules, and by making the necessary ones simple and easy to understand. Page-Jones (1988) describes the main types of coupling in best to worst sequence as: couphng by parameters, global coupling and content coupling. Coupling by parameters further subdivided into (again best to worst): data coupling , stamp coupling and control coupling. These are described as follows :

• Coupling by Parameters - * data coupling - normal case of passing parameters between modules. * stamp couphng - use of composite parameters which may not be immediately obvious * control coupling - use of flags to switch logic in a module (can make things obscure) • Global Coupling - Makes the program less modular through use of global variables and more difficult to understand. Makes errors more difficult to trace because errors propagate between modules, and the last update time of global data may not be clear. • Content Coupling - Program modules which need to know the workings of another module to carry out their activities. Defeats the purpose of modularity.

Cohesion

Cohesion is the measure of the "sameness" of activities within a module in terms of a logical grouping based on function, sequence and time [9]. A good design uses highly cohesive modules. 24

Additional guidelines

Page-Jones (1988) suggests other guidelines for good design, some of which include: • Factoring - separate code from one module into a new function to reduce module size, and avoid having the same function repeated in more than one module. • Decision spUtting - separate decision of what action to take, and execution of that action. • Error Reporting - adopt a consistent error reporting pohcy which reports errors from the module which detected the error, and knows why it happened.

4.6 CODING AND TESTTNG

Coding and testing was the most time consuming process. Special features of MATLAB were used to improve the efficiency of code, and to simpHfy the coding process, and to take advantage of some of the powerful library functions. Examples are described in Chapter 5. Testing involved determining and checking boundary conditions for the algorithms.

4.7 PROGRAM DOCUMENTATION

Due to the methodology used, documentation was an integral part of the development process. The point form requirements, decision trees, flowcharts and structure diagrams all served as documentation of the current state of design, and comments within the code documented the implementation. This form of documentation was helpful in measuring progress, for integrating new features or functions to the current design, and for trying to understand what had been done previously, as in some cases work was re- examined after eight months. The paper documentation was in rough, handwritten format which allowed for fast development and corrections, rather than trying to record all details on a computer.

4.8 OUALITY ASSURANCE

The quality assurance process was an overall examination of the performance of the encoder and decoder as a whole in terms of correctness (compHance with the standard). 25 and an assessment of how well the encoder and decoder perform (speed, visual quality, design considerations)

4.9 RESULTS

The structured analysis and design methodology used does not guarantee a successful design in terms of the criteria proposed at the start of this chapter. It just provides an approach which is more Ukely to produce a successful product. The encoder and decoder are successful in that they achieve the desired functions. The number of hnes of MATLAB code for both programs is over 7500 (including comments) which is a large amount considering that MATLAB is a fourth generation language, and should be able to do the same job as a third generation language such as C, but in fewer lines. Although the overall development time was much longer than initially anticipated (usually greater than double), this was primarily due to the complexity of the standard and ensuring that it was implemented correctly rather than the analysis and design approach taken. The advantage of structured analysis and design can be summarised as follows:

• It provided good overall perspective for the project • It provided guidelines for the development process • Making changes due to a misinterpretation of the standard, or errors, was easier because the analysis and design steps were documented 26

CHAPTER 5 MPEG-1 ENCODER

The MPEG-1 standard specifies the syntax and semantics of the coded bitstream and the signal processing in the decoder. It does not specify the encoding process. The consequence of this is that there are many valid implementations possible, and many valid coded bitstreams. This Chapter discusses the coded bitstream structure created by the encoder and the encoder implementation. These descriptions focus on the key features and implementation details, and not the theory behind aspects such as motion compensation, quantisation and the DCT as these have been previously covered in Chapter 3. Except where stated otherwise, the coded bitstream and decoding process developed in this project conform to the standard.

5.1 CODED BITSTREAM STRUCTURE

5.1.1 Sequence Layer

The sequence layer is the top layer in the bitstream hierarchy and consists of a start code, one or more sequence headers and one or more group of pictures. The encoder was written in a way to support multiple sequence headers and multiple groups of pictures in the bitstream, however, the encoder was tested using one sequence header and one group of pictures to simpHfy testing of the decoder.

The sequence header is used to define the key parameters needed by the decoder, and has been specified as a non-constrained parameters bitstream because the encoder bit rate is variable and therefore does not meet the constrained parameter constant bit rate limit of 1.856 Mbps. Key parameters in the sequence header are defined in Table 5.1.

5.1.2 Group of Pictures Layer

A group of pictures (GOP) consists of a start code and one or more I, B or P-pictures. D-pictures are not coded together with any of the other picture type and have not been included in the encoder. Pictures within a group of pictures may be ordered in display order (normal viewing order) or in bitstream order, where B-pictures are placed after each pair of I or P-pictures. The decoder requires pictures to arrive in bitstream order to 27

allow it to perform backward motion compensation. The encoder reorders the source pictures from display order to bitstream order simply by extracting the required picture from the source UVC file (see picture reorder function Figure 5.4). An offset was allowed so that first picture (extracted in display order) could be anywhere in the video sequence.

PARAMETER VALUE COMMENTS Horizontal Size 352 pels MPEG SIF minus first & last 4 columns Vertical Size 288 pels MPEG SIF Pel Aspect Ratio 0.9375 720x576 at 4:3 = 0.9375 Picture Rate 25 pictures/second PAL applications Bit Rate 3FFF Hexadecimal Variable bit rate Marker Bit 1 Avoids start code emulation Video Buffer Verifier Size 19 (* 16 * 1024) bits Maximum decoder buffer size Constrained Parameter 0 Not a constrained Flag bitstream

Table 5.1 Sequence Header Parameters

Pictures within a group of pictures must conform to the following rules [1]:

1. A GOP, in bitstream order must start with an I picture and may be followed by any number of I, P or B-pictures in any order 2. A GOP, in display order must start with an I or B-picture, and must end with an I or P-picture. A GOP can range in size from one I-picture to an unlimited size. 3. A GOP starts with a GOP header and ends at the next GOP header, or at the next sequence header or at the end of sequence code, whichever comes first.

Three different group of picture types were built into the encoder. All of these are defined as closed which means that they do not require a previously decoded picture from a previous GOP to be decoded. The first two groups were used primarily for testing, whilst the third is more typical of a group for a picture rate of 25 Hz with I- pictures spaced approximately every 0.5 seconds. 28

Type 1 in display order: 10 B1 P2 Type 1 in bitstream order: 10 P2 B1

Type 2 in display order: 10 B1 B2 P3 Type 2 in bitstream order: 10 P3 B1 B2

Type 3 in display order: 10 B1 B2 P3 B4 B5 P6 B7 B8 P9 BIO Bll P12 Type 3 in bitstream order: 10 P3 B1 B2 P6 B4 B5 P9 B7 B8 P12 Bll B12

Figure 5.1. GOP Types in Display and Bitstream Orders

Key parameters in the GOP header have been defined as follows:

PARAMETER VALUE COMMENTS Drop Frame Flag 0 Not 29.97 Hz Time Code 4096 Marker bit set to 1, remainder of time code not known and set to 0 Closed GOP 1 GOP is closed Broken link 0 Bitstream not edited

Table 5.2 Sequence Header Parameters

5.1.3 Picture / Slice / Macroblock / Block Layers

Pictures are divided into a number of slices which contain a start code and a number of macroblocks in raster scan order. Slices are used to help the decoder recover from errors in noisy environments. If the decoder detects that bits are in error, it can search for the next slice start code and resume decoding from that point. Slices can vary in size from 1 macroblock to the complete picture (396 macroblocks) but the first shce must start at the top left of the picture, the last slice must end at the bottom right of the picture, and the intermediate slices must be contiguous.

All pictures were divided into 18 slices of one shce per row as shown in Figure 5.2. Each slice therefore consisted of 22 macroblocks where each macroblock consisted of 29

four 8x8 pel luminance blocks and two 8x8 pel chrominance blocks as shown in Figure 5.3. Macroblocks were addressed in raster scan sequence starting from macroblock 0 (first macroblock in first slice) and ending with macroblock 395 (last macroblock in last slice).

Slice 1

y ''' '

" -yyf „

i i

Y""'

fi;

' ? ' ' ' ^ '' i

5-f , V " i i //

Slice 18

Figure 5.2. Arrangement of Slices Within a Picture

YO Y1 Cr Cb 8 pels 4 5 Y2 Y3 8 pels

Figure 5.3. Macroblock Structure

5.2 ENCODING PROCESS

The encoding process is summarised in the schematic diagram at Figure 5.4 30

5.2.1 Source Video

The source video used in this project was stored as computer files in MPEG SIF format with a luminance resolution of 360x288 pels, YCbCr sampling pattern of 4:1:1 and picture rate of 25 Hz. The source video was obtained from CCIR 601 format files with a resolution of 720x576 pels and a YCbCr sampling pattern of 4:2:2 at picture rate of 25 Hz. The conversion from the original CCIR 601 resolution and sampling pattern into MPEG SIF format was achieved using a number of different utility filtering and subsampling programs written for the Universal Video Codec Project [6]. Figure 5.5 shows the header of one of the source files (Susie) which was stored in "UVC" (Universal Video Codec) file format. Reading and writing of these files was performed using the MATLAB functions "uvc_open", "uvc_read_frame" and "uvc_write_frame" which were written by Mark Pickering.

Figure 5.4. Simplified Video Encoder Block Diagram

5.2.2 Motion Estimation

The key decisions to be made for motion estimation were: 31

• Use the original previous and future pictures or the decoded previous and future pictures as reference pictures? • Which block matching criteria should be used? • What search strategy should be used to find the best match? • What search range?

The standard states that using the original pictures as a reference gives the most accurate motion vectors, whilst using the decoded pictures gives the smallest values in the difference picture. This results in a trade-off between increased noise and greater spurious motion, however the standard states that there is usually little or no difference in quality between the two methods. The decoded previous and future pictures were used as reference pictures in this encoder.

### Header info for "susie.411 .sif' ###

Id (hex of UVCG or GCVU) = 0x55564347 Header version Number = 2 Offset Application field = 64 Offset Comment field = 112 Offset to video data = 226 Pixels per Line = 360 Lines per Frame = 288 Number of Frames = 125 Frame_rate = 25.000 Interlace fields per frame = 2 Sampling Shape = ORTHOGONAL Pixel Width - Aspect ratio = 1 Pixel height Aspect ratio = 1 Resolution Data format = UNSIGNED Video data Structure = YUV411

Figure 5.5. UVC Header File for Video Sequence "Susie"

The two main block matching criteria are the squared difference and the absolute difference between the motion compensated block and the current block. The absolute difference metric was chosen because it was anticipated that taking the absolute value would be a faster process than taking the square and therefore reduce processing time. 32

There are a number of search strategies for finding the best match macroblock. The optimal solution is always found when using a full search where every position within the search range is tested, however this strategy is computationally expensive. To allow for comparison tests, the full-search strategy and the faster three-step search strategy with a search range of+/- 7.5 pels were implemented. The three-step search strategy is a good trade-off between speed and complexity [13]. It uses four search grids with spacings of 4, 2, 1 an 0.5 pels respectively. The test macroblock is first shifted by 0, 4 and -4 pels in each direction. The best match from these nine searches is then used as the centre position for the 2 pel search grid, and the search repeated using shifts of 0, 2 and -2 pels. This process is continued until the best match is found for the 0.5 pel grid. Half pel values are computed using linear interpolation.

The motion estimation process in the encoder produces a file of 396 motion vectors for each reference picture in one step. This approach was taken to avoid recomputing motion vectors every time when testing the subsequent DCT, quantisation and VLC coding phases.

5.2.3 Motion Compensation

The decoded previous and future pictures were used as reference pictures for motion compensation. In B-pictures, the motion compensation can be forward, backward or interpolated. The interpolated motion compensated prediction is formed by taking the average of the forward and backward motion compensated prediction macroblocks. Once the motion vectors were computed, they were applied to the reference picture to form the motion compensated prediction for both luminance and chrominance components. The motion vectors for the chrominance components were obtained by halving the luminance motion vector, and rounding the magnitude towards half a pel (i.e. values from [0 .. 0.5> go to 0 and values from [0.5 .. 1> go to 0.5).

5.2.4 Discrete Cosine Transform

For I-pictures, the DCT input values have a range from 0 to 255. For P and B-pictures, the input values are the difference values between the current and predicted blocks which have a range from -255 to 255. The DCT outputs are within the range -2048 to 2047. The final version of the encoder and decoder uses a fast DCT programs written in C by Dr Michael Frater. 33

5.2.5 Quantisation

All macroblocks in I-pictures, and intra-coded macroblocks in P and B-pictures use a uniform quantiser. The DC coefficient is always quantised by dividing by 8 and rounding. The AC coefficients are quantised using the following equation, and then clipped to [-255, 255]:

Q = (8*dct_coeffs) / (quantizer_scale * iq_matrix)

where quantizer_scale is a number in the range 0 to 31 dct_coeffs is the 8x8 block of DCT coefficients iq_matrix is the 8x8 default intra quantizer matrix for MPEG-1: iq_matrix = [ 8 16 19 22 26 27 29 34 16 16 22 24 27 29 34 37 19 22 26 27 29 34 34 38 22 22 26 27 29 34 37 40 22 26 27 29 32 35 40 48 26 27 29 32 35 40 48 58 26 27 29 34 38 46 56 69 27 29 35 38 46 56 69 83 ] Non- intra-coded macroblocks in P and B-pictures use a uniform quantiser with a dead zone around zero. Coefficients are quantised using the following equation, and then clipped to [-255, 255]:

Q fix ((8 * dct_coeffs) / (quantizer_scale * niq_matrix))

where quantizer_scale is a number in the range 0 to 31 niq_matrix is the 8x8 default non-intra quantizer matrix for MPEG-1 (an 8x8 matrix with each element having a value of 8) dct_coeffs is the 8x8 block of DCT coefficients fix means truncate towards zero

The quantiser scale was initialised to a default value of 8 (in the slice layer) for I, B and P-pictures and was kept constant for the entire picture. Adjusting the quantiser scale is one of the primary mechanisms of rate control in the MPEG-1 algorithm. Because automatic rate control was not implemented, the quantiser scale was changed manually during testing to roughly meet the target rate of 1.15 Mbps. 34

5.2.6 VLC Coding

VLC coding is used for quantised coefficients, macroblock address increments, coded block patterns, motion vectors and other parameters. A technique used by Dr Michael Frater for generating forward and inverse VLC tables was used for VLC coding and VLC decoding. This technique is fast because the VLC code (or the value that it represents) is returned by directly indexing a table (matrix), rather than doing an exhaustive search of all possible values. The VLC tables were pre-computed and loaded into the encoder and decoder during initialisation. Addressing the tables and checking bounds is complicated, so MATLAB functions were written to hide the complexity. For example, the function get_dct_vlc() returns the VLC code for a given DCT coefficient run and level. Another function get_runjevel() returns the run and level for a given VLC code. Escape codes are not returned for any (run, level), but instead get_run_level() indicates when the table has been searched beyond it's bounds which indicates that either the VLC is an escape code or it is in error.

Each of the functions which makes the VLC tables documents how the VLC tables work. These are as follows:

make_dct_vlc.m : DCT coefficient (run,level) tables make_addr_inc_vlc.m : Address increment tables make_cbp_vlc.m : Coded block pattern tables make_mv_vlc.m : Motion vector tables

5.2.7 Encoder Output

Output from the encoder is expressed in binary format. This was packed into 8-bit bytes and written to a binary file.

5.2.8 Reconstructed Pictures

Because I and P-pictures are used as references for motion compensation, quantised macroblocks from these pictures were passed through an inverse quantiser, inverse DCT and then added to the motion compensated macroblock to form the reconstructed macroblock. This was stored in the future or previous picture buffer depending on the whether it was to be used for forward or backward motion compensation. 35

5.2.9 Rate Control

The encoder does not exercise rate control over the output. This means that the output bit rate is always variable.

5.2.10 Error Reporting

Errors are reported by the module in which they occur using the MATLAB error() function. 36

CHAPTER 6 MPEG-1 DECODER AND VIDEO PT AYRACK

6.1 DECODING PROCESS

The decoding process is essentially the reverse of the encoding process and will be explained by example. Consider a bitstream containing the following group of pictures:

Coded Bit Stream: 10 P3 B1 B2

After decoding the sequence and group of pictures header, the VLC decoder decodes the first picture header and detects that it is an I-picture with temporal reference 0 (see Figure 5.3). The VLC decoder then decodes the quantiser scale, followed by the quantised DCT coefficients which are inserted into 8x8 blocks according to the decoded run/length values. The coefficients are then passed through the inverse quantiser and inverse DCT to produce the actual pel values which are stored in the previous picture buffer and display buffer:

Arrival picture Previous Picture Future Picture Display Buffer 10 10 - 10

The VLC encoder decodes the next picture header and detects picture P3. The VLC decoder decodes motion vectors, and the pel values corresponding to a macroblock. The motion vectors are appUed to the previous picture to form a motion compensated prediction macroblock. This is added to the decoded macroblock to produce the reconstructed macroblock which is stored in the future picture buffer and display buffer:

Arrival picture Previous Picture Future Picture Display Buffer P3 10 P3 10, P3

The next picture detected is Bl. Decoded motion vectors are applied to the previous picture and (if applicable) to the future picture and a motion compensated prediction is formed, and added to the decoded macroblock. The result is stored in the display buffer. 37

Arrival picture Previous Picture Future Picture Display Buffer B1 10 P3 10, P3,B1 The next picture detected is B2 and this is decoded and stored in the same way as B1. The decoder detects that the pictures in the display buffer are sequential after sorting, and writes all four pictures to a UVC file in order of temporal reference. The display buffer is then cleared, the future picture store is copied into the previous picture store, and then the future picture store is cleared. The VLC decoder then looks for the next picture header.

Arrival picture Previous Picture Future Picture Display Buffer B2 10 P3 10, P3,B1,B2 P3

VLC Inv Inv Decoder Q DUI

Forward Previous Motion Picture Comp. Store

Future Picture Store

Decoded Video

Figure 6.1. Simplified Video Decoder Block Diagram 38

6.1.1 Error Reporting

Errors at the slice layer and below are reported to the slice layer decoding function which then looks for the next slice code. The missing sUce is filled by the corresponding slice from the previously decoded picture, or the previously decoded slice from the current picture. Errors above the slice layer (picture level, group of pictures level, are reported by the module in which they occur using the MATLAB "errorO" function.

6.2 VIDEO PLAYBACK

MATLAB has a number of utiHties for displaying a sequence of images as a movie. These were incorporated into two functions: load_image_deck() and play_video(). The function load_image_deck() takes colour pictures from a UVC file and inserts them into a MATLAB structure called an image deck (which is conceptually like a deck of cards). The function play_video() takes the image deck and plays it as a movie at the required rate.

Colour images in MATLAB are best displayed as an "indexed image". These consist of an image matrix and a colormap. The image matrix contains pel values which are used as an index to the colormap which determines the red, green and blue hardware look-up Table values for display on the screen. The maximum number of elements in the colormap is 256. This technique produced high quality colour still images, but poor quality colour video with speckled images, and incorrect colours. A number of experiments for video sequences of 13 pictures resulted in a method which displayed high quality video, summarised as follows:

1. Find the average colormap for the sequence (easiest approximation is to use the middle image) with: [picture, map] = rgb2ind (R,G,B,248,'dither'). 2. Colormap length must be less than or equal to 248, because MATLAB reserves 8 colours for graphics objects such as Figures and also graph lines and text. 3. Fit the average colormap to the rest of the images using: picture = rgb2ind(R,G,B,map). 4. Display pictures at their true resolution (rather than using the default MATLAB resolution) by using the truesize() function. 39

CHAPTER? RESULTS

This chapter discusses the performance of the encoder and decoder in terms of the bit rate of the coded bitstream, picture quahty, numerical accuracy within the encoder and decoder, and finally, processing speed. Results are not presented for the intermediate builds of the encoder and decoder, as performance tests on the encoder and decoder are only meaningful when the algorithm has been fully implemented.

The performance of the encoder and decoder is highly dependent on the scene content of the test video. For example, a scene with large, smooth areas and minimal motion between pictures will result in high compression due to the high degree of temporal and spatial correlation, but it will be more susceptible to blocking artifacts in the smooth regions if the quantiser scale is too large. A fast moving picture scene with a large amount of detail will result in lower compression, but less blocking artifacts in the busy regions for the opposite reasons.

The video scenes used to test the encoder and decoder roughly approximate these two cases. "Susie" is a typical video conferencing scene with a woman talking on a phone in front of a static, smooth background. There is minimal movement, and the colour changes are smooth and gradual. "Calendar" shows a toy train moving in front of a child's bedroom wall and calendar. Both the train and the calendar are moving quickly, the scene is "busy" with detail, and the colour range and colour contrast is very high. Figure 7.1 shows the luminance component of "Susie" and "Calendar".

7.1 CODED BITSTREAM BIT RATE

The coded bitstream rate was estimated by measuring the mean bit rate for I, B and a P- pictures, and extrapolating these figures to 25 pictures based on a typical group of pictures sequence (picture rate is 25 pictures/second), and then adding the sequence and group of pictures header bits, and sequence end code. This is shown below:

I-picture = i bits P-picture = p bits B-picture = b bits Figure 7.1(a) - Luminance Component of Susie

Figure 7.1(b) - Luminance Component of Calendar 40

Sequence header = 96 bits GOP header = 59 bits Sequence end code = 32 bits

Two GOP in display order: 10 B1 B2 P3 B4 B5 P6 B7 B8 P9 BIO Bll P12 113 B14 B15 P16 B17 B18 P19 B20 B21 P22 B23 B24 P25

For 25 pictures: No. of I-pictures = 2 No. of P-pictures = 7 No. of B-pictures = 16

Coded Bitstream Rate = (2 * i) + (7 * p) + (16 * b) + 96 + 59 + 32 bps

Figure 7.2 Computation of Coded Bitstream Bit Rate

The encoder produces a variable bit rate coded bitstream because it does not exercise rate control. With the encoder initialised to the default parameters described in Chapter 5, the bit rate for Calendar more than doubled the target rate of 1.15 Mbps, whilst the bit rate for Susie was approximately one third of the target rate. For Calendar, the bit rate was reduced by increasing the quantiser scale from the default value of 8 to 31 for B-pictures, and from 8 to 12 for I and P-pictures. This effectively lowers the overall bit rate by decreasing the quahty of B-pictures whilst maintaining the quality of I and P- pictures at a higher level. The quality of I and P-pictures is more important than that of B-pictures because I and P-pictures are used as references for motion compensation and therefore decreasing the quality of these pictures would significantly increase the motion compensated prediction error. Table 7.1 summarises the effect of quantiser scale on bit rate.

7.2 PICTURE QUALITY AT VARIOUS BIT RATES

Picture quality can be determined mathematically by measurement of the peak signal to noise ratio (PSNR), or visually by comparing the decoded video with the original video. The PSNR measures the correlation between two pictures and is defined as follows [13]: 41

PSNR = 10 logio (255''2/MSE) dB

N,M where MSB = 1/NM I (cy

and MSB = Mean Square Brror M = Number of lines in picture N = Number of pels per line cij = pel value at position (i,j) in reference picture di,j = pel value at position (i,j) in test picture

Figure 7.3 Peak Signal to Noise ratio (PSNR)

PSNR is measured in decibels and rough indicators are PSNR < 20 dB - poor correlation; PSNR > 30 dB - very good correlation; and PSNR = infinity for perfect correlation.

When visually assessing the quality of video, the following points were used as guides for classifying coding artifacts, or errors resulting from the coding process:

• Visible 16x16 pel or 8x8 pel block outlines in individual pictures • Spurious block motion between pictures • Loss of detail • Poor colour range and/or colour contrast • Loss of sharp edges • Non-uniform or blocky brightness in smooth areas • Flicker between pictures

Table 7.1 describes the picture quality at various bit rates in terms of PSNR and visual quality. 42

TEST I/B/P BIT PSNR VISUAL QUALITY VIDEO QUANT. RATE for SCALE (Mbps) I/P/B (1-31) (dB) Calendar 8/8/8 3.25 I -31.89 • excellent video quality almost P - 32.45 identical to original B - 32.67 • red area on ball a little patchy • ball's shadow not as sharp as original Calendar 12/31/12 1.00 I - 28.55 • good video quality P - 28.84 • number edges on calendar a little B - 23.98 soft, red numbers are faint • slight shimmering effect during playback Susie 8/8/8 0.416 I - 37.86 • overall video quality very good P-37.51 • slight shimmering effect during B-35.61 playback • background slightly different to original in smooth areas

Table 7.1 Picture quality at various bit rates

7.3 NUMERICAL ACCURACY

In addition to comparing the decoded pictures to the original pictures, the decoded pictures were compared with the reconstructed pictures in the encoder. It was found that there was no difference between the reconstructed and decoded I, P and B-pictures which confirms the correctness of the encoding and decoding processes.

7.4 PROCESSING SPEED

The encoder and decoder design and MATLAB implementation were developed with efficiency in mind, but not at the cost of making code too obscure. After the encoder and decoder were completed, various operations were timed with a view to identifying which operations were the most time consuming. 43

7.4.1 Efficient MATLAB Code

"MATLAB is a technical computing environment for high-performance numeric computation and visualisation" [8]. All numbers in MATLAB are double precision (64 bit) and numeric computation is optimised for matrices. MATLAB is an interpretive interactive language which has some C-like conmiands and syntax, and a powerful set of library functions for many applications such as image processing and graphing. MATLAB functions are analogous to functions in C, and MATLAB scripts are analogous to C main programs.

The MATLAB documentation recommends the following for improving processing speed [8]:

• MATLAB's built-in matrix operations are more than an order of magnitude faster than its compiler/interpreter operations. Code should be written for matrix manipulation wherever possible. eg instead of i = 0; for i= 1:1000 i = i + 1; y(i) = sin(t); end

use t= 1:1000; y = sin(t) • Preallocate memory for large matrices. For the above example use the following declaration first: y = zeros(1,1000) • There is no memory penalty for passing large matrices as arguments to functions, unless the matrices are altered within the function

7.4.2 Efficient Design

The encoder and decoder were designed with efficiency in mind. For example, all of the VLC tables are predefined and loaded into the main program during initialisation. This saves constructing the VLC tables every time the program is run. The tables are also far more efficient than conventional Table search methods because the row and column index are used to locate the code/value directly. 44

The built-in MATLAB DCT and IDCT functions are relatively slow. A fast C language DCT program and inverse DCT program (written by Dr Michael Frater) proved to be about an order of magnitude faster than their MATLAB equivalents, and was Hnked to the MATLAB code. These functions were timed using the MATLAB cputime() function which returns the amount of time in seconds that the operating system has allocated to the MATLAB process. The average timings are shown in Table 7.2.

FUNCTION AVERAGE EXECUTION TIME PER 8x8 PEL DCT (ms) MATLAB DCT 11.6 CDCT 1.67 MATLAB Inverse DCT 16.7 C Inverse DCT 1.67

Table 7.2 MATLAB and Fast C DCT Timings

7.4.3 Encoder and Decoder Execution Profile

The major functions of the encoder and decoder were timed for Calendar and Susie during a normal execution sequence running on a GCS computer with quad HyperSparc 90 MHz CPUs. The results are shown in Table 7.3. As expected there were large differences between the two scenes, however for the purposes of identifying the slowest activities, the relative differences between the times within the same function are significant. For Calendar, the most costly operation in the encoder is the coding of B- pictures, whilst in the decoder the most costly operation is the decoding of I-pictures. This suggests that these two operations should be examined first for possible optimisation, and then the timings should be repeated to gauge how effective the optimisation has been. Optimisation of these functions has not been undertaken, however the operations within a B-picture were examined more closely to determine where the time was being spent.

Within a B-picture slice, a series of tests is conducted to select the best macroblock type for coding efficiency and closeness to the reference macroblock. In the first slice of Calendar this produces 9 macroblocks which use interpolated motion compensation, 7 macroblocks which use forward motion compensation and 6 macroblocks which use 45 backward motion compensation. The average time spent on each function in the decision and coding process is shown in Table 7.4.

This test shows that the test for a skipped macroblock, and coding a B-macroblock take roughly equal lengths of time. This makes sense as the test for a skipped macroblock is to try and code the current macroblock using the same motion compensation mode as the previous macroblock (if it was non-intra-coded) and if after coding all the coefficients are zero, then the macroblock is skipped. This test shows that for the encoder, optimisation efforts should be targeted at the operations within the function which tests for skipped macroblocks in B-pictures, and the function that codes B- macroblocks.

MAJOR FUNCTION EXECUTION TIME EXECUTION TIME CALENDAR SUSIE ENCODER Initialise decoder 1.4 seconds 1.4 seconds Code I-picture 12.7 minutes 10.8 minutes P-picture forward motion 2.8 minutes 2.3 minutes compensation (full search) Code P-picture 16.5 minutes 7.2 minutes B-picture forward motion 2.5 minutes 2.2 minutes compensation (full search) B-picture backward motion 2.5 minutes 2.2 minutes compensation (full search) B-picture motion compensated 3.3 seconds 2.7 seconds interpolation Code B-picture 31.2 minutes 10.2 minutes Total Coding Time 68.2 minutes 34.9 minutes

DECODER Initialise decoder 1.4 seconds 1.4 seconds Decode I-Picture 25.5 minutes 18.1 minutes Decode P-Picture 17.5 minutes 5.0 minutes Decode P-Picture 17.0 minutes 3.0 minutes Total Decoding Time 60.6 minutes 26.7 minutes

Table 7.3 Execution profile for encoder and decoder 46

MAJOR FUNCTION IN AVERAGE FUNCTION CODING B SLICE EXECUTION TIME (CALENDAR) Test for skipped macroblock 2.3 seconds Select motion compensation mode 6.1 ms Intra/Non- intra decision 3.8 ms Code B macroblock 2.5 seconds

Table 7.4 Execution profile for functions within a B-slice 47

CHAPTERS CONCLUSION

8.1 SUMMARY OF FINDINGS A variable bit rate MPEG-1 encoder and MPEG-1 decoder have been successfully developed in the MATLAB language. The encoder takes source video in the MPEG-1 SIF 411 format from UVC files and compresses the video into a bitstream with a syntax defined by the standard. The bitstream is written to a binary file. The encoder output is variable because the encoder does not exercise automatic rate control over the output. Consequently, coded bit streams produced by this encoder do not conform to the MPEG-1 constrained parameters bitstream where a constant bit rate (of 1.856 Mbps or less) is required. Manual control over the encoder output can be exercised by setting the quantiser scale to suit the scene content of the test video. It is normally preferable to increase the quantisation scale of B-pictures whilst keeping the quantisation scale of I and P-pictures relatively constant to ensure accurate motion compensation. Depending on the scene content, good quality video can be produced at the target rate of approximately 1.15 Mbps.

The decoder takes a binary file as input and decodes it into a video sequence which is stored in a UVC file. The video can be played back at 25 pictures/second using separately written functions. The decoder has been made generic to handle a wide range of bitstreams. Aside from the source bit rate, which is ignored because the decoder assumes an infinite input buffer, the decoder is capable of coding a constrained parameters bitstream. The decoder can decode bit streams with different groups of pictures formats which are open or closed, and adjust to quantiser scale changes, quantiser matrix changes, and different slice lengths. The decoder has error resihence built in so that an error in decoding causes the decoder to look for the next slice start code, and the missing part of the slice is filled with macroblocks from the previous picture or previous slice.

The processing time performance of the encoder and decoder depends on the scene content of the test video. As a general rule, the most costly operations in terms of processing time are coding B-pictures, and decoding I-pictures. 48

8.2 SUGGESTIONS FOR FUTURE WORK

Two main areas are suggested for future work. The first is automatic rate control of the encoder output, and the use of a decoder buffer whose size is determined by the rate of the coded bitstream. This could be achieved by including a theoretical decoder buffer at the encoder output, and continuously monitoring its size. If the buffer was tending towards overflow, then the quantiser scale could be increased proportionally at the next slice or macroblock. A more extreme measure could involve discarding high frequency DCT coefficients until the buffer size returns to normal. If the buffer was tending towards underflow, then the quantiser scale could be reduced, macroblock stuffing could be inserted into the bitstream, or additional zeros could be prepended to start codes.

The second suggestion for future work is to optimise the computationally intensive operations by restructuring the design, or by rewriting parts of the design in the C language. BIBLIOGRAPHY

[1] ISO/IEC JTCl SC29 WGl 1 "Information Technology - Coding of moving pictures and associated audio - For digital storage media at up to about 1.5 Mbits/s CD 11172", Dec 6 1991

[2] ISO/IEC JTCl SC2 WG8 MPEG90/41 MPEG Video Simulation Model Two (SM2), Jul 90

[3] ISO/IEC JTCl SC2 WGl 1 MPEG90/41 MPEG Video Simulation Model Three (SMS), Jul 90

[4] Le Gall, D., "MPEG: A video compression standard for multi-media appHcations", Communications of the ACM, Apr 1991, Vol. 34, No. 4

[5] Thiele, A.N., and Burton, J., "Standards in Digital Video", Journal of Electrical and Electronics Engineering, Australia, Vol. 13, No. 3, Sep 1993

[6] Pang, K., Biggar, M., Dunstan, S., & Arnold, J., "The Australian Universal Video Codec Project", Journal of Electrical and Electronics Engineering, Australia, Vol. 13, No. 3, Sep 1993

[7] Thompson, C.M., and Shure, L., Image Processing Toolbox for Use with MATLAB The MathWorks, Inc., USA, 1993

[8] MATLAB User's Guide The MathWorks, Inc., USA, 1992

[9] Page-Jones, M., The Practical Guide to Structured Systems Design (2nd Edn.) Prentice-Hall, USA, 1988 [10] Kendall, P., Introduction to Systems Analysis and Design: A structured Approach (2nd Edn) Wm. C. Brown, USA, 1992

[11] Yourdon, E., Modem Structured Analysis Prentice-Hall, USA, 1989

[12] Recommendations and Reports of the CCIR, 1990 XVIIth Plenary Assembly, Dusseldorf, 1990 Volume XI - Part 1 Broadcasting Service (Television) Rec 601-2 - "Encoding parameters of digital television for studios"

[13] Cavenor M.C., and Arnold J.F., AELE.8248 Video Coding Course Notes, 1995 APPENDIX A

SAMPLE ENCODER / DECODER OUTPUT >> mpegl_code loading initialisation data ... initialisation of data = 0.0106 minutes

** START OF MPEGl LOG ** source_file cal.411.sif output_file cal.bin picture size 288 X 352 number of macroblocks 396 start coding sequence start gop output file size: -rw-r 1 bate j 5 19 Nov 27 14:10 cal.bin start coding picture 14 picture coding time = 25.282222 minutes output file size: -rw-r 1 batejS 19331 Nov 27 14:36 cal.bin start coding picture P6 motion fwd = 3.03 3 6 minutes

** Matrix of 396 macroblock motion compensation modes. ** Nos. in brackets indicate percentage of that macroblock type i = interpolative prediction (0.0) b = backward prediction (0.0) f = forward prediction (85.1) I = intra coded (1.0) s = skipped (1.8) N = forward prediction but no motion vector (12.1) mcp_modes =

N f f f f f f f f f f N f f f f f f f f f f N f f f f f s f f f s f f f f f f f f f f f f f f f f f f f f s f f f f f f f f f f f f N f f f f f f f f f f f f f f f f f f f f f N f f f f f f f f N f f f f f f f f f f f f N f f f f f f f f f f f f f f f f f f f f I f f f f f f f f f f f f f f f f f f f f f f N f f f f f f f f f f f f f f f f f f f f f N f f f f f f f f N f f f f f f f f f f f f N f f f f f N N f f f f f f f f f f f f f f N f f f f f s f f f f s f f f f f f f f f f N f f f f f f f f N f f f f s f f f f f f I N N f N f f f f f f N I f f N f f N N f f f N f f f f f f f f f N f f f f f f f f f f f N f f f f f f f I f f f f f f f f f f f f N N N N N N N f N f f f N f f N f f N N s f N N f f f f f f f f f f f f f f f N N f N N N N f f f f f f f f f f f f f f f f f f f N N picture coding time = 18.458611 minutes output file size: -rw-r 1 batej5 28317 Nov 27 14:56 cal.bin start coding picture B5 motion fwd = 2.7222 minutes motion bwd = 2.6478 minutes motion interp = 0.0556 minutes

** Matrix of 396 macroblock motion compensation modes. ** Nos. in brackets indicate percentage of that macroblock type i = interpolative prediction (16.7) b = backward prediction (16.9) f = forward prediction (15.4) I = intra coded (0.5) s = skipped (50.5) N = forward prediction but no motion vector (0.0) mcp_modes = b b b i b f b b b s s s s b f b i s s f s f b f s b s f s s s b s s i s s b b b f s f f i b i b b i f s s s s s i i b b f s i b f f b b i s s s s s s f s s s b s s i f s b s f i b f s s f f i s s s s s I f i i s f f s f i i b s s s f s s s b s s b f s s s s s s f i s s s f s I f s s s s s f i i b s s s s f b f s s s s s s b f f s s b i i b s s s s f i b b s s f s b f s b b s b i i b i f b s f b s s f s s s s s s s s f i i s s s s s s f b s f s s s s s s i i s b s f f i i i f b f b f f s s s s s s i i s s f s b b i b f s f i i f b s i b s b i f b s i s i i b i i i f b b f s s i i i i i s s s s s i i i s i f f b b i s s i i s i i s s s s s s s s i s f b i b b b i s f f s s i s i s s s s s s s s b b s s s s s s s s s s s s s s s s s f s s i b s s s s s s s s s s s s s s s s s s s s b picture coding time = 12.646667 minutes output file size: -rw-r 1 batej5 29719 Nov 27 15:10 cal.bin end gop end coding sequence Total coding time = 56.5219 minutes

** END OF MPEGl LOG ** loading initialisation data ... file cal.bin contains 29724 bytes getting next 10240 bytes from file ... picture size 288 X 352 Pel aspect ratio 0 .9375 Picture rate 25 pictures per second Bit rate 3FFFF i.e. variable Marker bit 1 VBV size 19 This is NOT a constrained parameters bit stream Drop frame flag Time code hours Time code minutes Marker bit Time code seconds Time code pictures Closed gop Broken link

** decoding picture 14 getting next 10240 bytes from file ... decode picture cputime = 28.049167 minutes

** decoding picture P6 previous picture: 14 , future picture: [] getting next 9244 bytes from file ... decode picture cputime = 16.732778 minutes previous picture: 14 , future picture: P6 ** decoding picture B5 decode picture cputime = 3.508889 minutes previous picture: P6 , future picture: [] Sequence end code detected ** END OF MPEGl LOG ** PSNR(I_ORIG,I_RECON) = 2 9.58 PSNR(I_ORIG,I_DECODE) = 2 9.58 PSNR{P_ORIG,P_RECON) = 29.83 PSNR(P_ORIG,P_DECODE) = 2 9.83 PSNR(B_ORIG,B_DECODE) = 2 6.77

Warning: Divide by zero PSNR(I_RECON,I_DECODE) = Inf Warning: Divide by zero PSNR(P_RECON,P_DECODE) = Inf

Total decoding time = 49.0278 minutes APPENDIX B-CODER STRUCTURE CHART

KEY: = Program = Module By Module Someone Else \ = Call to Module APPENDIX C - CODE l-PICTURE STRUCTURE CHART APPENDIX D - CODE P-PICTURE STRUCTURE CHART

code_p_picture

motion estim fwd code_p _slice get_slices

test mc test_ intra update_pic_store

code_p_mblock code i mblock

(see Appendix C for functions in codeJ_nriblock)

code ni block

get_p_type dct 8x8

get_addrjnc_vlc nonJntra_quant

code mv code coeffs

get_mv_vlc zig_zag

get_cbp_vlc get_dct_vlc

get_YCbCr_mblock inv_non_intra_quant

update_pic_store idct 8x8 APPENDIX E - CODE B-PICTURE STRUCTURE CHART APPENDIX F - MOTION ESTIMATION STRUCTURE CHART APPENDIX G-DECODER STRUCTURE CHART

mpeg1 _decode

^ 1 init coder var decode_seq_header init start codes

1 recover_matrix

open_bin_file decode_gop load vie tables uvc_open

decode_pic_type

decodeJ_picture

decode_p_picture

decode_b_picture

check order

uvc write frame APPENDIX H - DECODE l-PICTURE STRUCTURE CHART

decode._i_pictur e

1 decode_i_slice

1r decode addr inc decodej _mblock update_pic_store APPENDIX I - DECODE P-PICTURE STRUCTURE CHART APPENDIX J - DECODE B-PICTURE STRUCTURE CHART

decode_b_picture

(see Appendix H decode b slice for functions in decode_i_mblock) APPENDIX K

LIST OF MATLAB FUNCTIONS apply_mcp.m bytealigned.m check_order.m check_psnr.m code_ac.m code_b_mblock. m code b picture .m code_b_slice.m code_chrom_dc.m code_coeffs.m code_dc_uv.m code dc.._y.m code_gop.m code_i_block_uv.m code i block y.m code_i_mblock.m code i picture.m code_i_slice.m code_mv.m code_ni_block.m code_p_mblock.m code_p_picture.m code_p_slice.m code_seq_header.m code_sequence.m dct_2d.m decimal_to_simsbf.m decimal_to_uimsbf.m decode_addr_inc.m decode_b_mb_tYpe. m decode_b_mblock. m decode_b picture.m decode_b_slice.m decode_dc_uv.m decode_dc_y. m decode_gop .m decode_i_block. m decode_i_mblock. m decode_i__picture .m decode_i_slice.m decode_mv.m decode_ni_block.m decode_p_inb_tYpe. m decode_p_mblock. m decode_p_picture.m decode_p_slice.m decode_pic_tYpe.m decode_seq_header.m disp_y.m disp_yuv.m display_picture .m d i s p 1 ay_j/u v_p i c . m even.m find_f_codes.m find_variance.m full_search.m get_YCbCr_mblock.m get_addr_inc.m get_addr_inc_vlc.m get_b_type. m get_bits .m get_cbp.m get_cbp_vlc.m get_dct_vlc.m get_mv.m get_mv_vlc.m get_p_tYpe.m get_run_level.m get_slices.m get_tl.m get_top_lef t .m half_pel_search.m init_coder_var.m init_start_codes.m intra_quant.m inv_dct_2d.m inv_intra_quant.m inv_non_intra_quant.m load_image_deck.m load_tables.m log_message.m make_addr_inc_vlc.m make_cbp_vlc.m make_chrom_mcp.m make_dct_vlc.m make_mcp.m make_mv_vlc.m modulo.m mo t i on_e s t im_bwd.m mo tion_estim_fwd.m motion_estim_interp.m mpegl_code.m i^pegl_decode. m mse .m next_start_code.m nextbits.m non_intra_quant.m not_equal.m open_bin_file.m picture2yuv.m plaY_again.m plaY_video.m psnr.m read_bits.m recover_b_skipped.m recover_matrix.m recover_p_skipped. m rl_decode.m rl_decode_ac.m se1ect_mc_mode.m show_modes.m simsbf_to_decimal.m tell.m test_intra.m test_mc.m tes t_skipped.m uimsbf_to_decimal.m update_pic_store.m uvc_open.m uvc_read_frame. m uvc_wri te_frame.m wr ite_to_bitstream.m yuv2YCbCr.m yuv2picture.m yuv2rgb.m zig_zag.m APPENDIX T.

MATLAB FUNCTIONS function [mcp, err] = applY_mcp (previous_pic, t, 1, mv, full_pel_fwd) % Description % Applys motion vectors to a picture to form motion compensated I prediction macroblock. Returns 16 x 24 motion compensated prediction i macroblock. Motion vectors are positive to the right and down.

I Input: i previous_pic: motion vectors applied to macroblock in this picture i defined by (top, left) as below i top : top row of macroblock before motion vectors applied h left: left col of macroblock before motion vectors applied h mv : 1x2 array of horizontal & vertical motion vector ^ Array format is: h mv(coll, col2) = mv(mv_horiz, mv_vert); I full_pel_fwd : 0 for 0.5 pel resolution, 1 for full pel resolution h Output: h mcp : motion compensated prediction macroblock (16x24) h err : 0 if mcp valid, 1 for invalid mcp h by John Bateman jlobal HORIZ_SIZE jlobal VERT_SIZE

^ initialise values icp = zeros (16, 24) ; % mcp macroblock i correct motion vectors for 0.5 pel resolution .f ~full_pel_fwd mv = mv/2; 5nd

) bounds of best match macroblock ivh = mv (1) ; \w = mv(2) ; :op t + mw; )ottom = top + 15; -eft = 1 + mvh; right = left + 15; ifprintf('top left bot right= [%.lf %.lf %.lf %.If]\n',top,left,bottom,right); i Apply motion vectors if result is within bounds of picture • f (left>=l & right<=HORIZ_SIZE & top>=:l & bottom<=VERT_SIZE) half_row = (top - fix(top)) == 0.5; % true for half pel mv half_col = (left - fix(left)) =^0.5; % true for half pel mv

% retrieve bestmatch from half pel/full pel motion vectors if half_row & half_col % half pel row & col tA = top - 0.5; bA = tA + 15; lA = left - 0.5; rA = lA + 15; A = previous_pic(tA:bA, lA:rA); % top left integer macroblock B = previous__pic(tA:bA, lA+l:rA+l); % top right integer macroblock C = previous_pic(tA+1:bA+l, lA:rA); % bottom left integer macroblock D = previous_pic(tA+1:bA+l, lA+l:rA+l); % bottom right integer macroblock mcp_y^ = round((A + B + C + D)/4); elseif half_row % half pel row, integer col tA = top - 0.5; bA = tA + 15; lA = left; rA = lA + 15; A = previous__pic (tA:bA, lA:rA); % top left integer macroblock C = previous_pic(tA+1:bA+l, lA:rA); % bottom left integer macroblock mcp_y = round((A + C)/2); elseif half_col % integer row, half pel col tA = top; bA = tA + 15; lA = left - 0.5; rA = lA + 15; A = previous_pic(tA:bA, lA:rA); % top left integer macroblock B = previous_pic(tA:bA, lA+l:rA+l); % top right integer macroblock mcp_y = round((A + B)/2); else % integer row & col mcp_y = previous_pic(top:bottom, left:right); end else % error detected err = 1; status = sprintfCtop = %d, left = %d, right = %d, bottom = %d',top, ... left, right, bottom) log_message(status); log_message('motion vector out of bounds in Y block'); keyboard return end

% convert luminance motion vectors to chrominance motion vectors % (chrominance motion vectors obtained by halving and rounding towards % half a pel i.e. values from [0 .. 0.5> go to 0 and [0.5 .. 1> go to 0.5) temp = mv/2; mv_chrom = fix(temp); % truncate for i = 1:2 if (temp(i) - mv_chrom(i)) >= 0.5 mv_chrom(i) = mv_chrom(i) +0.5; % positive, add 0.5 elseif (temp(i) - mv_chrom(i)) <= -0.5 mv_chrom(i) = mv_chrom(i) - 0.5; % negative, subtract 0.5 end end

% find coordinates for u & v blocks (top-left pel) % (top,left) is coord of y macroblock top left pel tv = t - 8*((t - 1)/16); % V top Iv = 1 - 8*((1 - 1)/16) + HORIZ_SIZE; % v left tu = tv + VERT_SIZE/2; % u top lu = Iv;

% bounds of best match block for v block mvh = mv_chrom(l) ; mw - mv_chrom(2) ; top = tv + mw; bottom = top + 7; left = Iv + mvh; right = left + 7; h_min = H0RIZ_SIZE+1;h_max = HORIZ_SIZE+ H0RIZ_SIZE/2; v_min = 1; v_max = VERT_SIZE/2; %fprintf('chrominance mv = [%2.1f %2.If]\n',mv_chrom) ; %fprintf('top left bot right= [%.lf %.lf %.lf %.If]\n',top,left,bottom,right); % Apply motion vectors if result is within bounds of picture if (left>=h_min & right<=h_max & top>=v_min & bottom<=v_max) half_row = (top - fix(top)) == 0.5; % true for half pel mv half_col = (left - fix(left)) == 0.5; % true for half pel mv

% retrieve bestmatch from half pel/full pel motion vectors if half_row & half_col % half pel row & col tA = top - 0.5; bA = tA + 7; lA = left - 0.5; rA = lA + 7; A = previous_pic(tA:bA, lA:rA); % top left integer macroblock B = previous_pic(tA:bA, lA+l:rA+l); % top right integer macroblock C = previous_pic(tA+1:bA+l, lA:rA); % bottom left integer macroblock D = previous_pic(tA+1:bA+l, lA+l:rA+l); % bottom right integer macroblock n\cp_v = round((A + B + C + D)/4) ; elseif half_row % half pel row, integer col tA = top - 0.5; bA = tA + 7; lA = left; rA = lA + 7 ; A = previous_pic(tA:bA, lA:rA); % top left integer macroblock C = previous_pic(tA+1:bA+l, lA:rA); % bottom left integer macroblock mcp_v = round((A + C)/2); elseif half_col % integer row, half pel col tA = top; bA = tA + 7; lA = left - 0.5 rA = lA + 7; A = previous_pic(tA:bA, lA:rA) ; % top left integer macroblock B = previous_pic(tA:bA, lA+1:rA+l); % top right integer macroblock mcp_v = round((A + B)/2); else % integer row & col mcp_v = previous_pic(top:bottom, left:right); end else % error detected err = 1; status = sprintf('top = %d, left = %d, right = %d, bottom = %d',top, left, right, bottom) log_message(status); log_message('motion vector out of bounds in Cr block'); keyboard return end

% bounds of best match block for u block mvh = mv_chrom(l); mw = mv_chrom (2) ; top = tu + mw; bottom = top + 7; left = lu + mvh; right = left + 7; %fprintf('top left bot right= [%.lf %.lf %.lf %.If]\n',top,left,bottom,right) ; h_min = H0RIZ_SIZE+1;h_max = HORIZ_SIZE+ H0RIZ_SIZE/2; V_min = VERT_SIZE/2+l; v_max = VERT_SIZE;

% Apply motion vectors if result is within bounds of picture if (left>=h_min & right<=h_max & top>=v_min & bottom<=v_max) half_row = (top - fix(top)) == 0.5; % true for half pel mv half_col = (left - fix(left)) ==0.5; % true for half pel mv

% retrieve bestmatch from half pel/full pel motion vectors if half row & half col % half pel row & col tA top - 0.5; bA = tA + 7; lA = left - 0.5; rA = lA + 7; A previous_pic(tA:bA, lA: rA) ; % top left integer macroblock B previous_pic(tA:bA, lA+l:rA+l); % top right integer macroblock C previous pic(tA+1:bA+1, lA:rA) ; % bottom left integer macroblock D previous_jpic (tA+1: bA+1, lA+l:rA+l) % bottom right integer macroblock mcp_u = round((A + B + C + D)/4); elseif half_row % half pel row, integer col tA = top - 0.5; bA = tA + 7; lA = left; rA = lA + 7; A = previous_pic(tA:bA, lA:rA); % top left integer macroblock C = previous_pic(tA+1:bA+1, lA:rA); % bottom left integer macroblock mcp_u = round((A + C)/2); elseif half_col % integer row, half pel col tA = top; bA = tA + 7; lA = left - 0.5; rA = lA + 7; A = previous__pic (tA:bA, lA:rA); % top left integer macroblock B = previous_pic(tA:bA, lA+l:rA+l); % top right integer macroblock mcp_u = round((A + B)/2); else % integer row & col mcp_u = previous_pic(top:bottom, left:right) end else % error detected err = 1; status = sprintfCtop = %d, left = %d, right = %d, bottom = %d',top, ... left, right, bottom) log_message(status); log_message('motion vector out of bounds in Cb block') keyboard return end

% make up 16x24 macroblock with 16x16 y, 8x8 v & 8x8 u mcp = [mcp_y [mcp_v;mcp_u] ] ; function return_value = bytealigned() % function return_value = bytealigned() % % bytealigned will set return_value = 1 if the current position is % on a byte boundary, that is the next bit in the bitstream is the % first bit in a byte. It will set return_value = 0 otherwise. % % read_bitstream must be called first to initialise 1=1 global I

% use the modulus function to test for any remainder when the current % position in the bitstream (I) is divided by 8 and if no remainder, then % the current position is moved forward one bit to a byte boundary % i.e. byte boundaries at bits 1, 9, 17, 25 ... etc. if (I == 1) return_value = 1; % at byte boundary 1 elseif ~(modulo(I,8)) get_bits(l); % move to byte boundary 9, 17, 25 ... etc return_value = 1; else return_value = 0; % not at byte boundary end function sequential = check_order (buffer_index)

% can't be sequential if only 1 value in buffer if length(buffer_index) == 1, sequential = 0; return, end index = [ ] ; for i = 1:length(buffer_index) index = [index buffer_index(i)]; % add next pic number index = sort(index); % into ascending order sequential = 1; % initialise that picture order sequential for writing to file

% check if difference between consecutive pic numbers is > 1, and % exit loop if it is (i.e. not sequential to write to file) if length(index) > 1 for i = 2:length(index) diff = index(i) - index(i-1); if diff 1, sequential = 0; break, end % exit loop end end end return if sequential disp('sequential, pause'), pause else disp('NOT sequential, pause'), pause end %functio n buffer = cod.e_ac (q coeffs) % function buffer = code_ac (q_coeffs) % % description: % Returns an array (buffer) of vie coded AC coefficients (DC coefficient % is ignored) with the end of block code [1 0] last. The length of % the buffer will vary. It's size can be determined by the matlab % command: length(buffer). % % The array is generated as follows: % 1. put the input block into zig-zag scan sequence % 2. compute run and level for each non-zero element in the block % 3. get the vie code corresponding to the (run,level) combination % 4. concatenate all vie codes and put in the buffer % 5. add the end of block code [1 0] as the last two elements of % the buffer % % input: % q_coeffs : 8x8 integer matrix of dct coefficients % : each element is assumed to be in the range % -255 to 255 % % output: % buffer : variable length array of I's and O's in vlclbf format % error message : if q_coeffs not 8x8 matrix % % John Bateman 17/5/95

buffer = zeros (1,1000); buffer_ptr =1; % pointer to next bit in the buffer

% arrange coeffs in zig-zag scan sequence zz = zig zag(q coeffs);

% find indices of all non-zero AC coeffs in zz % note DC coefficient is ignored (set to zero) because only want to code % AC coeffs here zz(l) = 0; nz_indices = find(zz');

% compute run (of zeros), get level, get code & send code to buffer n = length(nz_indices); prev = 1; for i = l:n curr = nz_indices(i); run = curr - (prev + 1); prev = curr; level = zz(curr); dct_coeff_next = get_dct_vlc(run,level); code_length = length(dct_coeff_next); buffer(buffer_ptr:(buffer_ptr + code_length - 1)) = dct_coeff_next; buffer_ptr = buffer_ptr + code_length; end

% add end of block code (note buffer_ptr points to (last+l)th element) end_of_block = [1 0]; buffer(bufferj)tr:(buffer_ptr + 1)) = end_of_block;

% keep only the used part of the buffer buffer = buffer(1:(buffer_ptr + 1)); function code_b_mblock (curr_mblock, mcp, mb_addr, addr_inc, quantizer_scale, % mc_mode, mv_diff_fwd, mv_diff_bwd) % John Bateman

global MCP_FWD motion compensated prediction global MV_FWD motion vector array global MCP_BWD motion compensated prediction global MV_BWD motion vector array global FORWARD_F used in coding motion vectors global FORWARD_F_SIZE used in coding motion vectors global BACKWARD_F used in coding motion vectors global BACKWARD_F_SIZE used in coding motion vectors global RECPIC global RECON_PIC

% code luminance blocks and determine coded block pattern % note memory is preallocated to mblock_buffer after each iteration % to minimise execution time r = [1 8;1 8;9 16;9 16;1 8;9 16]; % block rows c = [1 8;9 16;1 8;9 16;17 24;17 24]; % block cols block = zeros (8,8); rec_mblock = zeros(16,24); mblock_buffer = []; % run length coded dct coeffs diff_mblock = curr_mblock - mcp; for i = 1:6 block = diff_mblock (r(i,1):r(i,2), c(i,1):c(i,2)); [rec_mblock(r(i,1) :r(i,2), c(i,1) : c (i,2)) , cbp(i), buffer] - ... code_ni_block (block, guantizer_scale); temp = mblock_buffer; mblock_buffer = zeros(1,length(mblock_buffer)+length(buffer)); mblock_buffer = [temp buffer]; end %rec_mblock ,disp('pause'), pause

% code mb_addr_inc %fprintf('addr_inc = %d\n', addr_inc); [mb_addr_inc, mai_err] = get_addr_inc_vlc (addr_inc); %% fprintf('addr_inc vie = ');% fprintf('%d',mb_addr_inc);% fprintf('\n');

% determine macroblock type and associated flag bits if mc_mode == 1 % interpolative if ~cbp macroblock_type = [10] % pred-i val = 2; len = 2; else % decimal val, code length macroblock_type = [11] val = 3; len = 2; % decimapred-ilc val, code length end elseif mc_mode == 2 % backward if ~cbp macroblock_type [0 1 0] % pred-b val = 2; len = 3 else % decimal val, code length macroblock_type [0 1 1] val = 3; len = 3 % decimapred-blc val, code length end elseif mc_mode == 3 % forward if ~cbp macroblock_type [0 0 1 0];% pred-f val = 2; len = 4 % decimal val, code length else macroblock_tYpe = [0 0 1 1];% pred-fc val = 3; len =4; % decimal val, code length end else log_message('Invalid motion compensation mode (mc_mode = 1,2 or 3)') end [flags, tYpe_err] = get_b_type (val,len); % get b macroblock type flags if tYpe_err log_message('bit overrun decoding macroblock type') err = 1; return; end macroblock_quant = flags(1) motion_forward = flags(2) motion_backward = flags(3) macroblock_pattern = flags(4) macroblock_intra = flags(5) macroblock_type = decimal_to_uimsbf (val, len); % fprintf('macroblock_tYpe = ');%fprintf('%d', macroblock_type);%fprintf('\n

% code quantizer scale if macroblock_quant log_message('coding quantizer scale not implemented'); else quantizer_scale = []; end

% code forward motion vectors % note: mv_fwd_vlc includes all horiz & vert forward motion information if motion_forward % fprintf('forward diff motion vectors (%d,%d)\n',mv_diff_fwd); mv_fwd_vlc = code_mv (mv_diff_fwd, FORWARD_F, FORWARD_F_SIZE); % fprintf('mv_fwd_vlc = ');% fprintf('%d',mv_fwd_vlc);% fprintf('\n'); else mv_fwd_vlc = []; end

% code backward motion vectors % note: mv_bwd_vlc includes all horiz & vert backward motion information if motion_backward % fprintf('backward diff motion vectors (%d,%d)\n',mv_diff_bwd); mv_bwd_vlc = code_mv (mv_diff_bwd, BACKWARD_F, BACKWARD_F_SIZE); % fprintf('mv_bwd_vlc = ');% fprintf('%d',mv_bwd_vlc);% fprintf('\n'); else mv_bwd_vlc = []; end

% code cbp if macroblock_pattern % fprintf('cbp = ');% fprintf('%d',cbp); % fprintf('\n'); cbp = uimsbf_to_decimal(cbp); % chrominance blocks not coded [coded_block_pattern, cbp_err] = get_cbp_vlc (cbp); % fprintf('cbp vie = ');% fprintf('%d',coded_block_pattern);% fprintf('\n') else coded_block_pattern = []; end

% make macroblock header and write to bitstream len = length(mb_addr_inc) + length(macroblock_tYpe) + ... length(quantizer_scale) + length(mv_fwd_vlc) + length(mv_bwd_vlc) + ... length (coded_block__pattern) + length (mblock_buffer) ; buffer = zeros(1,len); buffer = [mb_addr_inc macroblock_type quantizer_scale ... mv_fwd_vlc mv_bwd_vlc coded_block_pattern mblock_buffer]; bYtes_written = write_to_bitstream(buffer) ; % update future picture store from reconstructed macroblock and mcp if ~cbp % no blocks coded mblock = mcp; else % blocks coded according to cbp mblock = rec_mblock + mcp; end [top, left] = get_top_left (mb_addr); update_pic_store ('RECON_PIC', mblock, top, left); return % debugging check [top, left] = get_tl(mb_addr); RECPIC(top:top+15,left:left+23) = rec_mblock; %functio n code b picture (current pic, temporal_reference) % % John Bateman % % global PICTURE_START_CODE global HORIZ_SIZE global VERT_SIZE global MV_FWD MV_BWD global FULL_PEL_FWD FORWARD_F_CODE FORWARD_F FORWARD_F_SIZE global FULL_PEL_BWD BACKWARD_F_CODE BACKWARD_F BACKWARD_F_SIZE global MCP_FWD MCP_BWD MCP_INTERP global MSE_FWD MSE_BWD MSE_INTERP global PREVIOUS_PIC FUTURE_PIC log_message(['start coding picture B' int2str(temporal_reference)]); if temporal_reference < 0 | temporal_reference > 1023 log_message(' error in function code_p_picture: invalid temporal_reference') end

% motion estimation (to half pel accuracy) & motion compensation % note that this determines MCP_FWD_DIFF, MV_FWD and the various f-codes range =7; % search range -7.5 to + 7.5 % following 3 lines commented out for testing -> load results from file t=cputime; motion_estim_fwd (current_pic, PREVIOUS_PIC, range); e=cputime-t; fprintf('motion fwd = %.4f minutes\n\n',e/60); t=cputime; motion_estim_bwd (current_pic, FUTURE_PIC, range); e=cputime-t; fprintf('motion bwd = %.4f minutes\n\n',e/60); t=cputime; motion_estim_interp; e=cputime-t; fprintf('motion interp = %.4f minutes\n\n',e/60); % load motion estimation variables (motion vectors, motion compensated % predictions, and the various "f-codes" for interpolated, backward % and foirward motion compensation from files load motion_vectors_fwd load motion_comp_fwd load motion_vectors_bwd load motion_comp_bwd load motion_comp_interp

% make picture header temporal_reference = decimal_to_uimsbf (temporal_reference, 10); % 10 bits picture_coding_tYpe =[011]; % B picture vbv_delay =[1111 1111 1111 1111]; %variable bitrate % FULL_PEL_FWD, FULL_PEL_BWD loaded from file above FORWARD_F_CODE = decimal_to_uimsbf(FORWARD_F_CODE,3); BACKWARD_F_CODE = decimal_to_uimsbf(BACKWARD_F_CODE,3); extra_bit_picture = [0]; % no extra picture information following header = [PICTURE_START_CODE temporal_reference picture_coding_type ... vbv_delay FULL_PEL_FWD FORWARD_F_CODE FULL_PEL_BWD ... BACKWARD_F_CODE extra_bit_picture]; write_to_bitstream(header);

% divide current picture into slices [slices, slice_length] = get_slices(current_pic); % m is number of 16x16 macroblocks in the picture m = (HORIZ_SIZE * VERT_SIZE) /(16*16) ; no_of_slices = m / slice_length; %no_of_slices = 2; slice_length = 6; % code slices modes = [ ] ; last_intra_addr = -1; % initialise addr of last intra coded mblock for i = 1:no_of_slices slice = slices((i-1)*16+1:i*16 ,:) ; [last_intra_addr, new_slice] = code_b_slice (slice, i, slice_length, last_intra_addr); modes = [modes; new_slice]; end show_modes (modes) save modes modes function [last_intra_addr, modes] = code_b_slice (slice, svp, slice_length, last_intra_addr) global MB_WIDTH global START_CODE global MV_FWD % forward motion vector array global MV_BWD % backward motion vector array global MCP_FWD global MCP_BWD global MCP_INTERP global HORIZ_SIZE global RECON PIC

% code slice header %log_message(['* * coding slice ' int2str(svp)]) ; slice_start_code = [START_CODE decimal_to_uimsbf(svp,8)]; quantizer_scale 8; % set quantizer_scale (can be reset in macroblock) extra_bit_slice [0]; % no extra slice information following header zeros(1,38); header [slice_start_code decimal_to_uimsbf(quantizer_scale, 5) extra_bit_slice]; write_to_bitstream(header);

code macroblocks in slice mv_ fwd prev = [0 0]; % fwd predition vector is zero at slice start mv bwd prev = [00]; % bwd predition vector is zero at slice start dc_predictor = [1024 1024 1024]; % initialise for slice start prev_mb_addr = (svp - 1)*MB_WIDTH - 1; % prev_mb_addr from previous slice addr_inc = 1; % initialise address increment modes = zeros(1,length(slice_length)); curr mblock = zeros(16,24);

% check if last macroblock in previous slice intra coded if last_intra_addr == prev_mb_addr prev_mb_intra - 1; % initialise flag indicating last mb intra coded else prev_mb_intra =0; % initialise flag indicating last mb NOT intra coded end

h = waitbar(0,['coding slice ',int2str(svp)]); for i = 1:slice_length waitbar(i/slice_length) % progress feedback intra = 0; skipped = 0; % initialise variables left = (i-l)*16+l; % luminance left col 1 = left - 8*((left - 1)/16) + HORIZ_SIZE; % chrominance left col curr_mblock(:,1:16) = slice(:,left:left+15); % current luminance mblock curr_mblock(:,17:24) = slice(:,1:1+7); % chrom blocks mb addr = prev_mb_addr + i; % current address %fprintf(['\n** coding macroblock ' int2str(mb_addr)]);fprintf('\n')

% check for skipped macroblocks if i > 1 & i < slice_length & ~prev_mb_intra t^cputime; skipped = test_skipped (curr_mblock, mb_addr, mc_mode, ... quantizer_scale, mv_fwd_prev, mv_bwd__prev) ; %if svp==l,e=cputime-t;fprintf('test_skipped = %.4f s\n\n',e);end end

% find best mc mode, and compare with intra if ~skipped % not forced update intra or skipped t=cputime; [mc_mode, mcp] = select_mc_mode (mb_addr); % find best mc mode %if svp==l,e=cputime-t;fprintf('select_mc_mode = %.4f s\n\n',e);end % intra/non-intra coding decision t=cputime; [varc, vard] = find_variance (curr_mblock(:,1:16), mcp(:,1:16)); % var of y %if svp==l,e=cputime-t;fprintf('find_variance = %.4f s\n\n',e);end if varc < vard intra = 1; %disp('** intra mode selected ') else intra = 0; %disp('non-intra mode selected') end end

% code intra/non-intra macroblock if intra % disp('** intra mode') intra_d =[0001 1]; % vie for intra-d macroblock t=cputime; [dc_predictor, rec_mblock] = code_i_mblock (curr_mblock, ... intra_d, mb_addr, addr_inc, dc_predictor, quantizer_scale); %if svp==l,e=cputime-t;fprintf('code_i_mblock = %.4f s\n\n',e);end addr_inc = 1 ; ) coded macroblock mv_fwd_prev = [0 0]; ) reset fwd predition vector mv_bwd_prev = [0 0]; ) reset bwd predition vector prev_mb_intra = 1; i set prev intra flag las t_intra_addr= mb_addr; > reset last intra coded address [top, left] = get_top_left(mb_addr); update_pic_store ('RECON_PIC', rec_mblock, top, left); else if -skipped %fprintf('motion comp mode %d\n', mc_mode); % determine diff motion vectors & update prediction vectors as reqd %fprintf('mv_fwd = [%d %d]\n',MV_FWD(mb_addr+l, : ) ) ; %fprintf('mv_bwd = [%d %d]\n',MV_BWD(mb_addr+l,:)); %fprintf('mv_fwd_prev = [%d %d]\n',mv_fwd_prev); %fprintf ( 'mv_bwd_prev == [%d %d] \n' ,mv_bwd_prev) ;

if mc_mode == 1 % interpolative mv_diff_fwd = MV_FWD (mb_addr+l, :) - mv_fwd_prev; % DPCM mv_diff_bwd = MV_BWD (mb_addr+l, :) - mv_bwd_prev; % DPCM mv_fwd_prev = MV_FWD (mb_addr+l, :); % update fwd predictor mv_bwd_prev = MV_BWD (mb_addr+l, :); % update bwd predictor elseif mc_mode == 2 % backward mv_diff_bwd = MV_BWD (mb_addr+l, :) - mv_bwd_prev; mv_bwd_prev = MV_BWD (mb_addr+l, :); % update bwd predictor mv_diff_fwd = []; % no fwd diff motion vector elseif mc_mode == 3 % forward mv_diff_fwd = MV_FWD (mb_addr+l, :) - mv_fwd_prev; mv_fwd_prev = MV_FWD (mb_addr+l, :); % update fwd predictor mv_diff_bwd = []; % no bwd diff motion vector end % form difference macroblock (mcp obtained from select_mc_mode earlier) diff_mblock = curr_mblock - mcp;% form diff macroblock t=cputime; code_b_mblock (curr_mblock, mcp, mb_addr, addr_inc, quantizer_scale, .. mc_mode, mv_diff_fwd, mv_diff_bwd); %if svp==l,e=cputime-t;fprintf('code_b_mblock = %.4f s\n\n',e);end addr_inc = 1; % coded macroblock else %disp('** skipped macroblock') addr_inc = addr_inc + 1; % skipped macroblock end dc_predictor = [1024 1024 1024]; % reset dc predictors prev_mb_intra = 0; % reset prev intra flag end

% following code keeps track of mc modes for each macroblock %fprintf(' mv_fwd_prev = [%d %d] \n' ,mv_fwd__prev) ; %fprintf(' mv_bwd_prev = [%d %d]\n',mv_bwd_prev); if skipped modes(i) - 4; elseif intra modes(i) = 5; else modes(i) = mc_mode; end end close(h) %functio n [dc_size, dc_add_code, err] = code_chrom_dc (difference)

% % description: % Uses Tables 2-D.12 and 2-D.13 to determine dc_size and dc_add_code % for a given chrominance difference value. % % input: % difference : difference between current and previous dc chrominance % block values (integer) % : integer in range 0<=abs(difference)<=255 % % output: % dc_size : vie for size in bits of difference % dc_add_code : vie for difference value % % eg. chrominance difference of 10 (ten) would return % dc_size = [1110] and dc_add_code = ? % % John Bateman 9/4/95, 18/5/95, 14/6/95 % % % determine size and the vie code for size (and dc_add_code if difference = 0) if abs(difference) == 0 size =0; % size in bits of dc_add_code dc_size =[0 0]; % vie code for this size dc_add_code = []; % additional code for this size return; % exit function elseif abs(difference) == 1 size = 1; dc_size =[0 1]; % vie code for this size elseif abs(difference) >= 2 & abs(difference) <= 3 size = 2; dc_size = [1 0] ; elseif abs(difference) >= 4 & abs(difference) <= 7 size = 3; dc_size = [1 1 0] ; elseif abs(difference) >= 8 & abs(difference) <= 15 size = 4; dc_size = [1 1 1 0] ; elseif abs(difference) >= 16 & abs(difference) <= 31 size = 5; dc_size = [1111 0]; elseif abs(difference) >= 32 & abs(difference) <= 63 size = 6; dc_size = [1111 10]; elseif abs(difference) >= 64 & abs(difference) <= 127 size = 7; dc_size =[1111 110]; elseif abs(difference) >= 128 & abs(difference) <= 255 size = 8; dc_size =[1111 1110]; else log_message('difference level outside valid range [0,255]') err = 1, return end

% determine the vie code for de_add_eode for non-zero difference values if difference > 0 dc_add_code = deeimal_to_uimsbf (difference,size); % positive value else % take I's complement to generate code for negative value dc_add_eode = ~deeimal_to_uimsbf(abs(difference),size); % negative value end function buffer = code_coeffs (q_coeffs)

% description: % Returns an array (buffer) of vie coded coefficients with the end of block % code [1 0] last. % % The array is generated as follows: % 1. put the input block into zig-zag scan sequence % 2. compute run and level for each non-zero element in the block % 3. get the vie code corresponding to the (run,level) combination % 4. concatenate all vie codes and put in the buffer % 5. add the end of block code [1 0] as the last two elements of % the buffer % % input: % q_coeffs : 8x8 integer matrix of quantised dct coefficients % : each element is assumed to be in the range % -255 to 255 % % output: % buffer : variable length array of I's and O's in vlclbf format % % John Bateman 17/5/95 % global total %% buffer = zeros(1,1000); buffer_ptr =1; % pointer to next bit in the buffer

% arrange coeffs in zig-zag scan sequence zz = zig_zag(q coeffs);

% code first coeff first = zz(1); if first 0 if first == 1 vlc_first = [1 0]; elseif first == -1 vlc_first = [1 1]; else vlc_first = get_dct_vlc (0,first); % get vie for (run, level) end 1 = length(vlc_first); %% %%fprintf(l, 'run=%d lev=%d len=%d vie =', 0,first, 1); %% %%fprintf(1,'%d', vle_first);fprintf(1,'\n'); %% total = total + length(vle_first); %% zz(l) =0; % ignore first coeff in subsequent steps prev =1; % go to next non-zero coeff else prev =0; % first non-zero coeff end

% find indices of all non-zero coeffs in zz nz_indiees = find(zz');

% compute run (of zeros), get level, get code & send code to buffer n = length(nz_indices); for i = l:n curr = nz_indices(i); run = curr - (prev + 1); prev = curr; level = zz(curr); det_coeff_next = get_det_vle(run,level); eode_length = length(det_eoeff_next); buffer(buffer_ptr:(buffer_ptr + eode_length - 1)) = dct_coeff_next; buffer_ptr = buffer_ptr + code_length; %%fprintf(l, 'run=%2d lev=%3d len=%2d vie =', run,level, code_length); %% %%fprintf(1,'%d', dct_coeff_next);fprintf(1,'\n'); %% %%total = total + code_length; %% end

% add end of block code {note buffer_ptr points to (last+l)th element) end_of_block = [1 0]; %%total = total +2;fprintf(1,'\ntotal bits = %d\n\ntotal); %% buffer (buff er_ptr: (buff er_ptr + i) ) zz end_of_block;

% keep only the used part of the buffer buffer = buffer(1:(buffer_ptr + 1)); buffer = [vlc_first buffer]; %functio n [dc_size, dc_add_code, err] = code_dc_uv (difference) % % % description: % Uses Tables 2-D.12 and 2-D.13 to determine dc_size and dc_add_code % for a given chrominance difference value. % % input: % difference : difference between current and previous dc chrominance % block values (integer) % : integer in range 0<=abs(difference)<=255 % % output: % dc_size : vie for size in bits of difference % dc_add_code : vie for difference value % % eg. chrominance difference of 10 (ten) would return % dc_size = [1110] and dc_add_code = ? % % John Bateman 9/4/95, 18/5/95, 14/6/95 % % determine size and the vie code for size (and dc_add_code if difference = 0) if abs(difference) == 0 size =0; % size in bits of dc_add_code dc_size =[0 0]; % vie code for this size dc_add_code = []; % additional code for this size return; % exit function elseif abs(difference) == 1 size = 1; dc_size = [0 1]; % vie code for this size elseif abs(difference) >= 2 & abs(difference) <= 3 size = 2; de_size = [10]; elseif abs(difference) >= 4 & abs(difference) <= 7 size = 3; dc_size = [1 1 0] ; elseif abs(difference) >= 8 & abs(difference) <= 15 size = 4; dc_size = [1 1 1 0] ; elseif abs(difference) >= 16 & abs(difference) <= 31 size = 5; dc_size = [1111 0] ; elseif abs(difference) >= 32 & abs(difference) <= 63 size = 6; dc_size =[1111 10]; elseif abs(difference) >= 64 & abs(difference) <= 127 size = 7; dc_size = [1111 110]; elseif abs(difference) >= 128 & abs(difference) <= 255 size = 8; de_size =[1111 1110]; else log_message('difference level outside valid range [0,255]') err = 1, return end

% determine the vie code for dc_add_code for non-zero difference values if difference > 0 dc_add_code = deeimal_to_uimsbf (differenee,size); % positive value else % take I's complement to generate code for negative value dc_add_code = ~decimal_to_uimsbf(abs(differenee),size); % negative value end %functio n [dc_size, dc_add_code, err] = code_dc_7 (difference) % function [dc_size, dc_add_code] = code_dc_^ (difference) % % description: % Uses Tables 2-D.12 and 2-D.13 to determine dc_size and dc_add_code % for a given luminance difference value. % % input: % difference : difference between current and previous dc luminance % block values (integer) % : integer in range 0<=abs(difference)<=255 % % output: % dc s ize : vie for size in bits of difference % dc_add_code : vie for difference value % % eg. luminance difference of 10 (ten) would return % dc_size = [110] and dc_add_code = [10 10] % % Jolnn Bateman 9/4/95, 18/5/95, 14/6/95 % determine size and tlie vie code for size (and dc_add_code if difference = 0) if abs(difference) == 0 % size = 0 dc_size =[100]; % vie code for tliis size dc_add_code = []; % additional code for tliis size return; % exit function elseif abs(difference) == 1 size = 1; dc_size =[0 0]; % vie code for this size elseif abs(difference) >= 2 & abs(difference) <= 3 size = 2; dc_size = [01]; elseif abs(difference) >= 4 & abs(difference) <= 7 size = 3; dc_size = [1 0 1] ; elseif abs(difference) >= 8 & abs(difference) <= 15 size = 4; dc_size = [1 1 0] ; elseif abs(difference) >= 16 & abs(difference) <= 31 size = 5; dc_size = [1 1 1 0] ; elseif abs(difference) >= 32 & abs(difference) <= 63 size = 6; dc_size = [1111 0]; elseif abs(difference) >= 64 & abs(difference) <= 127 size = 7; dc_size = [1111 10]; elseif abs(difference) >= 128 & abs(difference) <= 255 size = 8; dc_size = [1111 110]; else log_message('difference level outside valid range [0,255]') err = 1, return end

% determine the vie code for dc_add_code for non-zero difference values if difference > 0 dc_add_code = decimal_to_uimsbf (difference,size); % positive value else % talce I's complement to generate code for negative value dc__add_code = ~decimal_to_uimsbf(abs(difference),size); % negative value end function code_gop (gop, fid_in)

% set global variables global GROUP_START_CODE global HORIZ_SIZE global VERT_SIZE global RECON_PIC I global PREVIOUS_PIC global FUTURE_PIC log_message('start gop');

% set codes drop_frame_flag = 0; % picture rate NOT 29.97 Hz time_code_hours decimal_to_uimsbf(0,5) % range 0-23, 5 bits time_code_minutes decimal_to_uimsbf(0,6) % range 0-59, 6 bits marker_bit 1; time_code_seconds decimal_to_uimsbf (0,6) % range 0-59, 6 bits time_code_pictures decimal_to_uimsbf (0,6) % range 0-59, 6 bits time_code = [drop_frame_flag time_code_hours time_code_minutes .. marker_bit time_code_seconds time code pictures]; closed_gop =1; % gop coded without motion vectors to previous gop broken_link =0; % no I or P-pictures edited out header = zeros(59); header = [GROUP_START_CODE time_code closed_gop broken_link]; write to bitstream (header); % write header

% code first picture as I picture [r,c] = size(gop); pic_tYpe = gop(l,l); pic_no = gop(l,2); if pic_type 1, error('first picture in coding order always I'), end [Y,U,V] = uvc_read_frame(fid_in, pic_no); ; read picture from uvc file if size(y) [VERT_SIZE H0RIZ_SIZE+8], error('incorrect pic size')/ end if size(u) [VERT_SIZE/2 (H0RIZ_SIZE+8)/2] error('incorrect pic size'),end if size(v) [VERT_SIZE/2 (H0RIZ_SIZE+8)/2] error('incorrect pic size'),end y = y(:,5:356) ;u=u(:,3:178) ;v=v(: ,3 :178) ; % keep significant pel area picture = yuv2picture (y,u,v); I_ORIG = picture;

%display yuv_.pic (picture) ; temporal_reference = modulo(pic_no, 1024); % modulo 1024 %fprintf('coding I%d pause...',pic_no),pause !ls -1 *.bin t=cputime; code_i__picture (picture, temporal_reference) ; e=cputime-t ; I picture fprintf('picture coding time = %f minutes\n\n',e/60) ;

I_RECON = RECON_PIC; save I I RECON I OHIO if r == 1, return, end exit if only 1 picture in gop

% load previous & future pic buffers for MCP purposes PREVIOUS_PIC = RECON_PIC; % reconstructed I picture FUTURE_PIC = []; pic_array = pic_no; % pic_no tracks coded picture numbers prev_pic_type= 1; % I picture p = pic_no;f = [];

% code remaining pictures in gop for i = 2:r % get next picture pic_type = gop(1,1); pic_no = gop(i,2); [y,u,v] = uvc_read_frame(fid_in, pic_no); % read picture from uvc file if size(Y) — [VERT_SIZE H0RIZ_SIZE+8], error('incorrect pic size'), end if size(u) [VERT_SIZE/2 (H0RIZ_SIZE+8)/2], error('incorrect pic size'),end if size(v) [VERT_SIZE/2 (H0RIZ_SIZE+8)/2], error('incorrect pic size'),end y = y(:,5:356);u=u{:,3:178)7v=v(:,3:178); % keep significant pel area picture = yuv2picture (y,u,v); %display_y^uv_:pic (picture) ; temporal_reference = modulo{pic_no, 1024); % modulo 1024

% code according to type t=cputime; if pic_type == 1 % I picture %fprintf('coding I%d pause...',pic_no), p , f, pause code_i_picture (picture, temporal_reference); e=cputime-t; !ls -1 *.bin I_RECON = RECON_PIC; save I_RECON I_RECON elseif pic_type == 2 % P picture %fprintf('coding P%d pausepic_no), p , f,pause P_ORIG = picture;

code_p_picture (picture, temporal_reference) ; e=cputime-t; lis -1 *.bin P_RECON = RECON_PIC; save P P_RECON P_ORIG elseif pic_type == 3 % B picture %fprintf('coding B%d pausepic_no), p , f, pause B_ORIG = picture;

code_b_picture (picture, temporal_reference); e=cputime-t; lis -1 *.bin B_RECON = RECON_PIC; save B B_RECON B_ORIG else error('Invalid picture type') end fprintf('picture coding time = %f minutes\n\n', e/60 ) ;

% reload previous & future picture buffers as required by gop structure pic_array = [pic_array pic_no]; % add picture no to array sequential = check_order(pic_array) ; % check if coded pic nos are sequential if (pic_type == 1 | pic_type == 2) & ... (prev_pic_type == 1 | prev_:pic_type ==2) & sequential % consecutive I/P pictures eg II IP PP PI PREVIOUS_PIC = RECON_PIC; p = pic_no; f = []; FUTURE_PIC = []; pic_array = pic_no; elseif sequential % all B pictures coded or IPI, previous = future, discard future buffer PREVIOUS_PIC = FUTURE_PIC; p = f; f = []; FUTURE_PIC = []; pic_array = future pic no; % reload array (earlier pics now redundant) elseif pic_type == 1 | pic_type == 2 % I or P which don't meet above conditions FUTURE_PIC = RECON_PIC; f = pic_no; future_pic_no = pic_no; end prev__pic_type = pic_type; % keep track of last picture type coded end log_message('end gop'); function [rec_block, buffer, dc_uv, err] = code_i_block_uv (block, dc_uv,

quantizer_scale) global IQ_MATRIX

% do 8x8 dct %coeffs = dct_2d(block); coeffs = dct8x8(block); % use fast C DCT

% get DC dct coefficient dc_value = coeffs(1,1); %fprintf('dc_value = %f \n',dc_value);

% clip dc value to [0,2048] if required if dc_value < 0 dc_value =0; % clip to 0 elseif dc_value > 2048 dc_value = 2048; % clip to 2048 end

% quantize and code DC DPCM difference difference = dc_value - dc_uv; % DPCM difference = round(difference/8); % quantize difference [dc_size_lum, dc_diff, dc_err] = code_dc_uv (difference); if dc_err, err = 1, return, end dc_buffer = [dc_size_lum dc_diff]; % keep vie codes %fprintf('dc_size_lum = ');fprintf('%d',dc_size_lum);fprintf('\n'); %fprintf('dc_diff = ');fprintf('%d',dc_diff);fprintf('\n');

% update dc predictors difference = difference*8; % inverse quantize difference dc_uv = dc_uv + round(difference); % update u/v dc_predictor %fprintf ( ' new dc__predictor = %f \n',dc_uv);

% quantize dct coeffs using IQ MATRIX and quantizer_scale q coeffs = intra_quant (coeffs, IQ_MATRIX, quantizer_scale);

% code AC coefficients ac_buffer = code_ac (q coeffs);

% decode block (inverse quantize and inverse dct) rec_coeffs = inv_intra_quant (q_coeffs, IQ_MATRIX, quantizer_scale); rec_coef f s (1,1) == dc_uv; % use decoded DPCM dc value %rec_block = inv_dct_2d(rec_coeffs); rec_block = idct8x8(rec_coeffs); % use fast C inverse DCT

% make block buffer buffer = zeros (1, lengtli (dc_buffer) + length (ac_buffer) ) ; buffer = [dc_buffer ac_buffer]; function [rec_block, buffer, dc_y, err] - code_i_block_Y (block, dc_y, quantizer_scale) global IQ_MATRIX

% do 8x8 dot %coeffs = dct_2d(block) ; coeffs = dct8x8(block); % use fast C DCT

% get DC dct coefficient dc_value = coeffs(1,1) ; %fprintf ('dc_value = %f \n' , dc_value)

% clip dc value to [0,2048] if required if dc_value < 0 dc_value = 0; % dip to 0 elseif dc_value > 2 048 dc_value = 2 048; % dip to 2 048 end

% quantize and code DC DPCM difference difference = dc_value - dc__y; % DPCM difference = round(difference/8); % quantize difference [dc_size_lum, dc_diff, dc_err] = code_dc_:/ (difference); if dc_err, err = 1, return, end dc_buffer = [dc_size_lum dc_diff]; % keep vie codes %fprintf('dc_size_lum = ');fprintf('%d',dc_size_lum);fprintf('\n'); %fprintf('dc_diff = ');fprintf('%d',dc_diff);fprintf('\n'); % update dc predictors difference = difference*8; % inverse quantize difference dc_j/ = dc^ + round (difference) ; % update y dc_predictor %fprintf ('new dc_predictor = %f \n',dc_j/);

% quantize dct coeffs using IQ_MATRIX and quantizer_scale q_coeffs = intra_quant (coeffs, IQ_MATRIX, quantizer_scale);

% code AC coefficients ac_buffer = code_ac (q_coeffs);

% decode block (inverse quantize and inverse dct) rec_coeffs = inv_intra_quant (q_coeffs, IQ_MATRIX, quantizer_scale); rec_coeffs(1,1) = dc_Y; % use decoded DPCM dc value %rec_block = inv_dct_2d(rec_coeffs); rec_block = idct8x8(rec_coeffs); % use fast C inverse DCT % make block buffer buffer = zeros (1, lengt]i(dc_buffer) + lengtli (ac_buf fer) ) ; buffer = [dc_buffer ac_buffer]; function [dc_predictor, rec_mblock] = code_i_riiblock (macroblock, . - - nib_type, mb_addr, addr_inc, dc_predictor, quantizer_scale)

% for intra coded macroblocks (I, P and B) code_block_pattern is not Tx, % but assumed to have the value 63 , i.e. all blocks are coded global RECPIC

% code mb_addr_inc %fprintf('addr_inc = %d\n', addr_inc); [mb_addr_inc, mai_err] = get_addr_inc_vlc (addr_inc); %fprintf('addr_inc vie = ') ;fprintf('%d',mb_addr_inc) ;fprintf('\n') ; %fprintf('mb_tYpe = ');fprintf('%d',mb_tYpe);fprintf('\n');

% make macroblock header header = [mb_addr_inc mb_tYpe];

% code luminance blocks % note memory is preallocated to mblock_buffer after each iteration % to minimise execution time r = [1 8;1 8;9 16;9 16]; % block rows c = [1 879 16;1 8;9 16]; % block cols block = zeros (8,8); rec_mblock = zeros(16,24); mblock_buffer = []; % run length coded dct coeffs dc_:y = dc_predictor (1) ; % luminance dc-predictor for i = 1:4 block = macroblock (r(i,l):r(i,2), c(i,l):c(i,2)); [rec_mblock(r(i,1):r(i,2), c(i,1):c(i,2)), buffer, dc_j/, err] = ... code i block y (block, dc_y, quantizer_scale); if err, log_message('error coding i block'), end temp = mblock_buffer; mblock_buffer = zeros(1,length(mblock_buffer)+length(buffer)); mblock_buffer = [temp buffer]; end dc_predictor(1) = dc_y; % keep dc predictor from last coded luminance block

% code V chrominance block block = macroblock (1:8,17:24); [rec_block_v, buffer_v, dc_predictor(2), err] = ... code_i_block_uv (block, dc_predictor(2), quantizer_scale);

% code u chrominance block block = macroblock (9:16,17:24); [rec_block_u, buffer_u, dc predictor(3), err] = ... code_i_block_uv (block, dc_predictor(3), quantizer_scale);

% write data to bitstream len = length(header) + length(mblock_buffer) + length(buffer_v) + ... length(buffer_u); buffer = zeros(1,len); buffer = [header mblock_buffer buffer_v buffer_u]; bytes_written = write_to_bitstream(buffer);

% form reconstructed macroblock rec_mblock(:,17:24) =[rec_block_v;rec_block_u]; %functio n code_i_picture (current_pic, temporal_reference) % John Bateman 21/6/95 global PICTURE_START_CODE global HORIZ_SIZE global VERT_SIZE global PREVIOUS_PIC log_message(['start coding picture I' int2str(temporal_reference)]); if temporal_reference < 0 | temporal_reference > 1023 log_message(' error in function code_p_picture: invalid temporal_reference') end if temporal_reference < 0 | temporal_reference>1023 log_message(' error in function code_i_picture: invalid picture size or tempor end

% make picture header temporal_reference = decimal_to_uimsbf {temporal_reference, 10); % 10 bits picture_coding_type = [001];%! picture vbv_delay _ =[1111 1111 llll 1111]; %variable bitrate extra_bit_picture = [O]; % no extra picture information following header = [PICTURE_START_CODE temporal_reference picture_coding_tYpe vbv_delaY extra_bit_picture]; write_to_bitstream (header) ;

% divide the picture into slices [slices, slice_length] = get_slices(current_pic);

% m is number of 16x16 macroblocks in the picture m= (HORIZ_SIZE * VERT_SIZE)/(16*16); no_of_slices = m / slice_length; %no_of_slices = 2; slice_length = 6;

% code slices for i = 1:no_of_slices slice = slices((i-1)*16+1:i*16,:); code_i_slice (slice, i, slice_length); end function code_i_slice (slice, svp, slice_length; global MB_WIDTH global START_CODE global HORIZ_SIZE

% code slice header %log_message(['** coding slice ' int2str(svp)]); slice_start_code = [START_CODE decimal_to_uimsbf(svp,8)]; quantizer_scale = 8; % set quantizer_scale (can be reset in macroblock) extra_bit_slice = [0]; % no extra slice information following header = zeros(1,38) ; header = [slice_start_code decimal_to_uimsbf(quantizer_scale, 5) extra_bit_slice]; write_to_bitstream(header);

% code macroblocks in slice addr_inc - 1; % initialise address increment prev_mb_addr = (svp - 1)*MB_WIDTH - 1; % get mb_addr from previous slice curr_mblock = zeros(16,24); intra_d =1; % vie for intra-d macroblock dc_predictor = [1024 1024 1024]; h = waitbar(0,['coding slice ',int2str(svp)]); for i = 1:slice_length waitbar(i/slice_length) % progress feedback mb_addr = prev_mb_addr + i; % current address [top, left] = get_top_left(mb_addr); % luminance top left pel 1 = left - 8*((left - 1)/16) + HORIZ_SIZE; % chrominance left col curr_mblock(:,1:16) = slice(:,left:left+15) % current luminance mblock curr_mblock(:,17:24) = slice(:,1:1+7); % chrom blocks mb_addr = prev_mb_addr + i; % current address % fprintf(['\n** coding macroblock ' int2str(mb_addr)]);fprintf('\n') t = cputime; [dc__predictor, rec_mblock] = code_i_mblock (curr_mblock, intra_d, ... mb_addr, addr_inc, dc_predictor, quantizer_scale); %if svp == 1,e=cputime-t;fprintf('code_i_mblock = %.4f s\n\n',e);end

update_pic_store ('RECON_PIC', rec_mblock, top, left); end close(h) function vie = code_mv (mv_diff, forward_f, forward_f_size)

% function requires "mv_vlc.mat" to be loaded into base workspace and % init_coder_var script executed

% apply modulus (32 * forward_f) to (mvh_diff, mw_diff) if required max_mv = 16 * forward_f - 1; % lower mv limit min_mv = -16 * forward_f; % upper mv limit i = find(mv_diff > max_mv); mv_diff(i) = mv_diff(i) - 32 * forward_f; % too big, subtract modulus i = find{mv_diff < min_mv); mv_diff(i) = mv_diff(i) + 32 * forward_f; % too small, add modulus mvh_diff = mv_diff(l); % horiz diff mv inw_diff = mv_diff(2); % vert diff mv

% find motion_horiz_fwd_code & motion_horiz_fwd_r if forward_f 1 if mvh_diff > 0 % positive numbers mv_quot = fix(mvh_diff / forward_f); motion_horiz_fwd_r = mvh_diff - (mv_quot * forward_f); elseif mvh_diff < 0 % negative numbers mv_quot = floor(mvh_diff / forward_f); % round towards -inf motion_horiz_fwd_r = abs(mvh_diff - (mv_quot * forward_f)); else % zero case mv_quot = 0; motion_horiz_fwd_r = 0; end [motion_horiz_fwd_code, mvh_err] = get_mv_vlc (mv_quot) ; % quotient vie % convert to binary if required %fprintf('mv_quot = %d\n',mv_quot); %fprintf('mv_rem = %d\n',motion_horiz_fwd_r); if forward_f_size ~= 1 % motion_horiz_fwd_r > 1 bit long motion_horiz_fwd_r = ... decimal_to_uimsbf(motion_horiz_fwd_r, forward_f_size); % rem uimsbf end else % motion vectors in range +/- 16 [motion_horiz_fwd_code, mvh_err] = get_mv_vlc (mvh_dif f) ; motion_horiz_fwd_r = []; % no remainder end

%fprintf('motion_horiz_fwd_code = ');fprintf('%d', motion_horiz_fwd_code); %fprintf('\n' ) ; %fprintf ( 'motion_horiz_fwd_r = ' ) ; fprintf ( ' %d' , motion_horiz_fwd_r) ; %fprintf('\n' ) ;

% find motion_vert_fwd_code & motion_vert_fwd_r if forward_f ~= 1 if mw_diff > 0 % positive numbers mv_quot = fix(mw_diff / forward_f) ; motion_vert_fwd_r = mw_diff - (mv_quot * forward_f) ; elseif mw_diff < 0 % negative numbers mv_quot = floor (mw_diff / forward_f) ; % round towards -inf motion_vert_fwd_r = abs(mw_diff - (mv_quot * forward_f) ) ; else % zero case ii^v_quot = 0 ; motion_vert_fwd_r = 0 ; end [motion_vert_fwd_code, mw_err] = get_mv_vlc (mv_quot) ,- % quotient vie % convert to binary if required %fprintf('mv_quot = %d\n',mv_quot); %fprintf('mv_rem = %d\n',motion_vert_fwd_r); if forward_f_size > 1 % motion_vert_fwd_r more than 1 bit long motion_vert_fwd_r = ... decimal_to_uimsbf(motion_vert_fwd_r, forward_f_size); % rem uimsbf end else % motion vectors in range [motion_vert_fwd_code, mw_err] - get_mv_vlc (mw_dif f) ; motion_vert_fwd_r = []; % no remainder end

%fprintf('motion_vert_fwd_code = ');fprintf('%d', motion_vert_fwd_code); %fprintf('\n'); %fprintf('motion_vert_fwd_r = ');fprintf('%d', motion_vert_fwd_r); %fprintf('\n');

% vie is concatenation of vie of signed quotient & fixed length code of % unsigned remainder from (mvh_diff / forward_f) len = length(motion_horiz_fwd_code) + length(motion_horiz_fwd_r) + ... length(motion_vert_fwd_code) + length(motion_vert_fwd_r); vie = zeros(1,len); vie = [motion_horiz_fwd_code motion_horiz_fwd_r motion_vert_fwd_code ... motion_vert_fwd_r]; function [skipped, intra] = code_p_mblock (curr_mblock, mcp, mb_addr, addr_inc, % quantizer_scale, mv_fwd_prev, mv_fwd) % % John Bateman 22/6/95, 24/6/95 I global PREVIOUS_PIC % previous picture store global MCP_FWD % motion compensated prediction global FORWARD_F % used in coding motion vectors global FORWARD_F_SIZE % used in coding motion vectors global MV_FWD % motion vector array global MB_WIDTH % number of macroblocks in slice global RECPIC

% code luminance blocks and determine coded block pattern % note memory is preallocated to mblock_buffer after each iteration % to minimise execution time r = [1 8;1 8;9 16;9 16;1 8;9 16]; % block rows C = [1 8;9 16;1 8;9 16;17 24;17 24]; % block cols block = zeros (8,8); rec_mblock = zeros(16,24) ; mblock_buffer = []; % run length coded dct coeffs diff_mblock = curr_mblock - mop; for i = 1:6 block = diff_mblock (r(i,1) :r(i,2), c (i,1) :c(i,2)) ; [rec_mblock(r(i,1):r(i,2), c(i,l):c(i,2)), cbp(i), buffer] = ... code_ni_block (block, quantizer_scale); temp = mblock_buffer; mblock_buffer = zeros(1,length(mblock_buffer)+length(buffer) ) ; itiblock_buffer = [temp buffer] ; end %rec_mblock , cbp, disp('pause'), pause

% check for Ist/last macroblock in slice (modulo returns 0 element if true) first_or_last = modulo ( [mb_addr, mb_addr+l ] , MB_WIDTH) ;

% check for skipped macroblock and update RECON_PIC if required if ~any(mv_fwd) & -any(cbp) & all(first_or_last) % skip this macroblock (no vie code) if mv = [0 0] & cbp =[000000] % and not first or last macroblock in slice skipped =1; % skipped macroblock intra =0; % coded as non-intra skipped %log_message(['skipped macroblock at macroblock address ' int2str(mb_addr)]); % copy macroblock from previous picture [top, left] = get_top_left (mb_addr); future_mblock = get_YCbCr_mblock (PREVIOUS_PIC, top, left); update__pic_store ( ' RECON_PIC' , future_mblock, top, left); return; elseif ~any(mv_fwd) & -any(cbp) & -all(first_or_last) % First or last macroblock in slice with no motion vector & no cbp. % Because can't code as non-intra (no cbp and can't skip), set flag % for coding as an intra macroblock % disp('cannot code as non-intra -> switching to intra coding') intra = 1; % set intra coding flag (can't code as non-intra) skipped =0; % not skipped return % exit function else % No motion vector, not skipped, cbp exists intra =0; % can code as non-intra skipped =0; % not skipped end

% determine other macroblock types and associated flag bits if ~mv_fwd % no MC %disp('pred-c') macroblock_tYpe =[0 1]; % pred-mc val = 1; len =2; % decimal val, code length else % MC if ~cbp %disp('pred-m') macroblock_type =[0 0 1]; % pred-m val = 1; len = 3; % decimal val, code lengtli else %di sp('pred-mc') macrobloc]c_tYpe = 1; % pred-mc val = 1; len =1; % decimal val, code lengtli end end [flags, tYpe_err] = get_p_type (val,len); % get p macrobloclc type flags if type_err 1 og_mes sage ('bit overrun decoding macroblocJc type') err = 1; return; end macrobloc>c_quant = flags (1) motion_forward = flags(2) motion_bac]<:ward = flags (3) macrobloc]c_pattern = flags (4) macrobloclc_intra = flags (5) macrobloclc_type = decimal_to_uimsbf (val, len) ;

%fprintf ('macrobloc]c_type = '); fprintf ( ' %d' , macrobloc]c_type) ; fprintf { ' \n' ) ;

% code mblocl<:_addr_inc %fprintf('addr_inc = %d\n', addr_inc); [mbloc]c_addr_inc, mai_err] = get_addr_inc_vlc (addr_inc) ; %fprintf('addr_inc vie = ');fprintf('%d',mblock_addr_inc) ;fprintf('\n') ; % code quantizer scale if macroblock_quant disp('coding quantizer scale not implemented'); else quantizer_scale = []; end % code forward motion vectors % note: mv_fwd_vlc includes all lioriz & vert forward motion information if motion_forward mv_diff_fwd = mv_fwd - mv_fwd_prev; %fprintf('forward diff motion vectors (%d,%d)\n',mv_diff_fwd); mv_fwd_vlc = code_mv {mv_diff_fwd, FORWARD_F, FORWARD_F_SIZE); %fprintf('mv_fwd_vlc = ');fprintf('%d',mv_fwd_vlc);fprintf('\n'); else mv_fwd_vlc = []; end mv_bwd_vlc = [];

% code cbp if macroblock__pattern %fprintf('cbp = ');fprintf('%d',cbp); fprintf{'\n'); cbp = uimsbf_to_decimal (cbp) ; % clirominance blocks not coded [coded_block_pattern, cbp_err] = get_cbp_vlc (cbp); %fprintf('cbp vie = ');fprintf('%d',coded_block_:pattern) ;fprintf('\n') else coded_block__pattern = []; end

% make macroblock lieader and write to bitstream len = length(mblock_addr_inc) + length(macroblock_type) + ... length(quantizer_scale) + length(mv_fwd_vlc) + length(mv_bwd_vlc) + ... length(coded_block_pattern) + length(mblock_buffer) ; buffer = [mblock_addr_inc macroblock_type quantizer_scale mv_fwd_vlc mv_bwd_vlc coded_block_pattern mblock_buffer] ' ' bytes_written = write_to_bitstream(buffer) ;

% update future picture store from reconstructed macroblock and mcp if ~cbp % no blocks coded future_mblock = mcp; else % blocks coded according to cbp future_mblock = rec_mblock + mcp; end [top, left] = get_top_left (mb_addr); update_pic_store ('RECON_PIC', future_mblock, top, left); return % debugging check [top, left] = get_tl(mb_addr); RECPIC(top:top+15,left:left+23) = rec_mblock; %functio n code p picture (current_pic, temporal_referencei % John Bateman global PICTURE_START_CODE global HORIZ_SIZE global VERT_SIZE global MV_FWD global FULL_PEL_FWD FORWARD_F_CODE FORWARD_F FORWARD_F_SIZE MODULUS global MCP_FWD global PREVIOUS_PIC log_message(['start coding picture P' int2str(temporal_reference)]); if temporal_reference < 0 | temporal_reference > 1023 log_message(' error in function code_p_picture: invalid temporal_reference') end % motion estimation (to half pel accuracy) & motion compensation % note that this determines MCP_FWD_DIFF, MV_FWD and the various f-codes range =7; % search range -7.5 to + 7.5 t=cputime; motion_estim_fwd (current_pic, PREVIOUS_PIC, range); e=cputime-t; fprintf('motion fwd = %.4f minutes\n\n',e/60);

% load motion_estim_fwd function output from files load motion_vectors_fwd load motion_comp_fwd

% make picture header temporal_reference = decimal_to_uimsbf (temporal_reference, 10); % 10 bits picture_coding_tYpe =[010]; % predictive-coded (P) vbv_delaY =[1111 1111 1111 1111]; %variable bitrate % FULL_PEL_FWD defined above FORWARD_F_CODE = decimal_to_uimsbf(FORWARD_F_CODE,3); extra_bit_picture = [0]; % no extra picture information following header = [PICTURE_START_CODE temporal_reference picture_coding_type ... vbv_delaY FULL_PEL_FWD FORWARD_F_CODE extra_bit_picture]; write_to_bitstream(header);

% divide motion compensated prediction difference into slices [slices, slice_length] = get_slices(current_pic); % m is nmber of 16x16 macrobloclcs in the picture m = (HORIZ_SIZE * VERT_SIZE)/(16*16) ; no_of_slices = m / slice_length; %no_of_slices = 2; slice_length = 6;

% code slices modes = []; last_intra_addr = -1; % initialise addr of last intra coded mbloclc for i = 1:no_of_slices slice = slices((i-1)*16+1:i*16,:); %slice = slices((i-1)*16 + l:i*16, 1:64) ; [last_intra_addr, new_slice] = code_p_slice (slice, i, slice_length, ... last_intra_addr) ; modes = [modes; new_slice]; end

show_modes(modes) save modes modes function [last_intra_addr, modes] code^_slice (slice, svp, slice_length,

% last_intra_addr) % global MB_WIDTH global START_CODE global MV_FWD % forward motion vector array global MCP_FWD global MCP_INTERP global HORIZ_SIZE global CODE_MCP

% code slice header %log_message(['** coding slice ' int2str(svp)]); slice_start_code = [START_CODE decimal_to_uimsbf(svp,8)]; quantizer_scale = 8; % set quantizer_scale (can be reset'in macroblock) extra_bit_slice = [0]; % no extra slice information following header = zeros (1,38); [slice_start_code decimal_to_uimsbf (quantizer_scale, 5). extra_bit_slice]; write_to_bit stream (header) ;

% code macroblocks in slice mv_fwd_prev = [0 0]; % fwd predition vector is zero at slice start dc_^redictor = [1024 1024 1024]; % initialise for slice start prev_mb_addr = (svp - 1)*MB_WIDTH - 1; % prev_mb_addr from previous slice addr_inc =1; % initialise address increment = zeros (1, length (slice_length) ) ; curr_mbloc]c = zeros(16,24); li = waitbar(0, ['coding slice ' , int2str (svp) ] ) ; for i = 1:slice_length waitbar (i/slice_length) % progress feedbacl^ intra = 0; slcipped =0; % initialise variables left = (i-l)*16+l; % luminance left col 1 = left - 8* (deft - 1)/16) + HORIZ_SIZE; % chrominance left col curr_mbloc]c( : , 1:16) = slice (:, left: left+15) ; % current luminance mbloc]^ curr_mblock(:,17:24) = slice(:, 1:1 + 7) ; % chrom blocks mb_addr = prev_mb_addr + i; % current address %fprintf(['\n** coding macroblock ' int2str(mb_addr)]);fprintf('\n')

% check for forced (intra) update if_(mb_addr - last_intra_addr) == 132 intra = 1; % force intra to stop accum. of IDCT mismatch errors %disp('forced update of intra block') else % not forced update intra t=cputime; [mv_fwd, mcp] = test_mc (curr_mblock(:,1:16),mb_addr); % mc or no mc %if svp==l,e=cputime-t;fprintf('test_mc = %.4f s\n\n',e);end

% intra/non-intra coding decision t=cputime; intra = test_intra (curr_mblock(:,1:16) , mcp(: , 1:16 ) ) ; % var of y %if svp==l,e=cputime-t;fprintf('test_intra = %.4f s\n\n',e);end end

% code non-intra macroblock if -intra [top, left] = get_top_left(mb_addr); update_pic_store ('CODE_MCP', mcp, top, left);

% determine diff motion vectors & update prediction vectors as reqd %fprintf('mv_fwd = [%d %d]\n',mv_fwd); %fprintf ('mv_fwd_prev = [%d %d] \n' , mv_fwd_prev) ; % form difference macroblock (mcp obtained from test_mc earlier) diff_mblock = curr_mblock - mcp;% form diff macroblock t=cputime; [skipped, intra] = code_p_mblock (curr_mblock, mcp, mb_addr, addr_inc, quantizer_scale, mv fwd prev, mv_fwd); %if svp==l,e=cputime-t;fprintf('code_p_mblock = %.4f s\n\n',e);end if -intra % successfully coded as non-intra macroblock (i.e. cbp non-zero) if skipped %disp('** skipped macroblock') addr_inc = addr_inc +1; % skipped macroblock else addr_inc = 1; % not skipped macroblock end mv_fwd_prev = mv_fwd; % update fwd predictor dc^redictor [1024 1024 1024] ; % reset dc predictors end end % code intra macroblock if intra %disp('** intra mode') % dc_predictor intra_d =[0001 1]; % vie for intra-d macroblock t=cputime; [dc_predictor, rec_mblock] = code_i_mblock (curr_mblock, ... intra_d, mb_addr, addr_inc, dc_predictor, quantizer_scale); %if svp==l,e=cputime-t;fprintf('code_i_mblock = %.4f s\n\n',e);end

t=cputime; [top, left] = get_top_left(mb_addr); update_pic_store ('RECON_PIC', rec_mblock, top, left); %if svp==l,e=cputime-t;fprintf('update pic .store = %.4f s\n\n',e);end

addr_inc = 1; % coded macroblock mv_fwd_prev = [0 0]; % reset fwd predition vector last_intra_addr= mb_addr; % reset last intra coded address end

% following code keeps track of mc modes for each macroblock %fprintf(' mv_fwd_prev = [%d %d]\n',mv_fwd_prev); if skipped modes(i) = 4; elseif intra modes(i) = 5; elseif ~mv_fwd modes(i) =6; % no mc else modes(i) =3; % fwd mv end end close(h) function code_seq_header (load_iq_matrix, 1oad_niq_matrix, iq.matrix, ... niq_matrix)

% John Bateman 28/6/95 global SEQUENCE_HEADER_CODE global HORIZ_SIZE global VERT_SIZE

% Assign empty matrices if default quantizer matrices are being used if ~(load_iq_matrix), iq_matrix = []; end if ~{load_niq_matrix), niq_matrix = []; end

% define parameters pel_aspect_ratio =[1000]; % CCIR601, 625 line picture_rate =[0011]; %25 pictures/s % set variable bit rate (must be < 1856000 for constrained bitstream) bit_rate =[11 1111 1111 1111 1111];% 0x3FFFF mar]cer_bi t = 1; % vbv_buffer_size of 19 gives 19*1024*16 = 311296 bits which is < max value % of 20*1024*16 = 311296 for constrained bitstream vbv_buffer_size =[00 0001 0011]; cpb =0; % this is not a constrained bitstream (because variable bit rate)

% malce sequence header % note: intra_quantizer_matrix and non_intra_quantizer_matrix are empty % arrays unless their respective load flags are set header = [SEQUENCE_HEADER_CODE decimal_to_uimsbf(HORIZ_SIZE, 12) decimal_to_uimsbf(VERT_SIZE, 12) ... pel_aspect_ratio picture_rate ... bit_rate marker_bit ... vbv_buffer_size cpb ... load_iq_matrix iq matrix ... load_niq_matrix niq_matrix]; write_to_bitstream(header); function code_sequence (source_file, pic_start, output_file, gop_type, gop_no)

% John Bateman 27/6/95, 28/6/95 global FID global REMAINDER I global SEQUENCE_END_CODE

log_message('start coding sequence'); % open output file for storing coded bitstream FID = fopen(output_file,'wb'); if FID == -1, error(['cannot open file ' output_file]), end

% open video source file fid_in = uvc_open(source_file, 'r'); if fid_in == -1, error{['cannot open file source_file]), end

% code sequence header (only 1 sequence header in bitstream) load_iq_matrix =0; % use default intra quantisation matrix load_niq_matrix =0; % use default non-intra quantisation matrix % intra_quantizer_matrix and non_intra_quantizer_matrix are not required in % the function code_seq_header() due to default matrices being used and are % therefore not passed as arguments code_seq_header (load_iq_matrix, load_niq_matrix);

% code group of pictures (only 1 gop in bitstream) if gop_type == 1 % I B P gop = [1 pic_start; 2 pic_start+2; 3 pic_start+l]; elseif gop_type - • 2 % I B B B P gop = [1 pic_start; 2 pic_start+4; 3 pic_start+l; 3 pic_start+2; ... 3 pic_start+3]; elseif gop_type == 3 %IBBPBBPBB P B B P gop = [1 pic_Start; 2z pic_start+3lcij: u-r ; 3 pic_start+l; 3 pic_start+2; 2 pic_start+6; 3 pic_start+4; 3 pic_start+5; 2 pic_start+9; 3 pic_start+7; 3 pic_start+8; 2 pic_start+12; 3 pic_start+10; 3 pic_start+ll]; else error('invalid gop'); end

% code first gop code_gop(gop, fid_in);

% code second gop [r,c] = size(gop); pic_start = pic_start + r; gop = [1 pic_start; 2 pic_start+3; 3 pic_start+l; 3 pic_start+2; 2 pic_start+6; 3 pic_start+4 7 3 pic_start+5; 2 pic_start+9; . 3 pic_start+7; 3 pic_start+8; 2 pic_start+12; 3 pic_start+10; 3 pic_start+ll]; % code_gop(gop, fid_in);

% end of bitstream bytes_written = write_to_bitstream(SEQUENCE_END_CODE);

% write any leftover bits to bitstream byte_stuffing = zeros(1,(8-length(REMAINDER))); % get zeros to fill byte last_byte = [REMAINDER byte_stuffing]; % last byte in bitstream REMAINDER = []; % no REMAINDER as last byte has stuffing write_to_bitstream(last_byte); % write last byte % close file bitstream.bin status = fclose(FID); if status == -1 log_message(['error in function code_sequence: cannot close file ' output_file return; end log_message('end coding sequence'); %functio n d = dct_2d (block) % function d = dct_2d (block) % % desription: % Returns 2-D discrete cosine transform (DCT) for 8x8 block of pels. The DCT % coefficients are not checked for range. % % input: % block : 8x8 integer matrix of pels % : each pel is assumed to be an integer in the range % -255<=pel<=255 % % output: 8x8 real matrix of DCT coefficients 0<=DC coeff<=2040 (not checked) -2048<=AC coeff<=2047 (not checked) %

% John Bateman 30/3/95, 18/5/95 global S % conversion factor for MATLAB function dct2()

% 2-D dct on block d = dct2(block)./S; %functio n binarY_no = decimal_to_simsbf (decimal_no, no_of_bits) % function binarY_no = decimal_to_simsbf (decimal_no, no_of_bits) % % description: % Converts decimal number to binary number of length no_of_bits % in signed integer msb first (simsbf) format

% input: % decimal_no decimal number % integer % no of bits number of bits in output binarY_no integer

% output: % binary_no : binary number (simsbf) array % % eg. decimal_no = -27, no_of_bits = 8 is converted to [1110 0101] % % John Bateman 9/4/95, 25/4/95, 10/5/95, 11/6/95 %

% handle 0 case if decimal_no -- 0 binary_no == zeros(1,no_of_bits); return; end

% get sign bit if decimal_no > 0 binary_no(l) =0; % positive else binary_no(l) =1; % negative decimal_no = decimal_no + 2^^ (no_of_bits - 1); end

% get rest of the bits for i = 2:no_of_bits if fix(decimal_no / 2^(no_of_bits - i)) >0 binary_no(i) = 1; decimal_no = decimal_no - 2^{no_of_bits - i); else binary_no(i) = 0; end end %functio n binarY_no = decimal_to_uimsbf (decimal_no, no_of_bits) % function binarY_no = decimal_to_uimsbf (decimal_no, no_of_bits) % % description: % Converts decimal number to binary number of length no_of_bits % in uimsbf format

% input: % decimal_no decimal number % integer % no of bits number of bits in output binarY_no integer

% output: % binarY_no : binary number (uimsbf) array % % eg- decimal_no = 27, no_of_bits = 8 is converted to [0001 1011] % % John Bateman 9/4/95, 25/4/95, 10/5/95 % for i = l:no_of_bits if fix(decimal_no / 2^(no_of_bits - i)) >0 binary_no(i) = 1; decimal_no = decimal_no - 2^^ (no_of_bits - i); else binary_no(i) = 0; end end function [mb_addr, addr_inc, vlc_err] = decode_addr_inc (mb_addr_prev) % global MACROBLOCK_STUFFING global MACROBLOCK_ESCAPE vlc_err = 0;

% remove any stuffing bits while nextbits(ll) == MACROBLOCK_STUFFING get_bits(11); end

% determine offset (if any) for macroblock_address_increment offset = 0; while nextbits(ll) == MACROBLOCK_ESCAPE get_bits(11); offset = offset + 33; end

% decode macroblock address increment no_match = 1; % flag rows =11; % no. of rows in ADDR_INC matrix vlc_err = 0 len = 0 val = 0 while no_match & len <= rows & ~vlc_err val = 2 * val + get_bits(l); % start with 1 bit val (smallest code) len = len + 1; [addr_inc, vlc_err] = get_addr_inc(len, val); if addr_inc no_match = 0; end end

if no_match | vlc_err log_message('invalid macroblock address increment detected'); end mb_addr = mb_addr__prev + addr_inc + offset; %fprintf('addr_inc = %d\n',addr_inc ); function [mb_tYpe, err] = decode_b_mb_type

% decode macroblock_tYpe val = get_bits(l); len = 1; [flags, type_err] = get_b_type (val,len); while -flags & ~tYpe_err val = 2 * val + get_bits(l); len = len + 1; [flags, tYpe_err] = get_b_type (val,len); if type_err log_message('bit overrun decoding macroblock type') err = 1; return; end end mb_type = decimal_to_uimsbf (val, len); %fprintf('macroblock_tYpe = '); fprintf('%d', mb_type); fprintf('\n') function [mv_fwd__prev, mv_bwd_prev, err] = decode_b_mblock (mb_type, mb_addr,.. quantizer_scale, mv_fwd_prev, full_pel_fwd, forward_f_code, .. . mv_bwd_prev, full_pel_bwd, backward_f_code)

global PREVIOUS_PIC % previous picture store global FUTURE_PIC % future picture store global DISPLAY_BUFFER % next picture to be displayed global RECPIC global DECODED_PIC % next picture to be displayed err = 0; %fprintf(['\n** decoding macroblock ' int2str(mb_addr)]);fprintf('\n') %fprintf('macroblock_type = '); fprintf('%d', mb_type); fprintf('\n'); % find flags based on macroblock type len = length{mb_type); val = uimsbf_to_decinial (mb_type) ; [flags, type_err] = get_b_type (val, len); if type_err, log_message('invalid macroblock type'), err = 1; return, end macroblock_quant = flags(1) motion_forward = flags (2) motion_backward = flags (3) macroblock__pattern = flags (4) macroblock_intra = flags(5) macroblock_type = decimal_to_uimsbf (val, len);

% determine quantizer_scale for those cases where macroblock_quant is asserted if macroblock_quant quantizer_scale = uimsbf_to_decimal (get_bits(5)); end

% decode forward motion vectors & apply to PREVIOUS_PIC [top, left] = get_top_left (mb_addr); % row,col of top-left pel if motion_forward right_fwd_prev = mv_fwd_prev(1); down_fwd_prev = mv fwd prev(2); [recon_mv_fwd(l), recon_mv_fwd(2)] = ... decode_mv (right_fwd__prev, down_fwd_prev, forward_f_code); [mcp_fwd, mcp_err] = apply_mcp (PREVIOUS_PIC, top, left, ... recon_mv_fwd, full_pel_fwd); if mcp_err log_message('error during^ forward motion compensation') err=l; return end mv_fwd_jprev = recon_mv_fwd; % update fwd prediction motion vectors else recon_mv_fwd = mv_fwd_prev7 % for no motion forward end

% decode backward motion vectors & apply to FUTURE_PIC if motion_backward right_bwd_prev = mv_bwd_prev(1); down_bwd_prev = mv_bwd_prev (2) [recon_mv_bwd(l), recon_mv_bwd(2)] = ... decode_mv (right_bwd_prev, down_bwd_prev, backward_f_code); [mcp_bwd, mcp_err] = apply_mcp (FUTURE_PIC, top, left, ... recon_mv_bwd, full_pel_bwd); if mcp_err log_message('error during backward motion compensation') err=l; return end inv_bwd_prev = recon_mv_bwd7 % update bwd prediction motion vectors else recon_mv_bwd = mv_bwd_prev; % for no motion backward end

% determine motion compensated prediction if motion_forward & motion_backward mcp = round((mcp_fwd + mcp_bwd)/2); % linear interpolation elseif motion_forward mcp = mcp_fwd; elseif motion_backward mcp = mcp_bwd; end

% decode cbp if macroblock_pattern no_matcli = 1; % flag rows = 9; % no. of rows in CBP matrix vlc_err = 0; len = 2; val = 2 * get_bits(l) + get_bits(l); 2 bit value while no._matc h & len <= rows & ~vlc_err val = 2 * val + get_bits(l); % start with 3 bit val (smallest code) len = len + 1; [cbp, vlc_err] = get_cbp(len, val); if vlc_err, log_message('invalid coded block pattern'), err=l, return, end if cbp cbp = decimal_to_uimsbf(cbp, 6); no_match = 0; end end if no_match, log_message('invalid coded block pattern'), err=l, return, end %fprintf('cbp = ');fprintf('%d',cbp);fprintf('\n')

% decode [YO Y1 Y2 Y3 Cr4 Cb5 ] r = [1 8;1 8,-9 16,-9 16;1 8;9 16]; % block rows c = [1 8;9 16;1 8;9 16;17 24;17 24]; % block cols decoded_mblock = zeros(16,24); for i = 1:6 if cbp(i) [decoded_mblock(r(i, 1) :r(i,2), c(i,l):c(i,2)), err] = ... decode_ni_block (quantizer_scale); if err, log_message('error decoding b block'), err=l, return, end end end displaY_mblock = decoded_mblock + mcp; % recover original macroblock else % original macroblock was not coded displaY_mblock = mcp; % no received coefficient information end

%decoded_mblock,display_mblock,keyboard

% update display buffer [top, left] = get_top_left(mb_addr); update_pic_store ('DECODED_PIC', display_mblock, top, left); return % debugging - clieck decoder recon against decoded [top, left] = get_tl(mb_addr); rec_mblock = RECPIC(top:top+15,left:left+23); if ~(rec_mblock-decoded_mblock) disp('&:&& rec_mblock = decoded_mblock' ) elseif -isempty(decoded_mblock) disp('&&& rec_mblock != decoded_mblock') %keyboard end function decode_b_picture ()

% set globals global START_CODE global USER_DATA_START_CODE global EXTENSION_START_CODE

% decode forward motion vector information full_pel_fwd = get_bits(l); % get full pel forward vector forward_f_code = uimsbf_to_decimal(get_bits(3)); % get forward_f_code %fprintf('full_pel_fwd = %d\n',full_pel_fwd);

%fprintf('forward_f_code = %d\n',uimsbf_to_decimal(forward_f_code));

% decode backward motion vector information full_pel_bwd = get_bits(l); % get full pel backward vector backward_f_code = uimsbf_to_decimal(get_bits(3)); % get backward_f_code %fprintf ( ' full__pel_bwd == %d\n' , full_pel_bwd) ; %fprintf ( 'backward_f_code = %d\n' , uimsbf_to_decimal (backward_f_code) ) ; % discard any extra picture information while nextbits(l) get_bits(l); % discard extra_bit_picture = 1 get_bits(8); % discard extra_information_picture end get_bits(l); % discard extra_bit_picture = 0

% bring up next start code next_start_code;

% discard any picture extension data if nextbits(32) == EXTENSION_START_CODE get_bits(32); % discard EXTENSION_START_CODE while not_equal {nextbits(24), START_CODE) get_bits(8); % discard picture_extension_data end next_start_code; end

% discard any user data if nextbits(32) == USER_DATA_START_CODE get_bits(32); % discard USER_DATA_START_CODE while not_equal (nextbits(24), START_CODE) get_bits(8); % discard user_data end next_start_code; end

% decode slices svp_past = 1; % initialise for first slice while nextbits(24) == START_CODE slice_start_code = nextbits(32); % look at next 32 bits svp = uimsbf_to_decimal (slice_start_code (25:32)); % extract svp if svp < 1 I svp > 175 I svp < svp_past % higher level start code or invalid svp return; end slice_start_code = get_bits(32); % get next 32 bits as slice code is valid err = decode_b_slice (svp, full_pel_fwd, forward_f_code,... full_pel_bwd, backward_f_code); if err, next__start_code; end % error during decoding svp_past = svp; % update svp_past end function err = decode_b_slice (svp, full_pel_fwd, forward_f_code, .-. full_pel_bwd, backward_f_code) err =0; % error flag

% decodes slice layer for slices in b pictures global MB_WIDTH global DISPLAY_BUFFER

% get quantizer_scale quantizer_scale = uimsbf_to_decimal (get_bits(5)); if ~(quantizer_scale) log_message ('invalid quantizer scale (quantizer_scale = 0 forbidden)'); end

% discard any extra slice information while nextbits{l) get_bits(l); % discard extra_bit_slice = 1 get_bits(8); % discard extra_information_slice end get_bits(1); % discard extra_bit_slice = 0

% decode macroblocks mb_addr_prev = (svp - 1) * MB_WIDTH - 1; mv_fwd_prev = [0 0]; % fwd predition vector is zero at slice start mv_bwd prev = [0 0]; % bwd predition vector is zero at slice start dc_:predictor = [1024 1024 1024]; % initialise for slice start intra_d =[0001 1]; % intra-d macroblock vie intra_q =[0000 01]; % intra-q macroblock vie h = waitbar(0,['decoding slice ',int2str(svp)]);i = 1; wliile any (nextbits (23 ) ) % while next 23 bits != 000 0000 0000 0000 0000 0000 waitbar(i/MB_WIDTH); i = i+1; % progress feedback % find macroblock address [mb_addr, addr_inc, addr_err] = decode_addr_inc (mb_addr_prev); if addr_err, log_message('error decoding mb_addr in B slice'), end % process skipped macroblocks if addr_inc > 1 recover_b_skipped (mb_addr_prev, addr_inc, mb_type_prev, ... mv_fwd_prev, full^el_fwd, mv_bwd_prev, full_pel_bwd) ; dc_predictor = [1024 1024 1024]; % reset dc predictors end

% decode macroblocks according to type [mb_type, err] = decode_b_mb_type; len = length(mb_type); val = uimsbf_to_decimal(mb_type); [flags, type_err] = get_b_type (val, len); if type_err, log_message('error decoding mb_type in B slice'), end macroblock_quant = flags(1); macroblock_intra = flags(5);

% check for intra-d or intra-q if ~macroblock_intra % decode predictively coded macroblock [mv_fwd_prev, mv_bwd_prev, err] = decode_b_mblock (mb_type, mb_addr, . quantizer_scale, mv_fwd_prev, full_pel_fwd, forward_f_code, ... mv_bwd__prev, full_pel_bwd, backward_f_code) ; if err, log_message('error decoding b macroblock in B slice'), end dc_predictor = [1024 1024 1024]; % reset dc predictors else % decode intra coded macroblock %disp('intra macroblock detected ') [decoded_mblock, dc_^redictor, err] = decode_i_mblock ... (macroblock_quant, mb_addr, dc_predictor, quantizer_scale); if err, log_message('error decoding i macroblock in B slice'), end % update display buffer [top, left] = get_top_left (mb_addr); update_pic_store ('DECODED_PIC', decoded_mblock, top, left) mv_fwd_prev = [0 0],- % reset fwd predition vector mv_bwd_prev =[00]; % reset bwd predition vector end % update previous values mb_type_prev = mb_type; mb_addr_prev = mb_addr; end close(]i) % bring up next start code nex t_s tar t_c ode; function [dc, dc_err] = decode_chrom_dc (previous_dc)

% description: % % input: % % output:

dc_err =0; % initially false

% get size in bits of the dc additional code val = 2 * get_bits(l) + get_bits(l); % get first 2 bits len = 2; while len <= 8 if len == 2 if ~val size = 0; dc = previous_dc; return; % difference = 0 in this case elseif val == 1 size = 1; break elseif val == 2 size = 2; break end elseif len == 3 & val == 6 size = 3; break elseif len == 4 & val •• - 14 size = 4; break elseif len == 5 & val == 30 size = 5; break elseif len == 6 & val == 62 size = 6; break elseif len == 7 & val == 126 size = 7; break elseif len == 8 if val == 254 size = 8; break else % invalid code dc_err = 1; log_message('error in decode_chrom_dc: invalid dc size code') return end end % get next bit val = 2 * val + get_bits(l); % start with 2 bit val (smallest code) len = len + 1; end

% get the dc additional code test_bits = get_bits(size); if test_bits(1,1) == 1 difference = uimsbf_to_decimal(test_bits) ; % positive difference test_bitsue else test_bits = ~test_bits; % take one's complement difference = uimsbf_to_decimal(test_bits) ; difference = -1 * difference; % negative difference test_bitsue end

% inverse quantise difference value difference = 8 * difference;

% recover original dc pel value dc = previous_dc + difference; function [last_pic, last_pic_tYpe, last_pic_no] = ... decode_gop (last__pic, las t_pic_tYpe, last_pic_no)

% set global variables global START_CODE global USER_DATA_START_CODE global EXTENSION_START_CODE global PICTURE_START_CODE global DECODED_PIC global PREVIOUS_PIC global FUTURE_PIC global VERT_SIZE global HORIZ_SIZE global FID_VIDEO_OUT

% decode time codes drop_frame_flag get_bits(1); time_code_hours uimsbf_to_decima1 (get_bits(5) ) ; time_code_minutes = uimsbf_to_decimal (get_bits(6)); marker_bit = get_bits(1); time_code_seconds = uimsbf_to_decimal (get_bits(6)); time_code_pictures = uimsbf to decimal (get_bits(6)); 1og_message(['Drop frame flag ',int2str(drop_frame_flag)]) log_message(['Time code hours ',int2str(time_code_hours)]) log_message(['Time code minutes ',int2str(time_code_minutes) ] ) log_message(['Marker bit ',int2str(marker_bit)]); log_message(['Time code seconds ',int2str(time_code_seconds) ] ) log_message(['Time code pictures ',int2str(time_code_pictures ] )

% decode closed gop closed_gop = get_bits(l); log_message(['Closed gop int2str(closed_gop)]);

% decode broken_link broken_link = get_bits(l) log_message(['Broken link int2str(broken_link)]); fprintf('\n');

% discard any extension data next_start_code; if nextbits(32) == EXTENSION_START_CODE get_bits(32); % discard EXTENSION_START_CODE while not_equal (nextbits(24), START_CODE) get_bits(8); % discard extension_data end next_s tar t_c ode; end

% discard any user data if nextbits (32) USER_DATA_START_CODE get_bits(32); % discard USER_DATA_START_CODE while not_equal (nextbits(24), START_CODE) get_bits(8); % discard user data end next_start_code; end

% decode first picture & load it into bufferl DECODED_PIC = zeros(2 88,528); t=cputime; [pic_type, pic_no] = decode pic type; decode_i_picture; e = cputime-t; I_DECODE = DECODED_PIC; save I_DECODE I_DECODE bufferl = DECODED_PIC; % load into bufferl buffer_index(1) = pic_no; % load picture number into index array fprintf('decode picture cputime = %f minutes\n\n',e/60); %display_j/uv_pic (DECODED_PIC) ; first_pic_type = pic_type; first_pic_no = pic_no; % save old values

% decode header of second picture (picture type & picture number) closed =1; % assert closed gop [pic_type, pic_no] = decode_pic_type7

% set up PREVIOUS_PIC, FUTURE_PIC picture matrices according to % whether or not there are B pictures at the start of gop, and whether % or not the gop is closed (i.e. picture from last gop reqd for decoding) if pic_type==3 & (pic_no < first_pic_no) if -closed PREVIOUS_PIC = last_pic; prev pic_no = last_pic_no; prev__pic_type = last pic type; else PREVIOUS_PIC = [ ] prev_pic_no = [ ] prev_pic_type= [ ] end FUTURE_PIC = DECODED_PIC; future_pic_no= first_pic_no; future_pic_type = f irst__pic_type; else PREVIOUS_PIC = DECODED_PIC; prev_pic_no = first pic no; prev_pic_type= f irst_pic_type; FUTURE_PIC = [ ] ; future__pic_no= [ ] ; future_pic_type = [ ] ; end

% disp stuff prev = [prev_pic_type prev_pic_no] ; future = [future_pic_type future_pic_no] ; fprintf('previous picture: ');tell(prev); fprintf(', future picture: ');tell(future) ; fprintf('\n' ) ;

% decode the rest of the pictures in gop % the loop exit condition is at the end DECODED_PIC = zeros(VERT_SIZE, HORIZ_SIZE + H0RIZ_SIZE/2); k = 2; % index to decoded picture buffer (already decoded first) while (1) t=cputime; % start timer if pic_type == 1 % decode I picture %disp('pause...'),pause decode_i__picture; e = cputime-t; I_DECODE = DECODED_PIC; save I_DECODE I_DECODE %display_yuv_pic (DECODED_PIC); elseif pic_type == 2 % decode P picture %disp('pause...'),pause de c ode__p__p i c tur e; e = cputime-t; P__DECODE = DECODED_PIC; save P_DECODE P_DECODE

%displaY_:^uv_pic (DECODED_PIC) % set up FUTURE_PIC, PREVIOUS_PIC dependent on previous picture details if ~isemptY(prev_pic_no) & abs(pic_no - prev_pic_no) > 1 FUTURE_PIC = DECODED_PIC; future_pic_no = pic_no; future_pic_tYpe = pic_tYpe; else PREVIOUS_PIC = DECODED_PIC; prev_pic_no = pic_no; prev_pic_tYpe = pic_type; FUTURE_PIC = []; future pic_no- []; future_pic_tYpe = []; end elseif pic_tYpe == 3 % B picture %disp('pause...'),pause decode b picture; e = cputime-t; B_DECODE = DECODED_PIC; save B_DECODE B_DECODE %displaY_YUV_pic (DECODED_PIC); % set up FUTURE_PIC, PREVIOUS_PIC dependent on previous picture details if -isemptY(future_pic_no) & abs(pic_no - future_pic_no) == 1 PREVIOUS_PIC = FUTURE_PIC; prev pic no = future_pic_no; prev pic type = future_pic_tYpe; FUTURE_PIC = []7 future_pic_no= []; future_pic_tYpe = []; end end fprintf('decode picture cputime = %f minutes\n\n',e/60);

% disp stuff prev = [prev_pic_tYPe prev_pic_no]; future = [future_pic_tYPe future_pic_no]; fprintf('previous picture: ');tell(prev); fprintf(', future picture: ');tell(future); fprintf('\n'); % load decoded picture into buffer (max of 12 buffers which means % that max number of consecutive B pictures is 10) if k = = 1, bufferl = DECODED,.PIC ; buffer_ .index (k) - pic_.no ; elseif k == 2, buffer2 DECODED..PIC ; buffer_ .index (k) = pic..no ; elseif k == 3, buffer3 DECODED..PIC ; buffer..inde x (k) = pic_.no ; elseif k 4, buffer4 DECODED..PIC ; buffer..inde x (k) - pic__no ; elseif k = = 5, buffers = DECODED..PIC ; buffer..inde x (k) - pic..no ; elseif k — — 6, buffer6 -- DECODED..PIC ; buffer..inde x (k) = pic..no ; elseif k = = 7, bufferV = DECODED..PIC ; buffer..inde x (k) = pic..no ; elseif k - — 8, buffers DECODED..PIC ; buffer..inde x (k) - pic..no ; elseif k = = 9, buffer9 = DECODED..PIC ; buffer..inde x (k) = pic..no ; elseif k 10, bufferlO = DECODED..PIC ; buffer..inde x (k) - pic._no ; elseif k 11, bufferll = DECODED..PIC ; buffer..inde x (k) pic._no ; elseif k = = 12, bufferl2 = DECODED..PIC ; buffer..inde x (k) - pic..no ; else error('maximum number of consecutive B pictures is 10') end

% check if all B pictures between I/P have been decoded so that the % buffered pictures can be written to file in correct order sequential = check_order (buffer_index); % picture nos in sequence ? if sequential sorted = sort(buffer_index); % sort into ascending order for i = 1:length(buffer_index); j is index of next buffer in display order to be written to file j = find(buffer_index == sorted(i)); if j 1 [y,u,v] = picture2yuv (bufferl); elseif j 2 [y,u,v] = picture2yuv (buffer2); elseif j == 3 [y,u,v] = picture2yuv (buffer3); elseif j == 4 [y,u,v] = picture2yuv (buffer4); elseif j == 5 [y,u,v] = picture2yuv (buffers); elseif j == 6 [y,u,v] = picture2yuv (buffer6); elseif j == 7 [y,u,v] = picture2yuv (buffer?); elseif j == 8 [y,u,v] = picture2yuv (buffers); elseif j == 9 [y,u,v] = picture2yuv (buffer9); elseif j == 10 [y,u,v] = picture2yuv (bufferlO); elseif j == 11 [y,u,v] = picture2yuv (bufferll); elseif j == 12 [y,u,v] = picture2yuv (bufferl2); else error('maximum number of consecutive B pictures is 10') end pic_number = uvc_write_frame(FID_VIDEO_OUT,y,u,v); end end

% check next start code if not_equal(nextbits(32) ,PICTURE_START_CODE) , break, end % no more pictures

% picture start code found so get next picture header [pic_type, pic_no] = decode__pic_type ;

% update decoded picture buffer index & buffer as required if sequential buffer_index = []; % start new decoded picture buffer k = 1; % new index else k = k + 1; % increment index end end

% save details of last decoded picture in gop last_j)ic = DECODED_PIC; last_pic_type = pic_type; last_pic_no = pic_no; function [decoded_block, err] = decode_i_block (dc, quantizer_scale) global IQ_MATRIX

% decode ac coeffs % initialise values & get first 2 bits from bitstream val get_bits(2); % get first 2 bits val 2 * val(l) + val(2); % convert to decimal len 2; % length of val in bits eob 2; % eob = [10] or 2 decimal j 1; % run_level row index escape_code [0 0 0 0 0 1] % escape code test_bits zeros(1,20) err 0; set error flag false max non esc 17; bits in max non-escape code

% decode run and level while val ~= eob no_match = 1; % (run, level) not found is true

% decode non-escape coeffs while no_match & len <= max_non_esc val = 2 * val + get_bits(l); len = len + 1; [run_level(j,:), vlc_err] = get_run_level (len, val); if vlc_err, brealc, end % len <= max_non_esc but val > max val if run_level(j,2) ~= 0 no_match = 0; end end

% check if unmatched string (len, val) is part of escape sequence if no_match test_bits = decimal_to_uimsbf(val, len); if not_equal(test_bits(1:6), escape_code) log_message('invalid coeff run/length') err = 1; return end % escape_code confirmed test_bits = [test_bits get_bits(20-len)]; % test..bit s now 20 bits run_level(j,1) = uimsbf_to_decimal(test_bits(7:12)); % get run val = uimsbf_to_decimal(test_bits(13:20)); if -val % 16 bit level 128 to 255 run_level(j,2) = uimsbf_to_decimal(get_bits(8) ) % get level if run_level(j,2) <12 8 err =1; % illegal code return; end elseif val == 128 % 16 bit level -255 to -128 run_level(j,2) = uimsbf_to_decimal(get_bits(8)) - 256; % get level if run_level(j,2) < -2 5 5 | run_level(j,2) > -12 8 log_message('invalid coeff run/length'), err = 1; return end else % 8 bit level -127 to 127 if val >= 129 val = val - 256; % handle two's complement numbers end; run_level(j,2) = val; % get level end end

%fprintf(1,'run = %d, level = %d\n',run_level(j,1),run_level(j,2)) val = get_bits(2); % get next two bits val = 2 * val(l) + val(2); % convert to decimal len =2; % 2 bits j = j + 1; % inc run_level row end

% recover quantised dct coefficients from (run,level) pairs q_coeffs = rl_decode_ac (run_level);

% inverse quantize the dct coefficients coeffs = inv_intra_quant (q_coeffs, IQ_MATRIX, quantizer_scale);

% include dc coefficient (already inverse quantized) coeffs(1,1) = dc;

% get pel values %decoded_block = inv_dct_2d(coeffs); decoded_block = idctSxS(coeffs); % use fast C inverse DCT

% clip decoded_block to [0,255] decoded_block = (decoded_block <= 255).*decoded_block + ... (decoded_block > 255)*255; % clip to 255 decoded_block = (decoded_block >= -255).*decoded_block; + ... (decoded_block < -255) * (-255) ,- % clip to -255 function [decoded_mblock, dc_predictor, err] = decode_i_mblock ..- (macroblock_quant, mb_addr, dc_predictor, quantizer_scale)

%fprintf(['\n** decoding macroblock ' int2str(mb_addr)]);fprintf('\n') %fprintf('macroblock_quant = %d\n', macroblock_quant);

% determine quantizer_scale for those cases where macroblock_quant is asserted if macroblock_quant quantizer_scale = uimsbf_to_decimal (get_bits(5)); end % decode luminance blocks [YO Y1 Y2 Y3 ] r = [1 8;1 8;9 16;9 16]; % block rows c = [1 8;9 16;1 8;9 16]; % block cols decoded_mblock = zeros(16,24); dc_7 = dc_predictor(1); for i = 1:4 % decode luminance dc coeff (dc) using dc predictor dc %fprintf('dc y predictor = %d\n', dc^) ; [dc_Y, dc_err] = decode_dc_7 (dc_Y); if dc_err, log_message ('invalid dc coeff), err = 1, return, end %fprintf('decoded dc y = %d\n', dc_y); [decoded_mblock(r(i,1):r(i,2), c(i,l):c(i,2)), err] = ... decode_i_block (dc_y, quantizer_scale); if err, log_message('error decoding intra block'), err=l; return, end end dc_predictor(1) = dc_/;

% decode Cr chrominance block %fprintf('dc Cr predictor = %d\n', dc_predictor(2)); [dc_predictor(2), dc_err] = decode_dc_uv (dc_predictor(2)); %fprintf('decoded dc Cr = %d\n', dc_predictor(2)); [decoded_mblock(l:8,17:decode_i_bloc2 4), errk ](dc_predictor(2) = ... , quantizer_scale); % decode Cb chrominance block %fprintf('dc Cb predictor = %d\n', dc_predictor(3)); [dc_predictor(3), dc_err] = decode_dc_uv (dc_predictor(3)); %fprintf('decoded dc Cb = %d\n', dc_predictor(3)); [decoded_mblock(9:16,17:24), err] = ... decode_i_block (dc_predictor(3), quantizer_scale); function decode_i_picture {)

% set globals global START_CODE global USER_DATA_START_CODE global EXTENSION_START_CODE global SEQUENCE_END_CODE global DECODED_PIC

% discard any extra picture information while nextbits(l) get_bits(l); % discard extra_bit_picture = 1 get_bits(8); % discard extra_information_picture end get_bits(l); % discard extra_bit_picture = 0

% bring up next start code next_start_code;

% discard any picture extension data if nextbits(32) == EXTENSION_START_CODE get_bits(32); % discard EXTENSION_START_CODE while not_equal (nextbits(24), START_CODE) get_bits(8}; % discard picture_extension_data end next_start_code; end

% discard any user data if nextbits(32) == USER_DATA_START_CODE get_bits(32); % discard USER_DATA_START_CODE while not_equal (nextbits(24), START_CODE) get_bits(8); % discard user_data end next_start_code; end

% decode slices svp_past =1; % initialise for first slice while nextbits(24) == START_CODE slice_start_code = nextbits(32); % look at next 32 bits svp = uimsbf_to_decimal (slice_start_code (25:32)); % extract svp if svp < 1 I svp > 175 I svp < svp_past % higher level start code or invalid svp return; end slice_start_code = get_bits(32); % get next 32 bits as slice code is valid err = decode_i_slice (svp); if err, next_start_code; end % error during decoding svp_past = svp; % update svp_past end function err = decode_i_slice (svp) % John Bateman 3 0/8/95 global MB_WIDTH global PREVIOUS_PIC err =0; % error flag

% get quantizer_scale quantizer_scale = uimsbf_to_decimal {get_bits(5)); if ~(quantizer_scale) log_message ('invalid quantizer scale (quantizer_scale = 0 forbidden)'); end % discard any extra slice information while nextbits(l) get_bits(l); % discard extra_bit_slice = 1 get_bits{l); % discard extra_information_slice end get_bits(l); % discard extra_bit_slice = 0

% decode macroblock layer mb_addr = (svp - 1) * MB_WIDTH - 1; % is -1 for first slice dc_predictor = [1024 1024 1024]; % initialise for slice start h = waitbar(0,['decoding slice ',int2str(svp)]);i = 1; while any(nextbits(23)) waitbar(i/MB_WIDTH); i = i+1; % progress feedback % find macroblock address [mb_addr, addr_inc, addr_err] = decode_addr_inc (mb_addr); if addr_err, err = 1; return, end if ~addr_inc, err = 1; return, end

% decode macroblocks according to type (intra-d/intra-q) val = get_bits(1); if val == 1 macroblock_quant =0; % intra-d macroblock else val = 2*val + get_bits(l); if val == 1 macroblock_quant = 1; % intra-q macroblock else log_message('error decoding macroblock type in I slice'); end end [decoded_mblock, dc_predictor, err] = decode_i_mblock ... (macroblock_quant, mb_addr, dc_predictor, quantizer_scale); if err, log_message('error decoding i macroblock in I slice'), end % update previous picture buffer [top, left] = get_top_left (mb_addr);

update_pic_store ('DECODED_PIC', decoded_mblock, top, left);

close(hend ) % bring up next start code next_start_code; function [recon_right_for_prev, recon_down_for_prev] = ... decode_mv (recon right for prev, recon_down_for_prev, forward_f_code)

% find f codes forward_f_size = forward_f_code -1; forward_f = 2'^f orward_f_size ;

% decode motion_horiz_fwd_code no_match =1; % flag rows = 11; % no. of rows in MV_TABLE matrix vlc_err = 0 ; len = 0; val = 0; while no_match & len <= rows & ~vlc_err val = 2 * val + get_bits(l); % start with 1 bit val (smallest code) len = len + 1; [motion_horiz_fwd_code, vlc_err] = get_mv(len, val) ; if motion_horiz_fwd_code 99 no_match 0 ; end end if no_match, fprintf('invalid horizontal motion vector detected'); end %fprintf ('motion_horiz_fwd_code = %d\n', motion_horiz_fwd_code) ;

% decode motion_horiz_fwd_r if forward_f_size motion_horiz_fwd_r = uimsbf_to_decimal(get_bits(forward_f_size)); else motion_horiz_fwd_r = 0; end %fprintf ('motion_horiz_fwd_r = %d\n', motion_horiz_fwd_r) ;

% decode motion_vert_fwd_code no_match =1; % flag rows =11; % no. of rows in MV_TABLE matrix vlc_err = 0; len = 0; val = 0; while no_match & len <= rows & ~vlc_err val = 2 * val + get_bits(l); % start with 1 bit val (smallest code) len = len + 1; [motion_vert_fwd_code, vlc_err] = get_mv(len, val) ; if motion_vert_fwd_code 9 9 no_match = 0; end end if no_match, fprintf('invalid vertical motion vector detected'); end %fprintf ('motion_vert_fwd_code = %d\n', motion_vert_fwd_code) ;

% decode motion_vert_fwd_r if forward_f_size motion_vert_fwd_r = uimsbf_to_decimal (get_bits (f orward_f_size) ) ; else motion_vert_fwd_r = 0; end %fprintf ('motion_vert_fwd_r = %d\n', motion_vert_fwd_r) ;

% find motion vector limits mv_max = (16 * forward_f) - 1; mv_min = -16 * forward_f;

% reconstruct horizontal motion vector recon_right_for = recon_right_for_prev + motion_horiz_fwd_code * forward_f . . . + motion_horiz_fwd_r; if recon_right_for > mv_max recon_right_for = recon_right_for - 32 * forward_f; elseif recon_right_for < mv_min recon_right_for = recon_right_for + 32 * fo3rward_f; end %fprintf('recon_right_for = %d\n', recon_right_for);

% reconstruct vertical motion vector recon_down_for = recon_down_for_jprev + motion_vert_fwd_code * forward_f ... + motion_vert_fwd_r; if recon_down_for > mv_max recon_down_for = recon_down_for - 32 * forward_f; elseif recon_down_for < mv_min recon_down_for = recon_down_for +32 * forward_f; end %fprintf('recon_down_for = %d\n', recon_down_for); % update previous motion vector recon_right_for_prev = recon_right_for; recon down for prev = recon_down_for; function [decoded_block, err] = decode_ni_block (quantizer_scale) global NIQ_MATRIX val = get_bits(2); % get next two bits val = 2 * val(l) + val(2); % convert to decimal len = 2 length of val in bits eob = 2 eob = [10] or 2 decimal j - 1 run level row index escape_code =[0000 01]; % escape code test_bits = zeros(l,20); vlc_err =0; % set error flag false err =0; % set error flag false max_non_esc =17; % bits in max non-escape code

% decode first coeff if run=0, and level=l (special case) if val > 1 if val == 2 run_level(j,:) =[0 1]; % run = 0, level = 1 else run_level(j,:) = [0 -1]; % run = 0, level = -1 end %fprintf(1first run = %d, level = %d\n',run_level(j,1),run_level(j,2)); val = get_bits(2); % get next two bits val = 2 * val(l) + val(2); % convert to decimal len =2; % lengtl:! of val in bits j = j + 1; end

% decode run and level wtiile val eob no_matc]i =1; % (run, level) not found is true % decode non-escape coeffs wliile no_match & len <= max_non_esc val = 2 * val + get_bits(l); len = len + 1; [run_level(j,:), vlc_err] = get_run_level (len, val); if vlc_err, brealc, end % len <= max_non_esc but val > max val if run_level(j,2) ~= 0 no_matcli =0; % if level = 0 end end

% chec]c if unmatched string (len, val) is part of escape sequence if no_match test_bits = decimal_to_uimsbf(val, len); if not_equal(test_bits(1:6), escape_code) log_message('invalid coeff run/level (failed esc test)'), err = 1; end % escape_code confirmed test_bits = [test_bits get_bits(20-len)]; % test_bits now 20 bits run_level(j,1) = uimsbf_to_decimal(test_bits(7:12)); % get run val = uimsbf_to_decimal(test_bits(13:20)) ; if -val % 16 bit level 128 to 255 run_level (j , 2) = uimsbf_to_decimal (get_bits (8) ) ; % get level if run_level(j,2) < 128 . log_message('invalid coeff level (level < 128)'), err = 1; end elseif val == 128 % 16 bit level -255 to -128 run_level (j , 2 ) = uimsbf_to_decimal (get_bits (8) ) - 256; % get level if run_level(j,2) < -255 | run_level(j,2) > -128 log_message('invalid coeff level (< -255 or > -128)'), err = 1; end else % 8 bit level -127 to 127 if val >= 129 val = val - 256; % handle two's complement numbers end; run_level(j,2) = val; % get level end end

%fprintf(1,'run = %d, level = %d\n',run_level(j,1),run_level(j,2)); val = get_bits(2); % get next two bits val = 2 * val(l) + val(2); % convert to decimal len =2; % 2 bits j = j + 1; % inc run_level row end

% recover quantised dct coefficients from (run,level) pairs q_coeffs = rl_decode(run_level);

% inverse quantize the dct coefficients coeffs = inv_non_intra_quant (q_coeffs, NIO MATRIX. quantizer_scale);

% get pel values %decoded_block = inv_dct_2d(coeffs); decoded_block = idctSxS(coeffs); % use fast C inverse DCT

% clip decoded_block decoded_block = (decoded_block <= 255).*decoded_block + ... (decoded_block > 255)*255; % clip to 255 decoded_block = (decoded_block >= -255) .*decoded_block; + ... (decoded_block < -255)* (-255) ; % clip to -255 function [mblock_type, err] = decode_p_mb_type

% decode macroblock_type val = get_bits(1); len = 1; [flags, tYpe_err] = get_p_tYpe (val,len); while -flags & ~tYpe_err val = 2 * val + get_bits(l); len = len + 1; [flags, tYpe_err] = get_p_type (val,len); if tYpe_err log_message('bit overrun decoding macroblock type') err = 1; return; end end mblock_tYpe = decimal_to_uimsbf (val, len); %fprintf('macroblock_type = '); fprintf('%d', mblock_type) ; fprintf('\n') ; function [mv fwd prev, err] = decode_p_mblock (mb_tYpe, mb_addr, ... quantizer_scale, mv_fwd_prev, full_pel_fwd, forward_f_code);

global PREVIOUS_PIC % previous picture store global DECODED_PIC % future picture store global DISPLAY_BUFFER % next picture to be displayed global RECPIC global DECODE_MCP

%fprintf(['\n** decoding macroblock ' int2str(mb_addr)]);fprintf('\n') %fprintf('macroblock_tYpe = '); fprintf{'%d', mb_type); fprintf('\n');

% find flags based on macroblock type len = length(mb_tYpe); val = uimsbf_to_decimal(mb_tYpe); [flags, tYpe_err] = get_p_type (val, len); if tYpe_err, log_message('invalid macroblock type'), err = 1; return, end macroblock_quant = flags(1) motion_forward = flags(2) motion_backward = flags(3) macroblock pattern = flags(4) macroblock_intra = flags(5) macroblock_type = decimal_to_uimsbf (val, len);

% determine quantizer_scale for those cases where macroblock_quant is asserted if macroblock_quant quantizer_scale = uimsbf_to_decimal (get_bits(5)); end

% decode forward motion vectors & apply to PREVIOUS_PIC [top, left] = get_top_left (mb_addr); % row,col of top-left pel if motion_forward right_fwd_prev = mv_fwd_prev(1); down, fwd prev = mv_fwd_prev(2); [recon_mv_fwd(l), recon_mv_fwd(2)] = ... decode_mv (right_fwd_prev, down fwd prev, forward_f_code); [mcp_fwd, mcp_err] = apply_mcp (PREVIOUS_PIC, top, left, ... recon_mv_fwd, full pel fwd); if mcp_err log_message('error during forward motion compensation') err=l; return end mv_fwd_prev = recon_mv_fwd; % update fwd prediction motion vectors else % forward motion vectors = [0 0], get macroblock from previous picture [top, left] = get_top_left (mb_addr); mcp_fwd = get_YCbCr_mblock (PREVIOUS_PIC, top, left); end update_pic_store ('DECODE_MCP', mcp_fwd, top, left);

% decode cbp %dbstop 5 5 if macroblock..pattern no_match = 1; % flag rows = 9; % no. of rows in CBP matrix vlc_err = 0; len = 2; val = 2 * get_bits(l) + get_bits(l); 2 bit value while no__matc h Sc len <= rows & ~vlc_err val = 2 * val + get_bits(l); % start with 3 bit val (smallest code) len = len + 1; [cbp, vlc_err] = get_cbp(len, val); if vlc_err, log_message('invalid coded block pattern'), err=l, return, end if cbp cbp = decimal_to_uimsbf(cbp, 6) ; no_match -- 0 ; end end if no_match, log_message('invalid coded block pattern'), err=l, return, end %fprintf('cbp = ')7fprintf('%d',cbp);fprintf('\n')

% decode [YO Y1 Y2 Y3 Cr4 Cb5] r = [1 8;1 8;9 16,-9 16;1 8;9 16]; % block rows c = [1 8; 9 16;1 8;9 16; 17 24;17 24]; % block cols decoded_mblock = zeros(16,24); for i = 1: 6 if cbp(i) [decoded_mblock (r ( i , 1) : r (i , 2 ) , c ( i , 1) : c (i , 2 ) ) , err] ... decode_ni_block (quantizer_scale); if err, log_message('error decoding b block'), err=l, return, end end end future_mblock = decoded_mblock + mcp_fwd; % recover original macroblock else % original macroblock was not coded future_mblock = mcp_fwd; % no received coefficient information end

%decoded_mblock,display_mblock,keyboard

% update DECODED_PIC store [top, left] = get_top_left(mb_addr); update_pic_store ('DECODED_PIC', future_mblock, top, left); return % following code for debugging - check decoded macroblock against reconstructed [top, left] = get_tl(mb_addr); rec_mblock = RECPIC(top:top+15,left:left+23); if ~ (rec_mblock-decoded_mblock) disp('Sc&&: rec_mblock = decoded_mblock') elseif -isempty(decoded_mblock) disp('&:&& rec_mblock 1= decoded_mblock' ) keyboard end function decode_p_picture ()

% set globals global START_CODE global USER_DATA_START_CODE global EXTENSION_START_CODE

% decode forward motion vector information full_pel_fwd = get_bits(1); % get full pel forward vector f orward_f_code = uimsbf_to_decimal (get_bits (3 ) ) ; % get f orward_f_code %fprintf{'full_pel_fwd = %d\n',full_pel_fwd); %fprintf('forward_f_code = %d\n',uimsbf_to_decimal(forward_f_code));

% discard any extra picture information while nextbits(l) get_bits(l); % discard extra_bit_picture = 1 get_bits(8); % discard extra_information_picture end get_bits(l); % discard extra_bit_picture = 0

% bring up next start code next_start_code;

% discard any picture extension data if nextbits(32) == EXTENSION_START_CODE get_bits(32); % discard EXTENSION_START_CODE while not_equal (nextbits(24), START_CODE) get_bits(8); % discard picture_extension_data end next_s tar t_c ode; end

% discard any user data if nextbits(32) == USER_DATA_START_CODE get_bits(32); % discard USER_DATA_START_CODE while not_equal (nextbits(24), START_CODE) get_bits(8); % discard user_data end next_start_code; end

% decode slices svp_.past = 1; % initialise for first slice while nextbits(24) == START_CODE slice_start_code = nextbits(32); % look at next 32 bits svp = uimsbf_to_decimal (slice_start_code (25:32)); % extract svp if svp < 1 I svp > 175 I svp < svp past % higher level start code or invalid svp return; end slice_start_code = get_bits(32); % get next 32 bits as slice code is valid err = decode_p_slice (svp, full_pel_fwd, forward_f_code); if err, next_start_code; end % error during decoding svp past - svp; % update svp_past end function err = decode_p_slice (svp, full_pel_fwd, forward_f_code) err =0; % error flag

% decodes slice layer for slices in p pictures global MB_WIDTH I global RECON_MV

% get quantizer_scale quantizer_scale = uimsbf_to_decimal (get_bits(5)) ; if ~(quantizer_scale) log_message ('invalid quantizer scale (quantizer_scale = 0 forbidden)') end

% discard any extra slice information while nextbits(l) get_bits(l); % discard extra_bit_slice = 1 get_bits(8); % discard extra_information_slice end get_bits(l); % discard extra_bit_slice = 0

% decode macroblocks inb_addr_prev = (svp - 1) * MB_WIDTH - 1; mv_fwd_prev = [0 0]; % fwd predition vector is zero at slice start dc_predictor = [1024 1024 1024]; % initialise for slice start intra_d =[0 0 0 1 1] ; % intra-d macrobloclc vie intra_q =[0000 01]; % intra-q macrobloclc vie h = waitbar(0,['decoding slice ',int2str(svp)]);i = 1; w]iile any (nextbits (23 ) ) waitbar(i/MB_WIDTH); i = i+1; % progress feedback % find macroblock address [mb_addr, addr_inc, addr_err] = decode_addr_inc (mb_addr_prev); if addr_err, log_message('error decoding mb_addr in P slice'), end

% process skipped macroblocks if addr_inc > 1 recover_p_skipped (mb_addr_prev, addr_inc) ; dc_predictor = [1024 1024 1024]; % reset dc predictors mv_fwd_prev =[0 0]; % reset motion vectors end

% decode macroblocks according to type [mb_type, err] = decode_p_mb_type; len = lengtln (mb_type) ; val = uimsbf_to_decimal (mb_type) ; [flags, type_err] = get_p_type (val, len); if type_err, log_message('error decoding mb_type in P slice'), end macroblock_quant = flags(1) motion_forward = flags(2) macroblock_intra = flags(5) if ~motion_forward, mv_fwd_prev = [0 0]; end %

% ctieck for intra-d or intra-q if ~macroblock_intra % decode predictively coded macroblock [mv_fwd__prev, err] = decode_p_mblock (mb_type, mb_addr, ... quantizer_scale, mv_fwd_prev, full_pel_fwd, forward_f_code) ; if err, log_mes sage ('error decoding p macroblock in P slice'), end dc_predictor = [1024 1024 1024]; % reset dc predictors RECON_MV(mb_addr+l, : ) = mv_fwd_prev;

else % decode intra coded macroblock %disp('intra macroblock detected ') [decoded_mblock, dc_predictor, err] = decode_i_mblock . . . (macroblock_quant, mb_addr, dc_predictor, guantizer_scale) ; if err, log_message('error decoding i macroblock in P slice'), end % update display buffer [top, left] = get_top_left (mb_addr); update_pic_store ('DECODED_PIC', decoded_mblock, top, left); mv_fwd_prev = [0 0]; % reset fwd predition vector end % update previous values mb_addr_prev = mb_addr; end close{h) % bring up next start code next_start_code; save RECON MV RECON MV function [pic_type, pic_no] = decode_pic_tYpe ()

% get picture start code picture_start = get_bits (3 2 )

% get temporal reference pic_no = uimsbf_to_decimal (get_bits(10));

% get picture coding type pic_type = uimsbf_to_decimal (get_bits(3)); if pic_type == 1 log_message(['** decoding picture I',int2str(pic_no)]); elseif pic_type == 2 log_message(['* * decoding picture P',int2str(pic_no)]) ; elseif pic_type == 3 log_message(['* * decoding picture B',int2str(pic_no)]) ; else log_message ('invalid picture coding type (not I, P or B)'); end % get vbv_delay (vbv_delay is not used at present) vbv_delay = uimsbf_to_decimal (get_bits (16) ) ; %fprintf ( ' vbv_delay = %d\n', vbv_delay) ; function decode_seq_header ()

% set globals global SEQUENCE_HEADER_CODE global HORIZ_SIZE global VERT_SIZE global EXTENSION_START_CODE global USER_DATA_START_CODE global START_CODE

% decode horizontal_size HORIZ_SIZE = uimsbf_to_decimal(get_bits(12)); if HORIZ_SIZE > 768 | HORIZ_SIZE < 16 error('Horizontal_size must be in [16,768]'); end

% decode vertical_size VERT_SIZE = uimsbf_to_decimal(get_bits(12)); if VERT_SIZE > 576 | VERT_SIZE < 16 error('Vertical size must be in [16,576]'); end log_message(['picture size : int2str(VERT_SIZE) ' x int2str(HORIZ_SIZE)]);

% decode pel aspect ratio pel_aspect_ratio = get_bits(4); if pel_aspect_ratio == [10 0 0] log_message('Pel aspect ratio 0.9375' ) ; else log_message(['Pel aspect ratio ',int2str(pel_aspect_ratio)]) end

% decode picture rate picture_rate = get_bits(4); if picture_rate == [0 0 11] log_message('Picture rate 25 pictures per second'); else log_message(['Picture rate ',int2str(picture_rate)]); end

% decode bit rate bit_rate = uimsbf_to_decimal(get_bits(18)); if bit_rate == 262143 log_message('Bit rate 3FFFF i.e. variable'); else log_message(['Bit rate ',int2str(bit_rate)]); end

% decode marlcer bit int2str(get_bits(1) )]) ; log_message ( [ ' Marlcer bit :

% decode vbv size vbv = uimsbf_to_decimal(get_bits(10) ) int2str(vbv)]); log_message(['VBV size : % decode constrained parameters flag cpb = get_bits(l); if cpb log_message ( ' Tliis is a constrained parameters bit stream'); else log_message ( ' Ttiis is NOT a constrained parameters bit stream'); end

% decode load_intra_quantizer_matrix flag load_iq_matrix = get_bits(l); if load_iq_matrix IQ_MATRIX = recover_matrix7 % recover zig-zag scan order values end

% decode load_non_intra_quantizer_matrix flag load_niq_matrix = get_bits(l); if load_niq_matrix NIO MATRIX = recover_matrix7 % recover zig-zag scan order values end next_start_code ;

% discard any extension data if nextbits(32) == EXTENSION_START_CODE get_bits(32); % discard EXTENSION_START_CODE while not_equal (nextbits(24), START_CODE) get_bits(8); % discard extension_data end next_start_code; end

% discard any user data if nextbits(32) == USER_DATA_START_CODE get_bits(32); % discard USER_DATA_START_CODE while not_equal (nextbits(24), START_CODE) get_bits(8); % discard user_data end next_start_code; end function disp_7 (fname, pic_no) fid_in = uvc_open(fname,'r'); [Y,U,V] = uvc_read_frame(fid_in,pic_no); picture = Yuv2picture (y,u,v); display_picture(picture); str = sprintf('Picture %d of file %s',pic_no, fname) title(str); fclose(fid_in); function (fname, pic_no) fid_in = uvc_open ( fname, 'r') ; [y,u,v] = uvc_read_frame(fid_in,pic_no); picture = yuv2picture (y,u,v); display__yuv__pic (picture) ; str = sprintf('Picture %d of file %s',pic_no, fname); title(str) ; fclose (fid_in) ; function display_picture (picture, pic_title) % displays picture % picture : 288x352 % pic_title : string for picture title if nargin == 1, pic_title = end

%figure; %truesize(size(picture)); I ^ mat2gray(picture, [0 255]); imshow(I, 256); title(pic_title); function display_j/uv_pic (picture, pic_title, map_length) if nargin == 1, pic_title = 'decoded picture'; map_length = 64; end if nargin == 2, map_length = 64; end % default map length 64

[y,u,v] = picture2yuv (picture); [r,g,b] = YUv2rgb(Y,u, v) ; R = r/256; % normalise to [0,1] G = g/256; B = b/256; [picture, map] = rgb2ind (R, G, B, map_length, ' dittier ' ) ; figure; truesize (size (picture) ) ; colormap (map) imshow(picture,map) ; title (pic_title) ; function H = entropy (A) % computes entropy for 8 bits/pixel picture A (256 pixel bins % John Bateman 31/8/94, 26/6/94

% scale elements of A from [0 255] to [0 1] I = mat2gray(A, [0 255]);

H = 0;

% return histogram n & bin location x for image I [n,x] = imhist{I);

% find image size [a b] = size(I); image_size = a*b;

% find length of histogram column vector 1 = length(n);

% compute entropy for i = 1:1 Pi = n(i)/image_size; if Pi ~= 0 H = H + Pi*log2(1/Pi); end end function boolean = even (numbers) % Description: % Tests if elements of a matrix are even or odd, Returns a matrix with % elements 1 where true and 0 where false. % % Input: % number : matrix of numbers in range [-2 048, 2047] i.e. 12 bits % % Output: % boolean: matrix of logical 0 or 1 for odd and even

[a b] = size(numbers); elements= a*b7 boolean == zeros(a,b);

% convert to binary & test Isb for i = 1:elements bin = decimal_to_uimsbf (abs(numbers(i)),12); if bin(12) == 0 boolean(i) = 1; % even else boolean(i) =0; % odd end end function [forward_f_code, modulus, forward_f_size, forward_f] - find_f_codes (mv, full_pel_fwd) % Description: % Finds f_code (forward or backward), modulus, forward_f_size & forward_f % by examining the range of the motion vectors, mv. % % Input: % 1. mv : 396x2 array of horizontal & vertical motion vectors % Array format is: % mv(row, coll, col2) = mv(mb_addr+l, mv_horiz, mv_vert); % 2. full_pel_fwd: 1 for ful pel, 0 for half pel accuracy motion vectors % % Output: % 1. forward_f_code : number 1-7 which determines number of bits appended % to differential motion vector vie. % 2. modulus: correction factor 32-2048 obtained from forward_f_code % 3. forward_fsize: forward_fsize = forward_f_code - 1 % 4. forward_f : forward_f = 1 << forward_fsize % % by John Bateman

% find forward_f_code and modulus smallest = min(min(mv)); % smallest motion vector largest = max(max(mv)); % largest motion vector if full pel_fwd % initialise for full pel low = -8; offset = 1; else % initialise for half pel low = -4; offset = 0.5; end modulus = 16; % starting value forward_f_code = []; for f = 1:7 low = 2 * low; high =: absdow + offset); modulus = modulus * 2; if smallest >= low & largest <= high forward_f_code = f; break; forward_f_code found, exit loop end end if isempty(forward_f_code), error('Motion vector out of range'), end

forward_f_size = forward_f_code - 1; forward_f = 2^^ (forward_f_size) ; function [varc, vard] = find_variance (pelc, pelp) % Description: % returns variance of the current macroblock & difference macroblock % % Input: % pelc : current macroblock % pelp : motion compensated macroblock % % Output: % varc : variance of (luminance component) of current macroblock % vard : variance of (luminance component) of difference macroblock % % John Bateman

% for difference macroblock vard = sum(sum((pelc - pelp).^2)) / 2 56; % assumes mean close to zero %fprintf ( ' vard== % . 2 f\n' , vard) ; % for current macroblock pel_sum = sum (sum (pelc) ) 7 varc = sum (sum (pelc . ^^2) ) ; varc = varc/256 - (pel_sum/256)*(pel_sum/256); %fprintf('varc= %.2f\n',varc); function [mcp, mv, mean_sq_err] = full_search (curr, prev, range) % Description: % Finds motion compensated prediction picture using full search technique % over search range (-range-0.5 to range+0.5). Also finds the % motion vectors to half pel accuracy.

Input: 1. curr current picture 2. prev previous picture 3 - range search range is -range-0.5 to range+0.5

Output: 1. mcp : motion compensated prediction picture (288x352) 2. mv: 396x2 array of horizontal & vertical motion vectors with format mv(row, coll, col2) = mv(mb_addr+l, mv_horiz, mv_vert); 3. mean_sq_err: array of mean square err values, mean_sq_err(row) = mse(mb_addr+l) John Bateman

% initialise values [size_rows, size_cols] size(curr); row_max size_rows - 16 + 1; row of top-left pel in last mb col_max size_cols - 16 + 1; col of top-left pel in last mb best_match zeros(16,16); best match mb in prev picture mcp zeros(size(curr)); mcp picture row 1; col 1; mb_addr -1; % start macroblock address mean sq err = zeros(size_rows*size_cols/256,1); h = waitbar(0finding motion vectors while row <= row_max waitbar(row/row_max) % progress feedback while col <= col_max % initialise values with first macroblock mb_addr - mb_addr + 1; %fprintf('motion estimation at macroblock address %d\n',mb_addr); curr_mblock curr((row:(row+15)),(col:(col+15))); prev_mblock prev((row:(row+15)),(col:(col+15))); min sum(sum(abs(curr_mblock - prev_mblock))) % mad best_match prev_mblock; mvh 0; mw 0;

block search through previous frame within +/-range pels from current macroblock for i = -range:range for j = -range:range if ~ (i==0 & skip centre position r = row+i; c = col+j; if (r>=l & r<=row_max & c>=l & c<=col_max) % keep within bounds prev_mblock = prev((r:(r+15)),(c:(c+15))); % search block mad = sum(sum(abs(curr_mblock - prev_mblock))); % mad if mad < min min = mad; % keep smallest mad mvh = j; % keep horizontal motion vector mw = i; % keep vertical motion vector end end end end end % do 0.5 pel search around best integer match [best_match, mv_half, mad_half] = ... half_pel_search (curr, prev, row+mvv, col+mvh, min); % store best match into motion compensated block, store motion vectors mcp ( (row: (row+15) ) , (col: (col4-15) ) ) = best_match; mv(mb_addr+l, 1) = mvh + mv_half(l); % store horiz motion vector in col 1 mv(mb_addr+l, 2) = mvv + mv_half(2); % store vert motion vector in col 2 % store mean_sci_err between current macromblock & mcp macroblock mean_sq_err(mb_addr+l) = mse (curr_mblock, best_match); col = col + 16; % move to next macroblock end row = row + 16; col = 1; move to next row of macroblocks end close(h) ; % remove waitbar function macroblock = get_YCbCr_mblock (picture, top, left) global HORIZ_SIZE global VERT_SIZE

% coordinates of u & v chrominance blocks tCr = top - 8*((top - 1)/16); % Cr top ICr = left - 8*((left - 1)/16) + HORIZ_SIZE; % Cr left tCb = tCr + VERT_SIZE/2; % Cb top ICb = ICr; % Cb left macroblock(:,1:16) = picture (top:top+15, left:left+15) Y macroblock(l:8,17:24) = picture (tCr:tCr+7, lCr:lCr+7); Cr macroblockO: 16, 17: 24)= picture (tCb:tCb+7, lCb:lCb+7); Cb function [addr_inc, err] = get_addr_inc (len, val) % returns addr_inc in (1,33) and err = 0 for valid addr_inc % returns addr_inc = 0 and err = 0 for addr_inc not found % returns addr_inc = [] and err = 1 for invalid addr_inc global ADDR_INC err =0; % set error flag false [rows, cols] = size(ADDR_INC); % get bounds if len > rows | val > cols err =1; % set error flag true return; elseif -len | -val addr_inc = 0 ; return; end addr_inc = ADDR_INC (len, val) ; function [vie, err] = get_ad.dr_inc_vlc (addr_inc) global ADDR_INC_VALUE ADDR_INC_LENGTH err = 0; if addr_inc < 1 | addr_inc > 3 3 err = 1; return; end val = ADDR_INC_VALUE(addr_inc); len = ADDR_INC_LENGTH(addr_inc); vie == decimal_to_uimsbf (val, len) ; function [flags, err] = get_b_tYpe (val,len) % description: % Finds macroblock_quant, motion_forward, motion_backward, macroblock_pattern % and macroblock_intra for b pictures % % input: % val : decimal value of vie code % len : length of vie code % % output: % flags is array of format: % [macroblock_quant, motion_forward, motion_backward, macroblock_pattern, ... % macroblock_intra] % % flags = 0 if code not valid found % err = 1 if vie length exceeds max code length, 0 otherwise

% check for valid length if len > 6 err =1; % vie not greater than 6 len flags =0; % vie not found return else err = 0 ; end

% setting flags for b pictures in formatas per Table 2-B.2c in format: % [macrobloek_quant motion_forward motion_backward macroblock_pattern ... % macrobloek_intra] if len == 2 & val == 2 % pred-i, vie = [10] flags =[01100]; elseif len == 2 & val == 3 % pred-ie, vie = [11] flags =[01110]; elseif len == 3 & val == 2 % pred-b, vie = [0 10] flags =[00100]; elseif len == 3 & val == 3 % pred-be, vie = [Oil] flags =[00110]; elseif len == 4 & val == 2 % pred-f, vie =[0010] flags =[01000]; elseif len == 4 & val == 3 % pred-fc, vie =[0011] flags =[01010]; elseif len == 5 & val == 3 % intra-d, vie =[0001 1] flags =[0 0 0 0 1]; elseif len == 5 & val == 2 % pred-ieq, vie =[0001 0] flags =[11110]; elseif len == 6 & val == 3 % pred-feq, vie =[0000 11] flags =[11010]; elseif len == 6 & val == 2 % pred-bcq, vie =[0000 10] flags =[10110]; elseif len == 6 & val == 1 % intra-q, vie =[0000 01] flags =[1 0 0 0 1]; else flags =0; % invalid code end function bits = get_bits (num_bits)

% BITSTREAM and I are global only to read_bitstream(), nextbits(), get_bits() % and push_bits so that the variables keep their value between function calls global BITSTREAM global I global EOF global FID global BLOCK_SIZE

% check for null input if ~num_bits bits - []; return7 end

% check for attempt to read past EOF if (I + num_bits - 1) > E0F*8 error('tried to read past end of file') end

% check if need new block from file if (I + num_bits - 1) > length(BITSTREAM) % keep unread bits from BITSTREAM if -isempty(BITSTREAM) remainder = BITSTREAM(I:length(BITSTREAM)); else remainder = []; end

% get next block of bytes from file file_ptr = ftell(FID); % get current position in file if (EOF - file_ptr) > BLOCK_SIZE bytes = BLOCK_SIZE; % take next block of bytes from file else bytes = (EOF - file_ptr); % take what's left from file end new_block = zeros(1,bytes*8); [buffer, count] = fread(FID, bytes, 'unsigned char');

% unpack 8 bit decimal numbers into 8 bits to form new bitstream fprintf('getting next %d bytes from file ...\n', bytes); for i = 1:bytes new_block ((i-1)*8+l:i*8) = decimal_to_uimsbf(buffer(i),8); end % join unread bits from last block & new block BITSTREAM = zeros(1,length(remainder)+length(new_block)); BITSTREAM = [remainder new_block]; 1=1; % reset BITSTREAM index end

% read bits & increment counter bits = BITSTREAMd: I + num_bits - 1); 1=1+ num_bits; function [cbp, err] = get_cbp (len, val)

% returns cbp in (1,63) and err = 0 for valid cbp % returns cbp = 0 and err = 0 for cbp not found % returns cbp = [] and err = 1 for invalid cbp global CBP err = 0; % set error flag false [rows, cols] = size(CBP); % get bounds if len > rows | val > cols err = 1; % set error flag true return; elseif -len | -val cbp = 0; return; end cbp = CBP(len,val); function [vie, err] = get_cbp_vlc (cbp) global CBP_VALUE CBP_LENGTH err = 0; if cbp < 1 I cbp > 63 err = 1; return; end val = CBP_VALUE(cbp); len = CBP_LENGTH(cbp); vie = deciinal_to_uimsbf {val, len) ; function vie = get_dct_vlc (run, level) global DCT_VLC_VALUE DCT_VLC_LENGTH if run < 0 | run > 63 | abs(level) >255 | level == 0 log_message('forbidden run and/or level value'); return; end val = DCT_VLC_VALUE(run+l,level+256); len = DCT_VLC_LENGTH(run+l,level+2 56); vie = decimal_to_uimsbf(val,len); function [mv, err] = get_mv (len, val)

% returns mv in (-16,16) and err = 0 for valid mv % returns mv = 99 and err = 0 for mv not found % returns mv = [] and err = 1 for invalid mv global MV_TABLE err = 0; set error flag false [rows, cols] = size(MV_TABLE) get bounds

if len > rows | val > cols err = 1; set error flag true return; elseif -len | -val mv = 997 return; end

mv = MV_TABLE(len,val); function [vie, err] = get_mv_vlc (mv) global MV_VALUE MV_LENGTH err = 0; if mv < -16 I mv > 16 err = 1; return; end val = MV_VALUE(mv+17) ; len = MV_LENGTH(mv+17) ; vie = decimal_to_uimsbf (val, len) ; function [flags, err] = get p type (val,len) % description: % Finds macroblock_quant, motion_forward, motion_backward, macroblock_pattern % and macroblock_intra for p pictures % % input: % val : decimal value of vie code % len : length of vie code % % output: % flags is array of format: [macroblock_quant motion_forward motion_backward.. , % macroblock_quant motion_forward motion_backward macroblock_pattern] % macroblock_intra] % flags = 0 if code not valid found % err = 1 if vie length exceeds max code length, 0 otherwise

% check for valid length if len > 6 err =1; % vie not greater than 6 len flags =0; % vie not found return else err = 0; end

% setting flags for p pictures in formatas per Table 2-B.2b in format: % [maeroblock_quant motion_forward motion_backward macroblock_pattern ... % macrobloek_intra] if len == 1 & val == 1 flags =[01010]; elseif len == 2 & val == 1 flags =[00010]; elseif len == 3 & val == 1 flags = [01000]; elseif len == 5 if val == 3 flags =[00001] elseif val == 2 flags = [110 10] elseif val == 1 flags = [10 0 10] else flags =0; % invalid code end elseif len == 6 & val == 1 flags =[10001]; else flags =0; % invalid code end function [run_level, vlc_err] = get_run_level (len, val global DCT_VLC_RUN global DCT_VLC_LEVEL vlc_err = 0 ; run_level = [0 0] ;

% check if run & level have exceeded table size [rows, cols] = size{DCT_VLC_RUN); if len > rows | val > cols vlc_err = 1; return end

% check if run & level are non-zero [rows, cols] = size(DCT_VLC_RUN); if -len I -val return; end

% get run & level run_level(l) = DCT_VLC_RUN(len,val); run_level(2) = DCT_VLC_LEVEL(len,val); function [slices, slice_length] = get_slices (picture)

% John Bateman 21/6/95

slice format is (where each Y represents 8x. Sc each Cr/Cb 8x44) slice 1 Y Y Y Y Y Y Y Y Cr Cr Cr Cr Y Y Y Y Y Y Y Y Cb Cb Cb Cb slice 2 Y Y Y Y Y Y Y Y Cr Cr Cr Cr Y Y Y Y Y Y Y Y Cb Cb Cb Cb

slice 18 Y Y Y Y Y Y Y Y Cr Cr Cr Cr Y Y Y Y Y Y Y Y Cb Cb Cb Cb global VERT_SIZE global HORIZ_SIZE global MB_WIDTH if size(picture) ~= [VERT_SIZE HORIZ_SIZE] 1og_message('error in function get_slices: invalid picture size') end

% separate picture into luminance & chrominance comp & make new picture [pic, pic_Cb, pic_Cr] = picture2yuv (picture); new pic = zeros(size(picture)); new_pic(:,1:HORIZ_SIZE) = pic;

% interleave Cr and Cb so that Cr & Cb are appended to each luminance slice chrom = zeros(VERT_SIZE, H0RIZ_SIZE/2); top_Cr = 1; top_Cb = 9; top_CbCr = 1; for i = 1:VERT_SIZE/16 chrom (top_Cr:top_Cr+7,:) = pic_Cr (top_CbCr:top_CbCr+7,:); % 8x176 Cr slice chrom (top_Cb:top_Cb+7,:) = pic_Cb (top_CbCr:top_CbCr+7,:); % 8x176 Cb slice top_Cr = top_Cr+16; top_Cb = top_Cb+16 7 top_CbCr = top_CbCr+8; end new_pic(:,H0RIZ_SIZE+1:(HORIZ_SIZE + HORIZ_SIZE/2)) = chrom;

% return slice length & combined Y and interleaved Cb/Cr picture slice_length = MB_WIDTH; % one slice per row slices = new pic; function [top, left] = get_top_left (macroblock_addr)

global MB_WIDTH row = fix (macroblock_addr / MB_WIDTH); % integer truncation col = macroblock_addr - fix(macroblock_addr/MB_WIDTH)*MB_WIDTH;

% top pel row and left pel col top = row*16+1; left = col*16+l; function [best_match, mv, mad] = half_pel_search (curr, prev, rE, cE, min) % % Description: % Finds half pel best match macroblock in prev picture for macroblock in % curr picture based on minimum absolute difference metric, and the half pel % motion vectors from the coords of the best integer pel match rE, cE.

Input: curr : current picture prev : previous picture rE : row of E (E is the best integer match macroblock in prev) cE : col of E (E is the best integer match macroblock in prev) min : min abs difference metric for best integer match macroblock E

Output: best match best match half pel macroblock in prev picture for macroblock in curr picture % mv(1) half pel horiz motion vector (-0.5, 0 or 0.5) % mv(2) half pel vert motion vector (-0.5, 0 or 0.5) % mad minimum absolute difference metric for best match

Note: The grid ABCDEFGHI is a 1 pel grid The grid JKLMENOPQ is a 0.5 pel grid

A B C J K L D M E N F 0 P Q G H I

= prev((rE:(rE+15)),(cE:(cE+15))); best_match = E; % initialise to integer best match mv = [0 0] ;

null_mblock zeros(16,16); [size_rows, size_cols] size(curr); row_max size_rows -16+1; % row of top-left pel in last mb col_max size_cols -16+1; % col of top-left pel in last mb prev_mblocks zeros(16, 144); curr mblocks zeros(16, 144);

k = 1; for i = -1 : 1 for j = -1 1 r = rE + i ; c = cE + j ; if r>=l Sc r<=row_max & c>=l & c<=col_max) % keep within bounds prev_mblocks(:,k:k+15) = prev((r:(r+15)) (c:(c+15))); % previous macrobloc curr_mblocks(:,k:k+15) = curr((r:(r+15)) (c:(c+15))); % current macrblock end k = k + 16; end end

% get previous macroblocks Ap = prev_mblocks(:,1:16); Bp = prev_mblocks(:,17:32) Cp = prev_mblocks(: , 33:48) Dp = prev_mblocks(: , 49 : 64) Ep = prev_mblocks(: , 65:80) Fp = prev_mblocks(: , 81:96) Gp = prev_mblocks(:,97:112); Hp = prev_mblocks(:,113:128); Ip = prev_mblocks(:,12 9:144); % get current macroblocks AC curr._mblocks( : ,1:16) ; Be curr._mblocks( : ,17:32) Cc curr _mblocks ( :,33:48 ) DC curr _mblocks(: ,49:64) EC curr _niblocks ( :,65:80 ) Fc curr _mblocks ( :,81:96 ) Gc curr _mblocks(: ,97:112); He curr _mblocks(: ,113:128); Ic curr _mblocks(: ,129 :144) ;

initialise all half pel macroblocks to zero JP null,.mblock ; Jc null, .mblock Kp null,.mblock ; Kc null, .mblock LP null,.mblock ; Lc null, .mblock Mp null,.mblock ; Mc null, .mblock Np null,.rnblock ; Nc null, .mblock Op null,.mblock ; Oc null, .mblock PP null,.mblock ; PC null, .mblock QP null .mblock; Qc null mblock

% linear interpolation to give 0.5 pel grid if Ac 0 & Be ~= 0 & Dc ~= 0 Jp = round((Ap+Bp+Dp+Ep)/4); Jc = round((Ac+Bc+Dc+Ec)/4); end if Be 0 Kp : round((Bp+Ep)/2); Kc : round((Bc+Ec)/2); end if Be 0 & Cc ~= 0 & Fc 0 Lp : round((Bp+Cp+Ep+Fp)/4); Lc : round((Bc+Cc+Ec+Fc)/4); end if Dc '= 0 Mp : round((Dp+Ep)/2) Me : round{(Dc+Ec)/2) end

if Fc '= 0 Np : round((Ep+Fp)/2) Nc : round((Ec+Fc)/2) end

if Dc 0 & Gc 0 & He 0 Op : round{(Dp+Ep+Gp+Hp)/4); Oc : round((Dc+Ec+Gc+Hc)/4); end

if He 0 Pp = round((Ep+Hp)/2) Pe = round((Ec+He)/2) end if Fc 0 & He ~= 0 & Ic 0 Qp = round((Ep+Fp+Hp+Ip)/4); Qc = round((Ee+Fc+Hc+Ie)/4); end

% arrange macroblocks into larger matrix for looping prev_half = [Jp Kp Lp Mp Np Op Pp Qp]; curr_half = [Je Ke Lc Mc Ne Oe Pc Qc] ; % find half pel motion vectors and best-match k = 1; for i = -0.5:0.5:0.5 for j = -0.5:0.5:0.5 if ~{i==0 & j==0) % don't search centre position curr_mt)lock = curr_half ( : , k: k+15) ; if curr_mblock 0 % skip any out of bounds blocks % then the macroblock exists (within picture) prev_mblock = prev_half(:,k:k+15) ; mad = sum(sum(abs(curr_mblock - prev_mblock))); % mad if mad < min min = mad; % keep smallest mad best_match = prev_mblock; % keep best match mv(l) = j; % keep horizontal motion vector mv(2) = i; % keep vertical motion vector end end k = k + 16; % move index to next macroblock end end end mad = min; % mad for best match % script init_coder_var % % description: % Initialises coder variables % % John Bateman 28/6/9 5

! rm mpegl.log p=path; path(P,'/eeicl5/icllib/mat/'); global RECON_MV global CODE_MCP global DECODE_MCP

% set global variables for coder global COUNT %% only temporary global IQ_MATRIX global NIQ_MATRIX global REMAINDER global BITSTREAM global I global HORIZ_SIZE global VERT_SIZE global MB_WIDTH global FID global S % conversion factor for MATLAB function dct2() global DCT_VLC_VALUE DCT_VLC_LENGTH DCT_VLC_RUN DCT_VLC_LEVEL global CBP_VALUE CBP_LENGTH CBP global ADDR_INC_VALUE ADDR_INC_LENGTH ADDR_INC global MV VALUE MV LENGTH MV TABLE global PREVIOUS_PIC previous picture store global FUTURE_PIC future picture store global DISPLAY_BUFFER next picture to be displayed global RECON_PIC reconstructed picture in encoder global DECODED_PIC decoded picture in decoder global MV_FWD global MCP_FWD MSE_FWD global MV_BWD global MCP_BWD MSE_BWD global MCP_INTERP MSE_INTERP global FULL_PEL_FWD FORWARD_F_CODE FORWARD_F FORWARD_F_SIZE MODULUS global FULL_PEL_BWD BACKWARD_F_CODE BACKWARD_F BACKWARD_F_SIZE MODULUS_BWD global RECPIC global FID_VIDEO_OUT global EOF global BLOCK SIZE

% set global variables for decoder

% set default quantisation matrices default_intra_q_matrix = [8 16 19 22 26 27 29 34 16 16 22 24 27 29 34 37 19 22 26 27 29 34 34 38 22 22 26 27 29 34 37 40 22 26 27 29 32 35 40 48 26 27 29 32 35 40 48 58 26 27 29 34 38 46 56 69 27 29 35 38 46 56 69 83] ; default_non_intra_q_matrix = 16 * ones(8,8);

% initialise variables IQ_MATRIX = default_intra_q_matrix; NIQ_MATRIX = default_non_intra_q_matrix; REMAINDER = []; % initially empty BITSTREAM = []; % initially empty I = 1; % bitstream index HORIZ_SIZE = 3 52; % picture width in pels VERT_SIZE = 288; % picture height in pels % no of macrowblocks per picture row MB_WIDTH = fix((H0RIZ_SIZE+15)/16); % current decoded picture in the encoder PREVIOUS_PIC zeros(VERT_SIZE, HORIZ_SIZE + H0RIZ_SIZE/2) FUTURE_PIC zeros(VERT_SIZE, HORIZ_SIZE + H0RIZ_SIZE/2) DISPLAY_BUFFER zeros(VERT_SIZE, HORIZ_SIZE + H0RIZ_SIZE/2) RECON_PIC zeros(VERT_SIZE, HORIZ_SIZE + H0RIZ_SIZE/2) DECODED_PIC zeros(VERT_SIZE, HORIZ_SIZE + H0RIZ_SIZE/2) RECPIC = zeros(288,528); % file ID for coder output file (bitstream) FID = -1; % file not open BLOCK SIZE = 10240;

% generate scaling factor to convert MATLAB dct2 to conventional % dct2 expression N = 8; S(l,l) = N; S(1,2:N) = ones(1,N-1)*N/sqrt(2) ; S(2:N,1) = ones(N-1,1)*N/sqrt(2); for u = 2:N for V = 2:N S(u,v) = N/2; end end S = 4*S; % script init_start_codes % % description: % Initialises start scodes % % John Bateman 28/6/95

% set globals global START_CODE global PICTURE_START_CODE global USER_DATA_START_CODE global SEQUENCE_HEADER_CODE global SEQUENCE_ERROR_CODE global EXTENSION_START_CODE global SEQUENCE_END_CODE global GROUP_START_CODE global MACROBLOCK_STUFFING global MACROBLOCK_ESCAPE

start codes START_CODE = [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1] ; % 0X000001 PICTURE_START_CODE = [ START_COD. E 0 0 0 0 0 0 0 0] % 0x00000100 USER_DATA_START_CODE = [ START_COD. E 1 0 1 1 0 0 1 0] % 0x000001B2 SEQUENCE_HEADER_CODE = [ START_COD. E 1 0 1 1 0 0 1 1] % 0x000001B3 SEQUENCE_ERROR_CODE = [ START_COD. E 1 0 1 1 0 1 0 0] % 0x000001B4 EXTENSION_START_CODE = [ START_COD. E 1 0 1 1 0 1 0 1] % OxOOOOOlBS S EQUENC E_END_C ODE = [ START_COD. E 1 0 1 1 0 1 1 1] % OxOOOOOlBV GROUP_START_CODE = [START._COD E 1 0 1 1 1 0 0 0] % OxOOOOOlBS

MACROBLOCK_STUFFING = [0 0 0 0 0 0 0 1 1 1 1] /

MACROBLOCK_ESCAPE -- [0 0 0 0 0 0 0 1 0 0 0] / %functio n Q = intra_quant (dct_coeffs, iqumatrix, quantizer_scale) % function Q = intra_quant (dct_coeffs, iq_matrix, quantizer_scale)

description: Returns quantised, rounded and clipped (where necessary) DCT coefficients. AC coefficients are clipped to the range -255 to 255. The DC coefficient is clipped to the range 0 to 255.

Whilst both the quantizer_scale and quantizer_matrix may vary for the AC coefficients, the DC coefficient is always obtained by dividing it by 8 and rounding. Note that for the AC coefficients, half integers (eg 2.5) are rounded towards 0 (i.e. to 2.0) to reduce the code size. AC coeffs with fractional parts > 0.5 are rounded towards +/- inf.

input: dct coeffs 8x8 real matrix of DCT coefficients 0<=DC coeff<=2040 -2048<=AC coeff<=2047 % iq_matrix 8x8 integer matrix for intra-quantization quantizer_scale : scaling factor to control bit rate (1 to 31)

output: Q 8x8 intra-quantized integer matrix 0<= DC coeff <=255 -255<= AC coeff <=255

% eg -0.5893 is rounded to -1, 2.231 is rounded to 2 % John Bateman 30/3/95, 26/4/95, 18/5/95

if quantizer_scale <1 | quantizer_scale >31 log_message('error in function quantize: function argument out of range'); end

% quantize and clip coefficients Q = (8*dct_coeffs)./(quantizer_scale * iq_matrix); % quantize AC coeffs Q(l,l) = round(dct_coeffs(1,1)/8); % quantize DC coeff if Q(l,l) < 0, Q(l,l) = 0; end % clip DC coeff to 0 Q = (Q <= 255).*Q + (Q > 255)*255; % clip all coeffs to 255 Q = (Q >= -255).*Q + (Q < -255)*(-255) ; % clip all coeffs to -255 % round number with fraction part <= 0.5 towards zero, others towards +/- inf Q = (abs(Q - fix(Q)) > 0 . 5) .*round(Q) + (abs(Q - fix(Q)) <= 0 . 5) .*fix(Q) ; %functio n id = inv_dct_2d (coeffs) % function id = inv_dct_2d (coeffs) % % desription: % Returns 2-D discrete cosine transform (DCT) for 8x8 block of pels. % Returns error message if block size is not 8x8. The DCT coefficients % are not checked for range as this is achieved in the quantize function % where the DCT coefficients are quantised, rounded and clipped (where % necessary).

input: coeffs 8x8 real matrix of DCT coefficients 0<=DC coeff<=2040 (not checked) -2048<=AC coeff<=2047 (not checked)

% output: % id : 8x8 integer matrix of pels % : each pel is an integer in the range % -255<=pel<=255 % % John Bateman 18/6/95 global S % conversion factor for MATLAB function dct2()

% 2-D inverse dct on 8x8 block id = round(idct2(coeffs.*S));

% find elements in id which are < -256 and clamp them to -256 id = (id >= -256).*id + (id < -256)*(-256);

% find elements in id which are > 255 and clamp them to 255 id = (id <= 255).*id + (id > 255)*255; %functio n iQ = inv_intra_quant (q_coeffs, iq_matrix, quantizer_scale) % function iQ = inv_intra_quant (q_coeffs, iq_matrix, quantizer_scale)

% description: % Performs inverse intra quantisation on coefficient matrix to produce iQ % % input: % q_coeffs : quantised coefficients % iq_matrix : 8x8 integer matrix for intra-quantization % quantizer_scale : scaling factor to control bit rate (1 to 31) % % output: % iQ : inverse intra quantised matrix

if quantizer_scale <1 | quantizer_scale > 31 log_message('error in function inv_intra_quant: argument out of range'); end % inverse quantize AC coefficients iQ = (2 * q_coeffs * quantizer_scale iq_matrix)/16;

% subtract 1 from the magnitude of even coeffs to avoid mismatch errors % eg 6 i.e. becomes 5 and -6 becomes -5 iQ = (~even(iQ)).*iQ + (even(iQ)).*(iQ - sign(iQ));

iQ = (iQ <= 2047).*iQ + (iQ > 2047)*2047; % clamp to 2047 iQ = (iQ >= -2048).*iQ + (iQ < -2048)*(-2048); % clamp to -2048 function iQ = inv_non_intra_quant (q_coeffs, niq_matrix, quantizer_scale) % % description: % Performs inverse intra quantisation on coefficient matrix to produce iQ. % % input: % q_coeffs : quantised coefficients % niq_matrix : 8x8 integer matrix for intra-quantization % quantizer_scale : scaling factor to control bit rate (1 to 31) % % output: % iQ : inverse intra quantised matrix if quantizer_scale <1 | quantizer_scale > 31 log_message('error in function inv_intra_quant: argument out of range'); end

% inverse quantize AC coefficients iQ = (((2 * q_coeffs) + sign(q_coeffs)) * quantizer_scale niq_matrix)/16;

% subtract 1 from the magnitude of even coeffs to avoid mismatch errors % eg 6 i.e. becomes 5 and -6 becomes -5 iQ = {-even(iQ)).*iQ + (even(iQ)).*(iQ - sign(iQ));

% clip iQ iQ = (iQ <= 2047).*iQ + (iQ > 2047)*2047; % clamp to 2047 iQ = (iQ >= -2048).*iQ + (iQ < -2048)* (-20 48) ; % clamp to -2048 function [deck, deck_size, map] = load_image_deck (filename, no_pics) if no_pics < 2, error('must have at least 2 pictures to make a video'), end

% read in middle picture middle = fix(no_pics/2); % get middle picture in video sequnce h = waitbar(0loading pictures into image deck...'); fid = uvc_open(filename, 'r'); [Y,U,V] = uvc_read_frame(fid,middle); % first picture

% convert to RGB and normalise RGB to [0,1] [R,G,B] = Yuv2rgb(Y,u,v); R = R/256 G = G/256 B = B/256

% convert RGB to index image with 240 level colormap 'map', % saving 16 levels for rest of display under MATLAB [picture, map] = rgb2ind(R,G,B,248,'dither'); % 248 colors, dithered

% load middle picture into image deck [m, n] = size(picture); deck = zeros(m, n*no_pics); % preallocate memory for image deck deck(imslice([m n no pics], middle+1) ) = picture; % load middle pic into deck waitbar(1/no pics);

% load rest of pictures into image deck for i = 0:no pics-1 if i ~= middle [y,u,v] = uvc_read_frame(fid, i) ; % next picture [r,g,b] = yuv2rgb(y,u,v); R = r/256; % normalise to [0,1] G = g/256; B = b/256; picture = rgb2ind(R,G,B,map); % fit RGB to existing colormap deck(imslice([m n no pics], i + 1) ) = picture; % load picture into deck waitbar((i+l)/no_pics) end end deck_size = [m n no_pics]; close(h) save image_deck deck deck_size map function load_pics (filename) fid = uvc_open(filename, 'r'); [y,u,v] = uvc_read_frame(fid,4); % frame 4 y = y(:,5:356);u=u(:,3:178);v=v(:,3:178); %[y,u,v] = yuv2YCbCr (y,u,v); prev = yuv2picture (y,u,v); %display_yuv_pic (prev) %display_picture(prev)

[y,u,v] = uvc_read_frame(fid,5); % frame 5 y = y(:,5:356);u-u(:,3:17 8);v=v(:,3:17 8); %[y,u,v] = yuv2YCbCr (y,u,v); curr = yuv2picture (Y,U,V);

[y,u,v] = uvc_read_frame(fid,7); % frame 7 y = y(:,5:356) ;u=u(:,3:17 8) ;v=v(: ,3 :178) ; %[y,u,v] = yuv2YCbCr (y,u,v); future = yuv2picture (y,u,v); function load tables

run scripts which generate vie lookup tables load dct_vlc dct vie table load cbp_vlc cbp vie table load addr_inc_vlc mbloek addr ine vie table load mv vie load mv vie table save tables function log_message (text) % function log_message (text)

% append message to log file fid = fopen('mpegl.log', 'a');

%new_fid = 1; %% temporary send to screen if fid == -1 error(' error in function log: cannot open file "mpegl.log" '); return; else fprintf(fid,text); fprintf(fid,'\n'); fprintf(text); fprintf('\n'); end

% close log file status = fclose(fid); if status == -1 error(' error in function log: cannot close file "mpegl.log" '); return; end % script make_addr_inc_vlc % ADDR_INC_VALUE addressed as ADDR_INC_VALUE(addr_inc) % ADDR_INC_LENGTH addressed as ADDR_INC_LENGTH(addr_inc) % ADDR_INC addressed as ADDR_INC(addr_inc_length, addr_inc_value; clear fprintf('Defining macroblock address increment VLC codesXn');

% make vie tables ADDR_INC_VALUE = zeros(1,33) ; ADDR_INC_LENGTH = zeros(1,33);

val = [1,3,2,3,2, 3,2,7,6,11, 10,9,8,7,6 23, 22,21,20,19,18 35,34,33,32,31, 3 0,29,28,27,26, 25,24];

len(l:9) =[133445577]; len(10:15) = 8*ones(l,6); len(16:21) = 10*ones(1,6); len(22:33) = ll*ones(1,12);

for i = 1:33 ADDR_INC_VALUE(i) = val(i); ADDR_INC_LENGTH(i) = 1en(i); end

% make inverse tables max_len = 11; max_val = 35; ADDR_INC = zeros(max_len, max_val); for i = 1:33 val = ADDR_INC_VALUE(i); len = ADDR_INC_LENGTH(i); ADDR_INC(1en,va1) = i; end

save addr_inc_vlc ADDR_INC_LENGTH ADDR_INC_VALUE ADDR_INC clear pack return

% following code used to check vie tables by printing to file !rm addr_inc_vlc.out load addr_inc_vlc fid = fopen('addr_inc_vlc.out' , 'w')

for i = 1:33 val = ADDR_INC_VALUE(i); len = ADDR_INC_LENGTH(i); vie = decimal_to_uimsbf(val,len); addr_inc = ADDR_INC(len,val) ; fprintf(fid,'vie = ',val, addr_inc);fprintf(fid,'%d%d%d%d ',vlc) fprintf(fid, ' addr_inc = %2d, val = %2d\n',addr_inc,val) ; end % script make_cbp_vlc % CBP_VALUE addressed as CBP_VALUE(cbp) % CBP_LENGTH addressed as CBP_LENGTH(cbp) clear all fprintf('Defining CBP VLC codes\n') CBP_VALUE = zeros(1,63); CBP_LENGTH = zeros(1,63) ; pattern = [60,4,8,16,32, 12,48,20,40,28, 44,52,56,1,61, 2,62,24,36,3, 63,5,9,17,33, 6,10,18,34,7, 11,19, 35,13,49,21,41, 14,50,22,42,15, .. 51,23,43,25,37, 26,38,29,45,53, 57,30,46,54,58, 31,47,55,59,27, 39]; val = [7,13,12,11,10, 19,18,17,16,15, 14,13,12,11,10, 9,8,15,14,13, . 12,23,22,21,20, 19,18,17,16,31, 30,29, 28,27,26,25,24, 23,22,21,20,19 18,17,16,15,14, 13,12,11,10,9, 8,7,6,5,4, 7,6,5,4,3, 2]; lend) = 3; len(2:5) = 4*ones(l,4); len(6:17) = 5*ones(1,12); len(18:21) = 6*ones(l,4)7 len(22:29) = 7*ones(1,8); len{30:57) = 8*ones(1,28); len(58:63) = 9*ones(1,6); for i = 1:63 CBP_VALUE(pattern(i)) =val(i); CBP_LENGTH(pattern(i)) = len(i); end

% make inverse tables CBP = zeros(1,63) ; for i = 1:63 val = CBP_VALUE(i); len = CBP_LENGTH(i); CBP(len,val) = i; end save cbp_vlc CBP_LENGTH CBP_VALUE CBP clear all pack return

% following code used to check vie tables by printing to file !rm cbp_vlc.out fid = f open ('cbp_vlc . out' , 'w') for i = 1:63 j = pattern(i); val = CBP_VALUE(j); len = CBP_LENGTH(j); code = decimal_to_uimsbf(val,len); cbp = CBP(len,val); fprintf(fid,'code = ',val, cbp);fprintf(fid,'%d%d%d%d ',code); fprintf(fid,' cbp = %2d, val = %2d\n',cbp,val); end function mcp = make_chrom_mcp (prev, mv_chrom) % Description % Applys motion vectors to Cr or Cb chrominance component of previous % picture to form motion compensated prediction with half pel accuracy. % Returns 144x176 motion compensated prediction chrominance matrix. % Motion vectors are positive to the right and down. % % Input: % prev: previous chrominance component of picture (Cr or Cb) % mv_chrom : 396x2 array of horizontal & vertical chrominance motion vectors % Array format is: % mv(row, coll, col2) = mv(block_addr+l, mv_horiz, mv_vert); % % Output: % mcp : chrominance motion compensated prediction (144x17 6) % % by John Bateman

% initialise values [size_rows, size_cols] size(prev); row_max size_rows -8+1 row of top-left pel in last block col_max size_cols -8+1 col of top-left pel in last block mcp zeros(size(prev)) mcp picture row 1; first row col 1; first column block addr -1; start block address h = waitbar(0forming mcp of chrominance component ...'); while row <= row_max while col <= col_max waitbar(row/row_max) % progress feedback % get current mcp block and block address block_addr = block_addr + 1; mvh = mv_chrom(block_addr+l, 1) ; mw = mv_chrom (block_addr+l, 2) ;

% bounds of best match chrominance block top = row + mw; bottom = top + 7; left = col + mvh; right = left + 7;

% Apply motion vectors if result is within bounds of picture if (left>=l & right<=size_cols & top>=l & bottom<=size_rows) half_row = (top - fix(top)) ==0.5; % true for half pel mv half_col = (left - fix(left)) ==0.5; % true for half pel mv

% retrieve bestmatch from half pel/full pel motion vectors if half_row & half_col % half pel row & col tA top - 0.5; bA = tA + 7; lA = left - 0.5; rA = lA + 7; A prev(tA:bA, lA:rA); % top left integer block B prev(tA:bA, lA+l:rA+l); % top right integer block C prev(tA+l:bA+l, lA:rA); % bottom left integer block D prev(tA+1:bA+l, lA+l:rA+l); % bottom right integer block mcp((row:(row+7)),(col:(col+7))) = round((A + B + C + D)/4); elseif half_row % half pel row, integer col tA = top - 0.5; bA = tA + 7 lA = left; rA = lA + 7; A = prev(tA:bA, lA:rA); % top left integer block C = prev(tA+l:bA+l, lA:rA % bottom left integer block mcp((row:(row+7)),(col:(col+7))) = round((A + C)/2) ; elseif half_col % integer row, half pel col tA = top; bA = tA + 7; lA = left - 0.5; rA = lA + 7; A = prev(tA:bA, lA:rA); % top left integer block B = prev(tA:bA, lA+l:rA+l); % top right integer block mcp((row:(row+7)),(col:(col+7))) = round((A + B)/2); else % integer row & col mcp((row:(row+7)),(col:(col+7))) = prev(top:bottom, left:right); end else log_inessage('motion vector out of bounds during chrominance mcp') end col = col +8; % move to next block end row = row + 8; col = 1; % move to next row of blocks end close (h) 7 % remove waitbar script make_dct_vlc

description: Generates matrices used for determining vie from dot run (of zeros) and level including all esacpe sequences as per Table 2-B5c-g. Generates matrices for determining run and level from vie and vie length (excluding escapes sequences)

input: nil

output: The following matrices are saved as dct vie.mat'

DCT_VLC_VALUE : (64,511) size matrix addressed as DCT_VLC_VALUE(run+1, level+256). value is decimal value of vie. DCT_VLC_LENGTH : (64,511) size matrix addressed as DCT_VLC_LENGTH(run+1, level+256). length is vie length in bits DCT VLC RUN : (max_len,max_val) size matrix addressed as DCT_VLC_RUN (vle_length, vlc_value). max_len is max max length in DCT_VLC_LENGTH array. max_val is max value in DCT_VLC_VALUE array. run is run of zeros since last non-zero dct coeff.

% DCT_VLC_LEVEL addressed as per DCT_VLC_RUN. Gives coeff level. % escape code test cases: run = 14, level = 13 0 DCT_VLC_VALUE 511193 8, DCT_VLC_LENGTH=:2 8 run = 14, level = -129 DCT_VLC_VALUE 5144703, DCT_VLC_LENGTH=2 8 run -- 3, level = -5, DCT_VLC_VALUE 17403, DCT_VLC_LENGTH=2 0 run = 3, level = 5, DCT VLC VALUE 17157, DCT_VLC_LENGTH=2 0

% John Bateman 22/9/95 fprintf('Defining DCT VLC codes\n') clear all; DCT_VLC_VALUE = zeros(64,511) % 511 possible levels per run DCT_VLC_LENGTH = zeros(64,511) % initialise all code lengths to 0 escape_2 0_bit = 2-^14; % escape value for 20 bit escape code escape_2 8_bit = 2^22; % escape value for 28 bit escape code

% load escape sequences for run = 0:63 for level = -255:255 if level -128 DCT_VLC_VALUE (run+1,level+256) = eseape_28_bit + run*2^16 ... + 2^15 + (level+256); DCT_VLC_LENGTH (run+1,level+256) = 28; elseif level > -128 & level < 0 DCT_VLC_VALUE (run+1,level+256) = escape_20_bit + run*2^8 ... + (level+256); DCT_VLC_LENGTH (run+1,level+256) = 20; elseif level > 0 & level < 128 DCT_VLC_VALUE (run+1, level + 2 5 6) = escape_2 0_bi t + run*2'^8 ... + level; DCT_VLC_LENGTH (run+1,level+256) = 20; elseif level > 128 DCT_VLC_VALUE (run+1,level+256) = eseape_28_bit + run*2^16 ... + level; DCT_VLC_LENGTH (run+1,level+256) = 28; % else do nothing when level = 0 (DCT_VLC_VALUE initialised to 0) end end end % load DCT_VLC_VALUE(run+1, level+256) matrix with decimal equivalent % of vie and DCT_VLC_LENGTH{run+1, level+2 56) matrix with vie length % in bits DCT_VLC_VALUE(1,216:296) = [33 , 35,37,39,41,43,45,47,49 , 33 , 35 , 37 , 39,41, 43 , . . . 45,47,49,51,53,5 5,57,59,61,63,47,49,51,53,3 3,3 9,49,5 9,21,67,77,13,11,9,7, .. . 0,6,8,10,12,7 6,6 6,2 0,58,48,3 8,3 2,52,50,48,46,62,60,58,5 6,54,52,5 0,48,4 6,... 44,42,40,3 8,3 6,3 4,3 2,48,46,44,42,40,3 8,3 6,3 4,32];

DCT_VLC_LENGTH(1,216: 29 6) = [ 16,16,16,16,16,16,16,16 , 16 , 15 , 15 , 15,15 , 15 , . . . 15,15,15,15,15,15,15,15,15,15,15,14,14,14,14,13,13,13,13,11,9,9,8,6,5, .. . 3,0,3, 5, 6, 8, 9, 9, 11, 13 , 13 , 13 , 13 , 14, 14, 14, 14, 15, 15, 15, 15, 15, 15, 15, 15, 15, .. . 15,15,15, 15, 15, 15,15,16,16,16,16,16,16,16,16,16] ;

DCT_VLC_VALUE(2,23 8:274) = [33,35,37,39,51,53,55,57,59,61,63,43,45,55,... 25,75, 13 , 7, 0,6,12,74,24,54,44,42,62,60,5 8,5 6,54,52,5 0,38,3 6,34,32] ;

DCT_VLC_LENGTH(2,23 8:274) = [ 17,17,17,17,16,16,16,16 , 16 , 16 , 16,14,14, . . . 13,11,9,7,4,0,4,7,9,11,13,14,14,16,16,16,16,16,16,16,17,17,17,17];

DCT_VLC_VALUE(3,251:2 61) = [41,41,23,9,11,0,10,8,22,40,40];

DCT_VLC_LENGTH(3,2 51:261) = [14,13,11,8,5,0,5,8,11,13,14];

DCT_VLC_VALUE(4,2 52:260) = [39,57,73,15,0,14,72,56,38];

DCT_VLC_LENGTH(4,252:2 60) = [14,13,9,6,0,6,9,13,14];

DCT_VLC_VALUE(5,2 53:259) = [37,31,13,0,12,30,36];

DCT_VLC_LENGTH(5,2 53:259) = [13,11,6,0,6,11,13];

DCT_VLC_VALUE(6,253:259) = [37,19,15,0,14,18,36];

DCT_VLC_LENGTH(6,253:259) = [14,11,7,0,7,11,14];

DCT_VLC_VALUE(7, 2 53 :259) [41,61,11,0,10,60,40];

DCT_VLC_LENGTH(7,253 :259) = [17,13,7,0,7,13,17];

DCT_VLC_VALUE(8,2 54:258) = [43,9,0,8,42];

DCT_VLC_LENGTH(8,2 54:258) = [13,7,0,7,13];

DCT_VLC_VALUE(9,2 54:258) = [35,15,0,14,34];

DCT_VLC_LENGTH(9,2 54:258) = [13,8,0,8,13];

DCT_VLC_VALUE(10,254:2 58) = [35,11,0,10,34];

DCT_VLC_LENGTH(10,2 54:258) = [14,8,0,8,14];

DCT_VLC_VALUE(11,254:258) = [33,79,0,78,32];

DCT_VLC_LENGTH(11,2 54:2 58) = [14,9,0,9,14];

DCT_VLC_VALUE(12,254:258) = [53,71,0,70,52];

DCT_VLC_LENGTH(12,2 54:258) = [17,9,0,9,17];

DCT_VLC_VALUE(13,2 54:258) = [51,69,0,68,50];

DCT_VLC_LENGTH(13,2 54:2 58) = [17,9,0,9,17];

DCT_VLC VALUE(14,2 54:258) = [49,65,0,64,48]; DCT_VLC_LENGTH(14,2 54:2 58) = [17,9,0,9,17];

DCT_VLC_VALUE(15,2 54:258) = [47,29,0,28,46];

DCT_VLC_LENGTH(15,2 54:258) - [17,11,0,11,17];

DCT_VLC_VALUE(16,2 54:258) = [45,27,0,26,44];

DCT_VLC_LENGTH(16,2 54:258) = [17,11,0,11,17];

DCT_VLC_VALUE(17,2 54:258) = [43,17,0,16,42];

DCT_VLC_LENGTH(17,2 54:258) = [17,11,0,11,17];

DCT_VLC_VALUE(18,2 55:257) - [63,0,62];

DCT_VLC_LENGTH(18,2 55:257) = [13,0,13];

DCT_VLC_VALUE(19,255:257) = [53,0,52];

DCT_VLC_LENGTH(19,2 55:257) = [13,0,13];

DCT_VLC_VALUE(20,255:257) = [51,0,50];

DCT_VLC_LENGTH(2 0,2 55:257) = [13,0,13];

DCT_VLC_VALUE(21,255:257) = [47,0,46];

DCT_VLC_LENGTH(21,2 55:2 57) = [13,0,13];

DCT_VLC_VALUE(22,2 55:2 57) = [45,0,44];

DCT_VLC_LENGTH(22,2 55:257) = [13,0,13];

DCT_VLC_VALUE(23,2 55:257) = [63,0,62];

DCT_VLC_LENGTH(23,2 55:257) = [14,0,14];

DCT_VLC_VALUE(24,2 55:257) = [61,0,60];

DCT_VLC_LENGTH(24,2 55:257) = [14,0,14];

DCT_VLC_VALUE(25,2 55:257) = [59,0,58];

DCT_VLC_LENGTH(25,2 55:257) = [14,0,14];

DCT_VLC_VALUE(26,2 55:2 57) = [57,0,56];

DCT_VLC_LENGTH(2 6,2 55:257) = [14,0,14];

DCT_VLC_VALUE(27,2 55:257) = [55,0,54];

DCT_VLC_LENGTH(27,2 55:2 57) = [14,0,14];

DCT_VLC_VALUE(28,255:2 57) = [63,0,62];

DCT_VLC_LENGTH(2 8,2 55:257) = [17,0,17];

DCT_VLC_VALUE(29,2 55:257) = [61,0,60];

DCT_VLC_LENGTH(2 9,2 55:257) = [17,0,17];

DCT_VLC_VALUE(3 0,2 55:257) = [59,0,58];

DCT_VLC_LENGTH(3 0,2 55:257) = [17,0,17]; DCT_VLC_VALUE(31,255:257) - [57,0,56];

DCT_VLC_LENGTH(31,2 55:257) = [17,0,17];

DCT_VLC_VALUE(32,255:257) - [55,0,54];

DCT_VLC_LENGTH(3 2,2 55:257) = [17,0,17];

% ma]ce inverse tables DCT_VLC_RUN (DCT_VLC_LENGTH, DCT_VLC_VALUE) % and DCT_VLC_LEVEL {DCT_VLC_LENGTH, DCT_VLC_VALUE) % note - escape codes not tiandled fprintf('Defining inverse VLC codes\n') max_len =17; % max lengtli in DCT_VLC_LENGTH array max_val =79; % max value in DCT_VLC_VALUE array DCT_VLC_RUN = zeros (max_len, max_val) ; DCT_VLC_LEVEL = zeros (max_len, max_val) ; for i = 1:64 for 3 = 1:511 len = DCT_VLC_LENGTH(i,j); if len < 20 & len % only consider non-escape & non-zero len val = DCT_VLC_VALUE(i,j); DCT_VLC_RUN(len,val) = i - 1; % get run DCT_VLC_LEVEL(len,val) = j-256; % get level end end end

% save variables to file 'dct_vlc.mat' fprintf('vlc tables saved to file "dct_vlc.mat"\n'); save dct_vlc DCT_VLC_LENGTH DCT_VLC_LEVEL DCT_VLC_RUN DCT_VLC_VALUE clear all pacJc %return %

% following code used to clieclc vie tables by printing to file Irm dct_vlc.out fid = fopen('dct_vlc.out', 'w'); load dct_vlc; % load tables fprintf(1, 'Printing dot vie codes to file "dct_vlc.out":\n\n') ; rl = [ 0 , 40 ; 1, 18 ; 2 , 5 ; 3 , 4 ; 4 , 3 ; 5 , 3 ; 6 , 3 ; 7 , 2 ; 8 , 2 ; 9 , 2 ; 10 , 2 ; 11, 2 ; . . . 12 , 2;13,2;14,2;15,2;16,2;17,1;18,1;29,1;20,1;21,1;22,1; - . . 23, 1;24,1;25,1;26,1;27,1;28,1;29,1;30,1;31,1] ; for i = 1: lengtli (rl) run = rl(i,l); no_levels = rl(i,2); for j = l:no_levels val = DCT_VLC_VALUE(run+1,256 + j) ; len = DCT_VLC_LENGTH{run+1,2 5 6 + j ) ; code = decimal_to_uimsbf(val,len) ; r = DCT_VLC_RUN(len,val); 1 = DCT_VLC_LEVEL(len,val); fprintf(fid,'r=%-2d, l=%3d, val=%-2d, len=%-2d, ir=%-2d, il=%3d, code= run,j, val,len,r,1); fprintf(fid,'%d', code);fprintf(fid,'\n');

3 = -j; val = DCT_VLC_VALUE(run+1,2 5 6 + j ) ; len = DCT_VLC_LENGTH(run+1,2 5 6 + j ) ; code = decimal_to_uimsbf(val,len) ; r = DCT_VLC_RUN(len,val); 1 = DCT_VLC_LEVEL(len,val); end end function mcp = make_incp (prev, mv) % Description % Applys motion vectors to previous picture to form motion compensated % prediction with half pel accuracy. Returns 288 x 352 motion compensated % prediction matrix. Motion vectors are positive to the right and down % % % Input: % prev: previous picture % mv : 396x2 array of horizontal & vertical motion vectors % Array format is: % mv(row, coll, col2) = mv(mb_addr+l, mv_horiz, mv_vert)• % % Output: % mcp : motion compensated prediction (288x352) % % by John Bateman

% initialise values [size_rows, size_cols] size(prev); row_max size_rows - 16 + 1; row of top-left pel in last mb col_max size_cols - 16 + 1; col of top-left pel in last mb mcp zeros(size(prev)); mcp picture row 1; first row col 1; first column mb addr -1; start macroblock address h = waitbar(0,'forming motion compensated prediction ...'); while row <= row_max while col <= col_max waitbar(row/row_max) % progress feedback % get current mcp macroblock and macroblock address mb_addr = mb_addr + 1; mvh = mv(mb_addr+l, 1); mw = mv(mb_addr+l, 2) ; % bounds of best match macroblock top = row + mw; bottom = top + 15; left = col + mvh; right = left + 15;

% Apply motion vectors if result is within bounds of picture if (left>=l Sc right< = size_cols & top>=l & bottom< = size_rows) half_row = (top - fix(top)) == 0.5; % true for half pel mv half_col = (left - fix(left)) ==0.5; % true for half pel mv

% retrieve bestmatch from half pel/ full pel motion vectors if half_row & half_col % half pel row & col tA = top - 0.5; bA = tA + 15; lA = left - 0.5; rA = lA + 15; A = prev(tA:bA, lA:rA); top left integer macroblock B = prev(tA:bA, lA+l:rA+l); % top right integer macroblock C = prev(tA+l:bA+l, lA:rA); % bottom left integer macroblock D = prev(tA+1:bA+l, lA+l:rA+l); % bottom right integer macroblock mcp((row:(row+15)),(col:(col+15))) = round((A + B + C + D)/4); elseif half_row % half pel row, integer col tA = top - 0.5; bA = tA + 15; lA = left; rA = lA + 15; A = prev(tA:bA, lA:rA); 5 top left integer macroblock C = prev(tA+1:bA+l, lA:rA); 5 bottom left integer macroblock mcp((row:(row+15)),(col:(col+15))) = round( (A + C)/2) ; elseif half_col % integer row, half pel col tA = top; bA = tA + 15; lA = left 0-5; rA = lA + 15; A = prev(tA:bA, lA:rA); top left integer macroblock B = prev(tA:bA, lA+l:rA+l); % top right integer macroblock mcp((row:(row+15)),(col:(col+15))) = round((A + B)/2); else % integer row & col mcp((row:(row+15)),(col:(col+15))) = prev(top:bottom, left:right); end else log_message('motion vector out of bounds') end col = col +16; % move to next macroblock end row = row + 16; col = 1; % move to next row of macroblocks end close(h); % remove waitbar % script make_mv_vlc % MV_VALUE addressed as MV_VALUE(mv+17) % MV_LENGTH addressed as MV_LENGTH(mv+17) % MV_TABLE addressed as MV_TABLE{mv_length, mv_value) clear fprintf('Defining motion vector VLC codes\n');

% make vie tables MV_VALUE = zeros(1,33) ; MV_LENGTH = zeros(l,33); val(l:16) = [2 5,27,2 9,31,3 3,3 5,19,21,23,7,9,11,7,3,3,3]; val(17) - 1; val(18:33) = val(16:-l:l) - 1; lend: 6) = ll*ones(l, 6) ; len(7:17) = [10 10 10 8 8 8 7 5 4 3 1]; len(18:33) = len(16:-1:1); for i = 1:33 MV_VALUE(i) = vaKi); MV_LENGTH(i) = len(i); end

% make inverse tables max_len = 11; max_val = 35; MV_TABLE = 99*ones(max_len, max_val); % fill with dummy values outside (-16,16) for i = 1:33 val = MV_VALUE(i); len = MV_LENGTH(i); MV_TABLE(len,val) = i - 17; end

save mv_vlc MV_LENGTH MV_VALUE MV_TABLE clear pack %return

% % following code used to check vie tables by printing to file !rm mv_vlc.out load mv_vlc fid = f open ('mv_vlc. out', 'w')

for i = -16:16 val = MV_VALUE(i+17); len = MV_LENGTH(i+17); vie = decimal_to_uimsbf(val,len); mv = MV_TABLE(len,val); fprintf(fid, 'vie = ',val, mv) ;fprintf(fid, '%d%d%d%d ',vlc); fprintf(fid,' mv = %2d, val = %2d\n',mv,val); end function integer = modulo (a, b)

% modulo operator is only defined for positive numbers, b is the modulo value. % eg modulo (395, 22) =21. A can be used for matrices also. if max(size(b)) > 1, error('divisor must be number not matrix'), end a = abs(a); b = abs(b); integer = a - fix(a/b)*b; function motion_estim_bwd (curr, future, range) % Description: % Performs backward motion estimation between curr and future pictures over a % range (-range-0.5 to range+0.5). Motion vectors and f-codes are saved to % file motion_vectors_bwd.mat. Motion compensated prediction (MCP_BWD) and % MSE saved to file motion_comp_bwd.mat

Input: 1. curr current picture (luminance & chrominance) 2. future future picture (luminance & chrominance) 3. range search range is -range-0.5 to range+0.5

Output: file motion_comp_bwd containing: a. MCP_BWD : backward motion compensated prediction picture (2 88x52 8) b. MSE_BWD: 3 96 element vector of mad cost functions for best match file motion_vectors_bwd.mat containing: a. MV_BWD: 3 9 6x2 array of horizontal & vertical motion vectors with format MV_BWD(row, coll, col2) = MV_BWD(mb_addr+l, mv_horiz, mv_vert); b. FULL_PEL_BWD: 0 for half pel units, 1 for integer pel units c. BACKWARD_F_CODE : number 1-7 which determines number of bits appended to differential motion vector vie. d. MODULUS_BWD: correction factor 3 2-2 048 obtained from BACKWARD_F_CODE e. BACKWARD_F_SIZE: BACKWARD_F_SIZE = BACKWARD_F_CODE - 1 f. BACKWARD_F : BACKWARD_F = 1 « BACKWARD F SIZE

curr_copy = curr; future_copy = future;

% separate pictures into luminance & chrominance components if size(curr) ~= size(future), error('pictures must be same size'), end [future,future_cb,future_cr] = picture2yuv (future); [curr,curr_cb,curr_cr] = picture2yuv (curr);

% perform full search with half pel accuracy [mcp_bwd, MV_BWD, MSE_BWD] = full_search (curr, future, range) ;

% find f codes FULL_PEL_BWD = 0; % half pel accuracy [BACKWARD_F_CODE, MODULUS_BWD, BACKWARD_F_SIZE, BACKWARD_F] = ... find_f_codes (MV_BWD, FULL_PEL_BWD); convert luminance motion vectors to chrominance motion vectors (chrominance motion vectors obtained by halving and rounding towards % half a pel i.e. values from [0 .. 0.5> go to 0 and [0.5 1> go to 0.5) temp = MV_BWD/2; mv_chrom = fix(temp); % truncate i = find((temp - mv_chrom) >= 0.5); j = find((temp - mv_chrom) <= -0.5); mv_chrom(i) = mv_chrom(i) + 0.5*ones(size(i)); % positive, add 0.5 inv_chrom(j) = mv_chrom(j) - 0 . 5*ones ( size (j ) ) ; % negativesubtrac. t 0.5

% make mcp for Cr and Cb chrominace components of picture & combine % with luminance component for storing in MCP_BWD mcp_cr = make_chrom_mcp (future_cr, mv_chrom) ; n^cp_cb = make_chrom_mcp (future_cb, mv_chrom) ; MCP_BWD = yuv2picture (mcp_bwd, mcp_cb, mcp_cr) ;

% convert half pel motion vectors to vectors with half pel units MV_BWD = MV_BWD * 2; % convert units of half pel curr = curr_copy; future = future_copy; %%% save motion_vectors_bwd MV_BWD FULL_PEL_BWD BACKWARD_F_CODE MODULUS_BWD BAGKWARD_F_SIZ E BAGKWARD_F save motion_comp_bwd MGP_BWD MSE_BWD function motion_estim_fwd (curr, prev, range) % description: % Performs forward motion estimation between curr and prev pictures over a % range (-range-0.5 to range+O.S). Motion vectors and f-codes are saved to % file motion_vectors_fwd.mat. Motion compensated prediction (MCP_FWD) and % MSE saved to file motion_comp_fwd.mat % % Input: % 1. curr : current picture (luminance and chrominance) % 2. prev : previous picture (luminance and chrominance) % 3. range : search range is -range-0.5 to range+O.S % % Output: % file motion_comp_fwd containing: % a. MCP_FWD : forward motion compensated prediction picture (288x528) % b. MSE_FWD: 3 96 element vector of mad cost functions for best match % file motion_vectors_fwd.mat containing: % a. MV_FWD: 396x2 array of horizontal & vertical motion vectors with format % MV_FWD(row, coll, col2) = MV_FWD(mb_addr+l, mv_horiz, mv_vert); % b. FULL_PEL_FWD: 0 for half pel units, 1 for integer pel units % c. FORWARD_F_CODE : number 1-7 which determines number of bits appended % to differential motion vector vie. % d. MODULUS: correction factor 32-2048 obtained from FORWARD_F_CODE % e. FORWARD_F_SIZE: FORWARD_F_SIZE = FORWARD_F_CODE - 1 % f. FORWARD_F : FORWARD_F = 1 « FORWARD_F_SIZE % separate pictures into luminance & chrominance components if size(curr) ~= size(prev), error('pictures must be same size'), end curr_copY = curr; % keep copy of curr prev_copy = prev; %%% [prev,prev_cb,prev_cr] = picture2yuv (prev); [curr,curr_cb,curr_cr] = picture2yuv (curr);

% perform full search with half pel accuracy [mcp_fwd, MV_FWD, MSE_FWD] = full_search (curr, prev, range);

% find f codes FULL_PEL_FWD =0; % half pel accuracy [FORWARD_F_CODE, MODULUS, FORWARD_F_SIZE, FORWARD_F] = ... find_f_codes (MV_FWD, FULL_PEL_FWD); % convert luminance motion vectors to chrominance motion vectors % (chrominance motion vectors obtained by halving and rounding towards % half a pel i.e. values from [0 .. 0.5> go to 0 and [0.5 .. 1> go to 0.5) temp = MV_FWD/2; mv_chrom = fix(temp); % truncate i = find((temp - mv_chrom) >= 0.5); j = find((temp - mv_chrom) <= -0.5); mv_chrom(i) = mv_chrom(i) + 0.5*ones(size(i)); % positive, add 0.5 mv_chrom(j) = mv_chrom(j) - 0.5*ones(size(j)); % negative, subtract 0.5

% make mcp for Cr and Cb chrominace components of picture & combine % with luminance component for storing in MCP_FWD mcp_cr = make_chrom_mcp (prev_cr, mv_chrom); mcp_cb - make_chrom_mcp (prev_cb, mv_chrom); MCP_FWD = yuv2picture (mcp_fwd, mcp_cb, mcp_cr);

% convert half pel motion vectors to vectors with half pel units MV_FWD = MV_FWD * 2; % convert units of half pel

% save to file curr = curr_copy; % get copy of original curr (luminance & chrominance) prev = prev_copy; %%% save motion_vectors_fwd MV_FWD FULL_PEL_FWD FORWARD_F_CODE MODULUS ... FORWARD_F_SIZE FORWARD_F save motion_comp_fwd MCP_FWD MSE_FWD curr function motion_estim_interp % description: % Performs interpolated motion compenstion by taking the average % of the forward and backward motion compensated predictions (MCP_FWD and % MCP_BWD) and saves the result to file motion_comp_interp.mat. Also % computes MSE for each macroblock in the interpolated prediction compared % with each macroblock in the current picture. % % Input: % 1. curr, MCP_FWD, MCP_BWD are loaded from files motion_comp_fwd.mat % & motion_comp_bwd.mat % % Output: % file motion_comp_interp containing: % a. MCP_INTERP : interpolated motion compensated prediction picture (288x3 52 % b, MSE_INTERP: 3 96 element vector of mean square error values for each % macroblock % John Bateman

% load MCP_FWD, MCP_BWD, curr from files load motion_comp_fwd load motion_comp_bwd

% separate curr into luminance & chrominance components [curr, curr_cb, curr_cr] = picture2yuv (curr);

% temporal (linear) interpolation MCP_INTERP = round((MCP_FWD + MCP_BWD)/2);

% find MSE for each macroblock in MCP_INTERP cf. each one in curr [size_rows, size_cols] = size(curr); row_max = size_rows -16+1; % row of top-left pel in last mb col_max = size_cols -16+1; % col of top-left pel in last mb row = 1; col = 1; mb_addr = -1; % start macroblock address MSE_INTERP = zeros(396,1); h = waitbar(0finding MSE for interpolative MCP ...'); while row <= row_max waitbar(row/row_max) % progress feedback while col <= col_max % initialise values with first macroblock mb_addr = mb_addr + 1; %fprintf('MSE at macroblock address %d\n',mb_addr); curr_mblock = curr( (row: (row+15)), (col: (col + 15) )) ; interp_mblock = MCP_INTERP((row:(row+15)),(col:(col+15))); % find mean square error MSE_INTERP(mb_addr+l) = mse (curr_mblock,interp_mblock); col = col +16; % move to next macroblock end row = row + 16; col = 1; % move to next row of macroblocks end

close(h); % remove waitbar

save motion_comp_interp MCP_INTERP MSE_INTERP % script mpegl_code % % MPEG-1 coder % input - UVC file selected via menu % output - binary file selected via menu % presently configured to code 1 GOP (GOP type selected from menu) and then % decode the binary file. %

% John Bateman 28/6/95

% clear workspace of all local and global variables clear all pack

% choose input files k = menuCselect input file', ' susie . 411. sifcal. 411. sifexit') ; if k == 1 source_file = 'susie.411.sif'; output_file = 'susie.bin'; elseif k == 2 source_file = 'cal.411.sif'; output_file = 'cal.bin'; else return end

% choose gop format gop_type = menu('select gop format','I BP','IBBBP', ... 'IBBPBBPBBPBB P','exit'); if gop_type == 4, return, end gop_no = 1; % load initialisation data t=cputime; disp('loading initialisation data ...') init_start_codes; % initialise global start codes init_coder_var; % initialise global coder variables load tables % run scripts which generate vie lookup tables e=cputime-t; fprintf('initialisation of data = %.4f minutes\n\n',e/60);

% start log log_message('** START OF MPEGl LOG **'); log_message(['source_file ' source_file]); log_message(['output_file ' output_file]); log_message(['picture size ' int2str(VERT_SIZE) ' x int2str(HORIZ_SIZE)]); m = int2str(VERT_SIZE*H0RIZ_SIZE/(16*16)); log_message(['number of macroblocks : ' m]);

% code video sequence pic_start = 4; s=cputime; code_sequence (source_file, pic_start, output_file, ... gop_type, gop_no) f=cputime-s;

t=sprintf('Total coding time = %.4f minutes\n',f/60);log_message(t)

% end of log log_message('** END OF MPEGl LOG **'); % decode stuff input_file = 'susie.bin'; output_file = 'susie.uvc'; s=cputime; mpegl_decode (input_file, output_file) f=cputime-s; t=sprintf('Total decoding time = %.4f minutes\n',f/60);log_message(t) check_psnr clear all pack output_file = 'susie.uvC; [deck, deck_size, map] = load_image_deck (output_file,13); save susie deck deck_size map %%plaY_video (deck, deck_size, map) function mpegl_decode (input_file, output_file)

% MPEG-1 decoder % input_file - MPEG_1 coded binary file % output_file - UVC file selected via menu % % John Bateman 28/6/95 disp('loading initialisation data ...') init_start_codes; % initialise global start codes init_coder_var; % initialise global coder variables load tables % run scripts which generate vie lookup tables

% open coded binary file open_bin_file (input_file);

% open output uvc file FID_VIDEO_OUT = uvc_open(output_filewtrial coder output');

% decode video sequence % set default values for last decoded picture in previous gop last_picture = zeros(VERT_SIZE, HORIZ_SIZE + H0RIZ_SIZE/2)7 last_pic_type = 1; last_pic_no = []; next_start_code; while nextbits(32) == SEQUENCE_HEADER_CODE get_bits(32); decode_seq_header; while nextbits(32) == GROUP_START_CODE get_bits(32); % need last (I or P) picture in case gop is not closed (i.e. requires % last I/P in last gop to decode pictures in current gop [last pic, last_pic_type, last pic no] = ... decode_gop (last_pic, last_pic_type, last_pic_no); end end

% find sequence end code if nextbits(32) == SEQUENCE_END_CODE get_bits(32); log_message('Sequence end code detected'); end

% end of log log_message('** END OF MPEGl LOG **'); function m = mse (reference, picture) % computes mean square error for difference % between two pictures % John Bateman 26/6/95

% find image size difference = reference - picture; [a b] = size(difference)7 image_size = a*b;

% find mean square error m = sum(sum(difference. )/image_size; function next_start_code () % function next_start_code () % % next_start_code removes any zero bit and zero byte stuffing and % locates the next start code global START_CODE

% move the current position in the bitstream forward until the next bit % to be decoded is the first bit in a byte %while ~(bytealigned) % get_bits(l); % remove '0' from bitstream %end

% move forward until the next 24 bits to be decoded are a start code while (not_equal(nextbits(24), START_CODE)) get_bits(8); % remove '0000 0000' from bitstream end function bits = read_bits (num_bits)

% BITSTREAM and I are global only to read_bitstream(), nextbitsO, get_bits() % and push_bits so that the variables keep their value between function calls global BITSTREAM global I global EOF global FID global BLOCK_SIZE

% check for null input if ~num_bits bits = []; return; end

% check for attempt to read past EOF if (I + num_bits - 1) > EOF*8 error('tried to read past end of file') end

% check if need new block from file if (I + num_bits - 1) > length(BITSTREAM) % keep unread bits from BITSTREAM if -isempty(BITSTREAM) remainder = BITSTREAM(I:length(BITSTREAM)); else remainder = [ ] ; end

% get next block of bytes from file file_ptr = ftell(FID); % get current position in file if (EOF - file_ptr) > BLOCK_SIZE bytes = BLOCK_SIZE; % take next block of bytes from file else bytes = (EOF - file_ptr); % take what's left from file end new_block = zeros(1,bytes*8 ) ; [buffer, count] = fread(FID, bytes, 'unsigned char');

% unpack 8 bit decimal numbers into 8 bits to form new bitstream fprintf('getting next %d bytes from file ...\n', bytes); for i = 1:bytes new_block ( (i-1)*8 + l:i*8) = decimal_to_uimsbf(buffer(i) ,8); end % join unread bits from last block & new block BITSTREAM = zeros(1,length(remainder)+length(new_block)); BITSTREAM = [remainder new_block]; 1=1; % reset BITSTREAM index end

% read bits but don't increment counter bits = BITSTREAMd: I + num_bits - 1); %functio n Q = non_intra_quant (dct_coeffs, niq_matrix, quantizer_scale) % % description: % Returns quantised, rounded and clipped (where necessary) DCT coefficients. % AC coefficients are clipped to the range -255 to 255. The DC coefficient is % clipped to the range 0 to 255. % % Whilst both the quantizer_scale and quantizer_matrix may vary for the % AC coefficients, the DC coefficient is always obtained by dividing it by % 8 and rounding. Note that for the AC coefficients, half integers (eg 2.5) % are rounded towards 0 (i.e. to 2.0) to reduce the code size. AC coeffs % with fractional parts > 0.5 are rounded towards +/- inf. % % input: % dct_coeffs : 8x8 real matrix of DCT coefficients 0<=DC coeff<=:2040 -2048<=AC coeff<=2047 % quantizer_matrix % : 8x8 integer matrix for intra-quantization % quantizer_scale : scaling factor to control bit rate (1 to 31) % % output: % Q : 8x8 intra-quantized integer matrix 0<= DC coeff <=255 -255<= AC coeff <=255 % % eg -0.5893 is rounded to -1, 2.231 is rounded to 2 % John Bateman 30/3/95, 26/4/95, 18/5/95 %

if quantizer_scale <1 | quantizer_scale > 31 log_message('error in function non_intra_quant: function argument out of range end % quantize and clip coefficients Q = fix((8 * dct_coeffs)./(quantizer_scale * niq_matrix)); % quantize AC coeffs Q = (Q <= 255).*Q + (Q > 255)*255; % clip all coeffs to 255 Q = (Q >= -255).*Q + (Q < -255)* (-255) ; % clip all coeffs to -255 function value = not_equal (a, b) % function value = not_equal (a, b) % % returns value = 1 if matrix a is not equal to matrix b and value = 0 % otherwise value = ~(sum(abs(a - b)) == 0); function open_bin_file (file_name) global BITSTREAM global I global EOF global FID global BLOCK_SIZE

%BITSTREAM = [] %file_name = 'test.bin'; %BLOCK_SIZE = 10 %fclose all

1=1; % initialise index to start of BITSTREAM array % open output file FID = fopen(file_name, 'rb'); if FID == -1, error('invalid file operation'), end

% find number of bytes in file status = fseek(FID,0,'eof'); % put file marker at end of file if FID == -1, error('invalid file operation'), end EOF = ftell(FID); % find number of bytes in file status = fseek(FID,0,'bof'); % put file marker at beginning of file fprintf('file %s contains %d bytesXn', file_name, EOF); function [y,u,v] = picture2Yuv (picture)

[r c] = size(picture); a = r/27 % [a b] is size of chrominance component of 4:1:1 picture b = c/3; y = picture(:, l:2*b); % luminance V = picture(l:a, 2*b+l:3*b); % red color difference u = picture(a+1:2*a, 2*b+l:3*b); % blue color difference function play_again (fig_no) load mov if -nargin fig_no = gcf; %colormap(map); truesize(deck_size(1:2)); end figure(fig_no); movie(mov, repetitions, fps); function play_video (deck, deck_size, map, fps, repetitions, fig_title) if nargin ~= 6 if nargin < 3, error('must supply deck, deck_size and map')/ end fig_title = ['video sequence of ',int2str(deck_size(3)),' pictures']; if nargin < 5 repetitions = -10; % play video 10 times end if nargin < 4 fps =25; % 25 frames per sec end end fig_title = [fig_title,' at ',int2str(fps),' pictures/second'];

figure; colormap (map) ; truesize(deck_size (1: 2)) ; disp('loading video sequence from image deck'); mov = immovie(deck, deck_size); title(fig_title); movie(mov, repetitions, fps); % play 10 times at -10 fps save mov mov repetitions fps map fig_title deck_size function p = psnr (reference, picture) % computes peak signal to noise ratio for difference diff % between two pictures % John Bateman 31/8/94, 26/6/95

% find image size difference = reference - picture; [a b] = size(difference); image_size = a*b;

% find mean square error and peak SNR mse = sum(sum(difference.^2))/image_size; p = 10*logl0(255^2/mse); function bits_read = read_bits (num_bits)

% BITSTREAM and I are global only to read_bitstream(), nextbitsO, get_bits() % and push_bits so that the variables keep their value between function calls global BITSTREAM global I global EOF global FID global BLOCK_SIZE

% check for illegal file access if (I + nii[n_bits - 1) > EOF*8 error{'tried to read past end of file') end

% check if need new block from file if (I + num_bits - 1) > length(BITSTREAM) % keep unread bits from BITSTREAM if -isempty(BITSTREAM) remainder = BITSTREAM(I:length(BITSTREAM) ) ; else remainder = []; end

% get next block of bytes from file file__ptr = ftell(FID) % get current position in file if (EOF - file_ptr) > BLOCK_SIZE bytes = BLOCK_SIZE % take next block of bytes from file else bytes = (EOF - file_ptr) % take what's left from file end new_block = zeros(1,bytes*8); % take what's left from file [buffer, count] = fread(FID, bytes, 'unsigned char')

% unpack 8 bit decimal numbers into 8 bits to form new bitstream for i = 1:bytes new_block ((i-1)*8 + l:i*8) = decimal_to_uimsbf(buffer(i) , 8) ; end

% join unread bits from last block & new block BITSTREAM = zeros(1,length(remainder)+length(new_block)); BITSTREAM = [remainder new_block]; 1=1; % reset BITSTREAM index end

% read bits & increment counter bits = BITSTREAMd: I + num_bits - 1) 1=1+ num bits function recover_b_skipped (mb_addr_prev, addr_inc, mb_tYpe__prev, . . - mv_fwd_prev, full_pel_fwd, mv_bwd_prev, full_jpel_bwd) global PREVIOUS_PIC % previous picture store global FUTURE_PIC % future picture store global DISPLAY_BUFFER % next picture to be displayed global DECODED_PIC

% find motion_forward & motion_backward based on previous macroblock type len = length(mb_type_prev);

val = uimsbf_to_decimal(mb_type_prev); [flags, type_err] = get_b_type (val, len); motion_f orward = flags(2); motion_backward = flags(3);

% recover skipped macroblocks from PREVIOUS_PIC/FUTURE_PIC skipped_addr = mb_addr_prev +1; % first skipped macroblock next_addr = mb_addr_prev + addr_inc; % next non-skipped macroblock while skipped_addr < next_addr %log_message(['\n** decoding SKIPPED macroblock ' int2str(skipped_addr)]); [top, left] = get_top_left (skipped_addr);

% forward motion compensation if motion_forward %fprintf{'mv_fwd_prev = [%d %d]\n', mv_fwd_prev); [mcp_fwd, mcp_err] = applY_mcp (PREVIOUS_PIC, top, left, mv_fwd__prev, . . . full_pel_fwd) ; if mcp_err log_message('error during forward motion compensation') err=l; return end end % backward motion compensation if motion_backward %fprintf('mv_bwd_prev = [%d %d]\n', mv_bwd_prev); [mcp_bwd, mcp_err] = apply_mcp (FUTURE_PIC, top, left, mv_bwd_prev, ... full_pel_bwd); if mcp_err log_message('error during backward motion compensation') err=l;return end end

% determine motion compensated prediction if motion_forward & motion_backward mcp_mblock = round((mcp_fwd + mcp_bwd)/2); % linear interpolation elseif motion_forward mcp_mblock = mcp_fwd; elseif motion_backward mc p_mblock = mcp_bwd; end

% update decoded picture buffer update_pic_store ('DECODED_PIC', mcp_mblock, top, left);

skipped_addr = skipped_addr + 1; % next skipped macroblock addr end function matrix = recover_matrix () % recovers 8x8 quantizer matrix from bitstream matrix = zeros(8,8);

% set up zig-zag scan sequence zz_index = [1 1;1 2;2 1;3 1;2 2,-1 3;1 4; 2 3;3 2;4 1;5 1;4 2;3 3;2 4;1 5;1 6;... 2 5 ; 3 4 ; 4 3 ; 5 2 ; 6 1; 7 1; 6 2 ; 5 3 ; 4 4 ; 3 5 ; 2 6 ; 1 7 ; 1 8 ; 2 7 ; 3 6 ; 4 5 ; 5 4 ; 6 3 ; . . . 7 2 ; 8 1; 8 2 ; 7 3 ; 6 4 ; 5 5 ; 4 6 ; 3 7 ; 2 8 ; 3 8 ; 4 7 ; 5 6 ; 6 5 ; 7 4 ; 8 3 ; 8 4 ; 7 5 ; . . . 6 6 ; 5 7 ; 4 8 ; 5 8 ; 6 7 ; 7 6 ; 8 5 ; 8 6 ; 7 7 ; 6 8 ; 7 8 ; 8 7 ; 8 8 ] ;

% zig-zag scan values into matrix for j = 1:64 matrix(zz_index(i,1) ,zz_index(i,2)) = uimsbf_to_decimal(get_bits(8) ) ; end function recover p.._skipped (mb_addr_prev, addr_inc) global PREVIOUS_PIC % previous picture store

% recover skipped macroblocks from PREVIOUS_PIC skipped_addr = mb_addr_prev +1; % first skipped macroblock next_addr = mb_addr_prev + addr_inc; % next non-skipped macroblock while skipped_addr < next_addr %log_message(['\n** decoding SKIPPED macroblock ' int2str(skipped_addr)]); [top, left] = get_top_left (skipped_addr); future_mblock = get_YCbCr_mblock (PREVIOUS_PIC, top, left); % copy macroblock from previous picture into DECODED_PIC store update_:pic_store ('DECODED_PIC', future_mblock, top, left); skipped_addr = skipped_addr +1; % next skipped macroblock addr end function C3_coeffs = rl_decode (run_level) % performs run_length decoding to give 8x8 block zz_index = zeros(64,2); q_coeffs = zeros(8,8);

% set up zig-zag scan sequence zz_index = [1 1;1 2;2 1;3 1;2 2;1 3;1 4;2 3;3 2;4 1;5 1;4 2;3 3;2 4;1 5;1 6;... 2 5 ; 3 4 ; 4 3 ; 5 2 ; 6 1; 7 1; 6 2 ; 5 3 ; 4 4 ; 3 5 ; 2 6 ; 1 7 ; 1 8 ; 2 7 ; 3 6 ; 4 5 ; 5 4 ; 6 3 ; . . . 7 2 ; 8 1; 8 2 ; 7 3 ; 6 4 ; 5 5 ; 4 6 ; 3 7 ; 2 8 ; 3 8 ; 4 7 ; 5 6 ; 6 5 ; 7 4 ; 8 3 ; 8 4 ; 7 5 ; . . . 6 6 ; 5 7 ; 4 8 ; 5 8 ; 6 7 ; 7 6 ; 8 5 ; 8 6 ; 7 7 ; 6 8 ; 7 8 ; 8 7 ; 8 8 ] ;

% run length decoding i = 0; % Start at first coefficient [rows, cols] = size{run_level); for j = 1:rows i = i + 1 + run_level(j,1); % get zz_index row; q_coeffs(zz_index(i,1),zz_index(i,2)) = run_level(j,2); end function q_coeffs = rl_decode_ac (run_level) % performs run length decoding on ac coeffs to give 8x8 block zz_index = zeros(64,2); q_coeffs = zeros(8,8);

% set up zig-zag scan sequence zz_index = [1 1;1 2;2 1;3 1;2 2;1 3;1 4;2 3;3 2;4 1;5 1;4 2;3 3;2 4;1 5;1 6;... 2 5;3 4;4 3;5 2;6 1;7 1;6 2;5 3;4 4;3 5;2 6;1 7;1 8/2 7;3 6;4 5;5 4/6 3;... 7 2;8 1;8 2; 7 3;6 4;5 5;4 6;3 7;2 8;3 8;4 7;5 6;6 5;7 4;8 3/8 4;7 5;... 6 6 ; 5 7 ; 4 8 ; 5 8 ; 6 7 ; 7 6 ; 8 5 ; 8 6 ; 7 7 ; 6 8 ; 7 8 ; 8 7 ; 8 8 ] ;

% run length decoding i = 1; % Start at first ac coefficient (row 1, col 2 of 8x8 block) [rows, cols] size (run_level) ; for j - 1:rows i = i + 1 + run_level(j,1)7 % get zz_index row; c3_coef fs (zz_index(i, 1) , zz_index(i, 2) ) = run_level (j , 2) ; end function [mc_mode, mcp] = select_mc_mode (mb_addr) % Description: % Selects motion compensation mode for non-intra macroblocks in B % pictures. The motion compensated (luminance) macroblock with the % smallest MSE compared with the current macroblock is selected as % having the best mode. In event of a tie, the order is: % 1. interpolative, 2. backward and then 3. forward to minimise the % vie code length for the mode type. The mcp for this mode is also % returned. global MSE_INTERP global MSE_BWD global MSE_FWD global MCP_INTERP global MCP_BWD global MCP_FWD global HORIZ_SIZE global VERT_SIZE

% get MSE values for best match macroblocks computed previously mse_interp = MSE_INTERP{mb_addr + 1); mse_bwd = MSE_BWD(mb_addr + 1); mse_fwd = MSE_FWD(mb_addr + 1);

% find coordinates for y macroblock & u & v blocks (top-left pel) [top, left] = get_top_left (mb_addr); % y top, left tv = top - 8*((top - 1)/16); % V top Iv = left - 8*((left - 1)/16) + HORIZ_SIZE; % v left tu = tv + VERT_SIZE/2; % u top lu = Iv; % u left

% select mc_mode with smallest MSE (interp is default) min = mse_interp; inc_mode = 1 ; mcp(1:16,1:16) = MCP_INTERP(top:top+15, left:left+15); % y mcp(1:8,17:24) = MCP_INTERP(tv:tv+7,Iv:lv+7); % v mcp(9:16,17:24) = MCP_INTERP(tu:tu+7,lu:lu+7); % u

% choose backward mode only if smaller if mse_bwd < min min = mse_bwd; mc_mode = 2 ; mcpd : 16, 1 : 16) = MCP_BWD( top: top+15, lefleft+15)t ; % y mcp(1:8,17:24) = MCP_BWD(tv:tv+7,Iv:lv+7) % V mcp(9:16,17:24) = MCP_BWD(tu:tu+7,lu:lu+7 ) % u end

% choose forward mode only if smaller if mse_fwd < min min = mse_fwd; mc_mode = 3; mcpd : 16, 1:16) = MCP_FWD( top: top+15, leftleft+15) ; % y mcpd : 8, 17: 24) = MCP_FWD (tv: tv+7 , Iv: lv+7 ) % V mcp(9:16,17:24)= MCP_FWD(tu:tu+7,lu:lu+7) % u end function show_modes (modes) % show mcp modes

A = 65; a = A + 32; i = find{modes==l); % i modes(i) = (a+8)*ones(size(i)); b = find(modes==2); % b modes(b) = (a+1)*ones(size(b)); f = find{modes==3); % f modes(f) = (a+5)*ones(size(f)); s = find(modes==4); % s modes(s) = (a+18)*ones(size(s)) ; I = find(modes==5); % I modes(I) = (A+8)*ones(size(I) ) ; N = find(modes==6); % N modes(N) = (A+13)*ones(size(N)) ; [r c] = size(modes); total = r*c; fprintf('** Matrix of %d macroblock motion compensation modes.\n',total); fprintf('** Nos. in brackets indicate percentage of that macroblock type.\n\n'); fprintf('i = interpolative prediction (%.If)\n',100*length(i)/total) fprintfCb = backward prediction (%. If) \n', 100*length (b)/total) fprintf('f = forward prediction (%.If)\n',100*length(f)/total) fprintf('I = intra coded (%.If)\n',100*length(I)/total); fprintfCs = skipped (%. If) \n', 100*length ( s)/total) ; fprintf('N = forward prediction but no motion vector (%.If)\n',... 100*length(N)/total) modes_out = 32 * ones(r,2*c); % load ' ' char modes_out(:,1:2:2*c) = modes(:,1:c); % load modes into every odd col mcp_modes = setstr(modes_out) %functio n decimal_no = simsbf_to_decimal (binary_no) % function decimal_no = simsbf_to_decimal (binarY_no) % % description: % Converts simsbf format binary number into a decimal number % % input: % binary_no : binary number in simsbf format % % output: % decimal_no : decimal number (integer) % % eg. [1 1110 0 1 0 1] is converted to decimal -27 % % John Bateman 11/6/95

% handle 0 case if sum(binary_no) == 0 decimal_no = 0; return; end

if binary_no(1,1) == 1 sign = -1; % negative number one = zeros(1,length(binary_no)); one (1, length (bina2ry_no) ) = 1; binary_no = ~{binary_no) + one; % form base 2 array binary_no = binary_no(2:length(binary_no)); % strip off sign bit else sign =1; % positive number end

% convert into decimal number by expanding &summing base 2 array no_of_bits = length{binary_no); decimal_no = 0; for i = l:no_of_bits decimal_no = decimal_no + binary_no(i) * 2^^ (no_of_bits - i); end % include the sign decimal_no = sign * decimal_no; function tell(pic) if isemptY(pic), fprintf('[] '), return, end pic_tYpe = pic(l); pic_no = pic(2); if pic_tYpe == 1 text = [ 'I',int2str(pic_no) , ' ']; elseif pic_tYpe == 2 text = ['P',int2str(pic_no) , ' ']; elseif pic_tYpe == 3 text = ['B',int2str(pic_no),' ']; else error('wrong pic type') end function intra = test_intra (pelc, pelp) % Description: % returns variance of the current macroblock & difference macroblock

% Input % pelc current macroblock % pelp motion compensated macroblock

% Output: % varc : variance of (luminance component) of current macroblock % vard : variance of (luminance component) of difference macroblock % % John Bateman

% for difference macroblock vard = sum (sum ( (pelc - pelp).'^2)) / 256; % assumes mean close to zero %fprintf('vard= %.2f\n',vard); % test if vard small enough for non-intra mode if vard <= 64 intra = 0 ; %disp('non-intra mode selected') return end

% for current macroblock pel_sum = sum(sum(pelc)); varc = sum ( sum (pelc . )) ; varc = varc/256 - (pel_sum/256)*(pel_sum/256); %fprintf('varc= %.2f\n',varc);

% test remaining conditions for intra / non-intra mode if varc >= vard intra = 0; %disp('non-intra mode selected') else intra = 1; %disp('** intra mode selected ') end function [mv_fwd, mcp] = test_mc (curr_Y, mb_addr) % Description: % Selects motion compensation mode for non-intra macroblocks in P % pictures. The motion compensated (luminance) macroblock with the % smallest MSE compared with the current macroblock is selected as % having the best mode. In event of a tie, the order is: % 1. interpolative, 2. backward and then 3. forward to minimise the % vie code length for the mode type. The mcp for this mode is also % returned. global MCP_FWD global HORIZ_SIZE global VERT_SIZE global MV_FWD global PREVIOUS_PIC mcp_Y = zeros(16,16) prev_Y = zeros(16,16) mcp = zeros(16,24)

% find coordinates for Y macroblock & Cb & Cr blocks (top-left pel) [top, left] = get_top_left (mb_addr); % Y top, left tCr = top - 8*((top - 1)/16); % Cr top ICr = left - 8*((left - 1)/16) + HORIZ_SIZE; % Cr left tCb = tCr + VERT_SIZE/2; % Cb top ICb = ICr; % Cb left

% get zero motion vector and mcp luminance macroblock prev_Y = PREVIOUS_PIC (top:top+15, left:left+15); % Y mcp_Y = MCP_FWD(top:top+15, left:left+15); % Y

% get sum of abs differences between curr & zero mv macroblock Z = sum(sum(abs(curr_Y - prev_Y))); X = Z/256; %fprintf('x = %3.1f\n',x);

% first test if zero motion vector case is small enough for no MC if X <= 1 % no MC mcp(:,1:16) = prev_Y; mcp(1:8,17:24) = PREVIOUS_PIC(tCr:tCr+7 , ICr:lCr+7) ; % Cr mcp(9:16,17:24)= PREVIOUS_PIC(tCb:tCb+7,ICb:lCb+7); % Cb mv_fwd = [0 0]; return; end

% examine remainder of conditions for MC/no MC M = sum(sum(abs(curr_Y - mcp_Y) ) ) ; Y = M/256; %fprintf('y = %3.1f\n',Y); ifx>l&x<3&Y>= x/2 % no MC mcp(:,1:16) = prev_Y; mcp(l:8,17:24) = PREVIOUS_PIC(tCr:tCr+7,ICr:lCr+7); % Cr mcp(9:16,17:24)= PREVIOUS_PIC(tCb:tCb+7,ICb:lCb+7); % Cb mv_fwd = [0 0]; return; elseif X == 3 & Y >= 1•5 % no MC mcp(:,1:16) = prev_Y; mcp(l:8,17:24) = PREVIOUS_PIC(tCr:tCr+7,ICr:lCr+7); % Cr mcp(9:16,17:24)= PREVIOUS_PIC(tCb:tCb+7,iCb:lCb+7); % Cb mv_fwd = [0 0]; return7 elseif X > 3 & y >= x/1.1 % no MC mcp(:,1:16) = prev_Y; mcp(l:8,17:24) = PREVIOUS_PIC(tCr:tCr + 7,ICr:lCr + 7 ) ; Cr mcp(9:16,17:24)= PREVIOUS_PIC(tCb:tCb+7,ICb:lCb+7); Cb mv_fwd = [0 0]; return; else % MC mcp(:,1:16) = mcp_Y; mcp{l:8,17:24) = MCP_FWD(tCr:tCr + 7,ICr:lCr + 7 ) ; % Cr mcp(9:16,17:24)= MCP_FWD(tCb:tCb+7,ICb:lCb+7); % Cb mv_fwd = MV_FWD (mb_addr + 1, : ) ; return; end function skipped = test_skipped (curr, iiib_addr, mc_mode, quantizer_scale, . . . mv_ fwd, mv_bwd) % ' global PREVIOUS_PIC % reconstructed previous I or P picture global FUTURE_PIC % reconstructed future I or P picture global FULL_PEL_FWD % 0 for half pel, 1 for full pel global FULL_PEL_BWD % 0 for half pel, 1 for full pel global RECON_PIC

% form motion compensated prediction macroblock using same form of % motion compensation & motion vectors as previous macroblock [top, left] = get_top_left (mb_addr); if mc_mode == 1 % interpolative prediction [mcp_fwd, mcp_err] = apply.mcp (PREVIOUS_PIC, top, left, mv_fwd, FULL_PEL_FWD; [mcp_bwd, mcp_err] = applY_mcp (FUTURE_PIC, top, left, mv_bwd, FULL_PEL_BWD); mcp = round((mcp_fwd + mcp_bwd)/2); % linear interpolation elseif mc_mode == 2 % backward prediction [mcp, mcp_err] = applY_mcp (FUTURE_PIC, top, left, mv_bwd, FULL_PEL_BWD); elseif mc_mode == 3 % forward prediction [mcp, mcp_err] = apply.mcp (PREVIOUS_PIC, top, left, mv_fwd, FULL_PEL_FWD); else error('Invalid motion compensation type - not interp, fwd or bwd') end % form difference macroblock macroblock = curr - mcp;

% code luminance blocks to determine coded block pattern % note memory is preallocated to mblock_buffer after iteration % to minimise execution time r = [1 87I 8;9 16,-9 16;1 8;9 16]; % block rows c = [1 8;9 16;1 8;9 16;17 24;17 24]; % block cols block = zeros (8,8); rec_mblock = zeros(16,24); mblock_buffer = []; % run length coded dct coeffs for i = 1:6 block = macroblock (r(i,l):r(i,2), c(i,l):c(i,2)); [rec_mblock(r(i,1) :r(i,2) , c (i,1) : c (i,2)), cbp(i), buffer] = ... code_ni_block (block, quantizer_scale); temp = mblock_buffer; mblock_buffer = zeros(1,length(mblock_buffer)+length(buffer)); mblock_buffer = [temp buffer]; end % check if any blocks coded if ~cbp skipped =1; % no blocks coded update_pic_store ('RECON_PIC', rec_mblock, top, left); else skipped = 0; % some blocks coded end %functio n decimal_no = uimsbf_to_decimal (binary_no) % function decimal_no = uimsbf_to_decimal (binary_no) % % description: % Converts uimsbf format binary number into a decimal number % % input: % binary_no : binary number in uimsbf format % % output: % decimal_no : decimal number (integer) % % eg. [0001 1011] is converted to decimal 27 % % John Bateman 18/5/95

% check for null input if isempty(binary_no) decimal_no - [ ] ; return; end no_of_bits = length(binary_no); decimal_no = binary_no(1); for i = 2:no_of_bits decimal_no - 2 * decimal_no + binary_no(i end function update_pic_store (pic_store, macroblock, top, left); global HORIZ_SIZE global VERT_SIZE global DISPLAY_BUFFER global PREVIOUS_PIC global FUTURE_PIC global RECON_PIC global DECODED_PIC global CODE_MCP global DECODE_MCP

% clip macroblock to [0,255] macroblock = (macroblock <= 255).*macroblock + ... (macroblock > 255)*255; % clip to 255 macroblock = (macroblock >= 0).*macroblock; % clip to 0

% coordinates of u & v chrominance blocks tv = top - 8*((top - 1)/16); % V top Iv = left - 8*((left - 1)/16) + HORIZ_SIZE; % v left tu = tv + VERT_SIZE/2; % u top lu = Iv; % u left

% insert macroblock into pic_store if strcmp(pic_store, 'DISPLAY_BUFFER'); DISPLAY_BUFFER (top:top+15, left:left+15) = macroblock(:,1:16);% y DISPLAY_BUFFER (tv:tv+7, lv:lv+7) = macroblock(1:8,17:24); % v DISPLAY_BUFFER (tu:tu+7, lu:lu+7) = macroblock(9:16,17:24); % u elseif strcmp(pic_store, 'PREVIOUS_PIC'); PREVIOUS_PIC (top:top+15, left:left+15) = macroblock(:,1:16); % y PREVIOUS_PIC (tv:tv+7, lv:lv+7) = macroblock(1:8,17:24); % v PREVIOUS_PIC (tu:tu+7, lu:lu+7) = macroblock(9:16,17:24); % u elseif strcmp(pic_store, 'RECON_PIC'); RECON_PIC (top:top+15, left:left+15) = macroblock(:,1:16); % y RECON_PIC (tv:tv+7, lv:lv+7) = macroblock(1:8,17:24); % v RECON_PIC (tu:tu+7, lu:lu+7) = macroblock(9:16,17:24); % u elseif strcmp(pic_store, 'DECODED_PIC'); DECODED_PIC (top:top+15, left:left+15) = macroblock(:,1:16); % y DECODED_PIC (tv:tv+7, lv:lv+7) = macroblock(1:8,17:24); % V DECODED_PIC (tu:tu+7, lu:lu+7) = macroblock(9:16,17:24); % u elseif strcmp(pic_store, 'CODE_MCP'); CODE_MCP (top:top+15, left:left+15) = macroblock(:, 1:16) ; % y CODE_MCP (tv:tv+7, lv:lv+7) = macroblock(1:8,17:24); % v CODE_MCP (tu:tu+7, lu:lu+7) = macroblock(9:16,17:24); % u elseif strcmp(pic_store, 'DECODE_MCP'); DECODE_MCP (top:top+15, left:left+15) = macroblock(:,1:16); % y DECODE_MCP (tv:tv+7, lv:lv+7) = macroblock(1:8,17:24); % v DECODE_MCP (tu:tu+7, lu:lu+7) = macroblock(9:16,17:24) ; % u

else error('invalid picture type') end functionfid = uvc_open ( filename, mode, comment) ; % %UVC_OPEBOpen UVC file. % % HD = UVC_OPEN(FILENAME,MODE,COMMENT) opens the UVC file % gecified by the string argument FILENAME. MODE may be one % d the strings: % % 'r' read % 'w' write (create if necessary) % % CM4ENT is an optional string argument which can be used to add % acomment to a new UVC file opened for writing. % % 5 the UVC file is to be opened for writing, a default header is % witten to the file with the number of frames et to zero. % % 1 the UVC file is to be opened for reading, the first four bytes % d the file are checked for the UVC file identifier. % % 2 the open is successful, FID gets a scalar MATLAB integer, the % file identifier, to be used as the first argument to other FilelO % 23utines. If the open was not successful, -1 is returned for FID. % % ^ example, to open the file containing the standard sequence % '-fobile and Calendar", with the filename /eeicl7/images/calendar_601 % fee reading: % % fid = uvc_open('/eeicl7/images/calendar_601','r'); % % TS open the new UVC file cdr_out.uvc for writing the output of a % tial coding process: % % fid = uvc_open('cdr_out.UVC','w','Trial coder output.') % % Q exit, FID points at the beginning of the next frame. % % S§e also UVC_READ_FRAME, UVC_WRITE_FRAME. %

if nargin~= 2 & nargin ~= 3 error ( 'SDmpulsory argument omitted. ' ) ; end

if strcmptnode, 'w') == 0 & strcmp (mode, ' r' ) == 0 error ( liacceptable MODE for uvc_open. ' ) ; end

if strcmp^ode, 'w' )

if nargii < 3 commefc = '' ; end

comm_s±E -- size (comment) ; if isst:<(comment) == 0 erroi^fCOMMENT must be a of type character string.'); end comm_leigth = length (comment) ;

fid = fpen ( filename, 'w+') ; if fid= -1 errorfFile could not be opened for writing.') end fseek(fid,0,-1);

fwrite(fid,1431716679,'uint'); fwrite(fid,2,'uint'); fwrite(fid,0,'uint'); fwrite(fid,64,'uint'); fwrite{fid,64+comm_length,'uint' fwrite(fid,720,'uint'); fwrite(fid,576,'uint'); fwrite(fid,0,'uint'); fwrite(fid,25.0,'float32'); fwrite(fid,2,'uint') fwrite(fid,0,'uint') fwrite(fid,1,'uint') fwrite(fid,1,'uint') fwrite(fid,8,'uint') fwrite(fid,1,'uint') fwrite(fid,2,'uint')

fwrite(fid,comment,'char'); fseek(fid,0,1); elseif strcmp(mode,'r') == 1

fid = fopen(filename,'r'); if fid == -1 error('File could not be opened for reading.') end

fseek(fid,0,-1) ; uvcg = fread(fid,1,'uint32'); if uvcg 1431716679 fclose(fid); error('Not a UVC file.'); end

% move the file pointer to the beginning of the video data fseek(fid,16,-1); video_offset = fread(fid,1,'uint'); fseek(fid,video_offset,-1); end function [Y,U,V] = uvc_read_frame(fid,frame_number) % %UVC_READ_FRAME Read a frame from a UVC file. % % [Y,U,V] = UVC_READ_FRAME(FID,FRAME_NUMBER) reads the frame specified % by FRAME_NUMBER from the UVC file specified by FID and returns the % luminance component in Y and the two chrominance components in U and V, % If U and V are not specified the luminance component only is returned. % % For example to read the 5 0th frame of the sequence "Mobile and % Calendar" contained in the file /eeicl7/images/calendar_601: % % fid = uvc_open('/eeicl7/images/calendar_601' , 'r'); % [y,u,v] = uvc_read_frame(fid, 50) ; % % If FRAME_NUMBER is not specified, the next frame is read % % On exit, FID points at the beginning of the next frame to be read. % % See also UVC_OPEN, UVC_WRITE_FRAME. %

if nargin == 1 frame_number = 0; elseif nargin 2 help uvc_read_frame error('Compulsory argument omitted.' ) ; end

cur_pos = ftell(fid);

fseek(fid,16,-1); video_offset = fread(fid,1,'uint'); width = fread(fid,1uint'); height = fread(fid,1,'uint'); max_frames = fread(fid,1,'uint'); fseek(fid,60,-1); structure = fread(fid,1,'uint');

if structure < 1 | structure > 4 error('This structure not supported by uvc_read_frame.'); end

if frame_number < 0 | frame_number > (max_frames-1) error('Frame number out of range.'); end

if structure == 1 offset 3*width*height*frame_number+video_offset; cw = width; ch = height; elseif structure == 2 offset = 2*width*height*frame_number+video_offset; cw = width/2; ch = height; elseif structure == 3 offset = 1.5*width*height*frame_number+video_offset; cw = width/2; ch = height/2; elseif structure == 4 offset = width*height*frame_number+video_offset; cw = 0; ch = 0; end if nargin == 1 status = fseek(fid,cur_pos,-1); else status = fseek(fid,offset,-1); end if status == -1 error('Could not find correct position in file.'); end y = fread(fid,[width,height],'uchar'); Y = Y' ; if structure ~= 4 u = fread(fid,[cw,ch],'uchar'); u = u' ; V = fread(fid,[cw,ch],'uchar'); V = v' ; end function frame_number = uvc_write_frame(fid,y,u,v) % %UVC_WRITE_FRAME Read a frame from a UVC file. % % FRAME_NUMBER = UVC_READ_FRAME(FID,Y,U,V) writes the luminance and % chrominance data specified by Y,U and V at the end of the UVC file % specified by FID and returns the total number of frames written to % the UVC file in FRAME_NUMBER. U and V are optional arguments and if % they are not specified, only the luminance component is written to % the UVC file. % % If the UVC file contains no previously written luminance and % chrominance data, when the first frame is written to the UVC file, % the header is updated to show the width, height and structure of % the first frame. For each subsequent frame, the width, height and % structure of the data is checked against the header of the UVC file % and if they are not identical, the frame is not written and an error % message is returned. % % For example, to write the first two frames of output from a trial % coder contained in the variables yl,ul,vl and y2,u2,v2 to the UVC % file coder_out.UVC: % % fid = uvc_open('cdr_out.UVC','w','Trial coder output.') % frame_number = uvc_write_frame(fid,yl,ul,vl); % frame_number = uvc_write_frame(fid,y2,u2,v2); % % The value of the variable FRAME_NUMBER will now be % % frame_number = % % 2 % % On exit, FID points at the beginning of the next frame to be written % % See also UVC_OPEN, UVC_READ_FRAME. % if nargin < 2 | nargin > 4 help uvc_write_frame error('Compulsory argument omitted.'); end

fseek(fid,20,-1); width = fread(fid,1,'uint'); height = fread(fid,1,'uint'); frame_number = fread(fid,1,'uint');

fseek(fid,60,-1); structure = fread(fid,1,'uint');

[frm_height,frm_width] = size(y); if nargin == 2 frm_structure = 4; elseif nargin == 4 lum_size = f 2rm_width* f rm_height ; [he,wc] = size(u); chr_size = wc*hc;

if lum_size == chr_size frm_structure = 1; elseif lum_size/2 == chr_size frm_structure = 2; elseif lum_size/4 == chr_size frm structure = 3; else error('This structure not supported by uvc_write_frame.'); end end if frame_number == 0

frame_number = frame_number+l;

fseek(fid,20,-1); fwrite(fid,frm_width,'uint'); fwrite{fid,frm_height,'uint'); fwrite(fid,frame_number,'uint');

fseek(fid,60,-1); fwrite(fid,frm_structure,'uint');

fseek(fid,0,1); y = y' ; fwrite(fid,Y,'uchar');

if frm_structure == 1 | frm_structure == 2 | frm_structure == 3 u = u' ; fwrite(fid,u,'uchar'); V = v' ; fwrite(fid,V,'uchar'); end else f raine_nuinber = f rame_number+l ;

if frm_strueture structure error('Frame structure not compatible with file.') elseif frm_width ~= width error('Frame width not compatible with file.') elseif frm_height height error('Frame height not compatible with file.') end

fseek(fid,28,-1) ; fwri te(fid,frame_number, 'uint');

fseek(fid,0,1); y = y' ; fwrite(fid,Y,'uchar');

if frm_structure == 1 | frm_structure == 2 | frm_structure == 3 u = u' ; fwrite(fid,u,'uchar'); V = V' ; fwrite(fid,V,'uchar'); end end %functio n bYtes_written = write_to_bitstream (buffer) % function bYtes_written = write_to_bitstream (buffer) % % description: % Takes input buffer of vlclbf code, packs each contiguous 8-bit block % into a byte, then writes these bytes to the binary file % 'bitstream.bin' in 'unsigned char' precision. Any leftover bits % at the end of the buffer (i.e. < 8 bits) are kept as a REMAINDER % and are added to the start of the next buffer to be written to file. % 'bitstream.bin' is opened and closed in the main workspace. % % input % buffer : variable length array of I's and O's in vlclbf format % % output: % the buffer is written byte-wise to binary file 'bitstream.bin' % bytes_written : number of bytes written to file 'bitstream.bin' % REMAINDER : the lasts few bits in the buffer which do not fit into % a byte - added to start of next buffer to be written % : REMAINDER is a global variable % % John Bateman 18/5/95, 10/6/95

% REMAINDER is a global variable to the base workspace and this function global REMAINDER global FID

% add leftover bits from last write to file to start of new buffer tot_buffer = zeros(length(REMAINDER) + length(buffer)); tot_buffer = [REMAINDER buffer]; buffer_size = length(tot_buffer); num_bytes = fix(buffer_size/8); % number of bits to number of bytes

% keep leftover bits (from unfilled byte) for next buffer REMAINDER = tot_buffer((num_bytes * 8 + 1):buffer_size);

output = zeros(1,num_bytes); % array of bytes index = 1; % index to output array

% convert each 8 bits of buffer to a decimal number and store for i = l:num_bytes output(i) = uimsbf_to_decimal(tot_buffer(index:i*8) ) ; index = index + 8; end bytes_written = fwrite(FID, output, 'unsigned char'); function [Y,Cb,Cr] = Yuv2YCbCr (Y,U,V)

% normalise to [0,1] Y = Y/256 u = u/256 Y = v/256

% scale Y to [16, 235] wliere 16=blac]c, 235=w]iite Y = 16 + 219 * Y;

% scale Cb & Cr to [0, 224] and use twos complement centred on 12 8 Cb = u * 224 - 128; Cr = V * 224 - 128;

fprintf('range Y = %.2f %.2f\n', min(min(Y)), max(max(Y))); fprintf('range Cb = %.2f %.2f\n', min(min(Cb)), max(max(Cb))); fprintf('range Cr = %.2f %.2f\n\n', min(min(Cr)), max(max(Cr))); function picture = Yuv2picture (Y,U,V)

[a, b] = size(u); picture = zeros(2*a, 3*b); picture(:, l:2*b) = y; % luminance picture{l:a, 2*b+l:3*b) = v; % Cr picture(a+l:2*a, 2*b+l:3*b) = u; % Cb function [r,g,b] = yuv2rgb(y,u, v) % %YUV2RGB YUV to RGB conversion. % % [R,G,B] = YUV2RGB(Y,U,V) converts the luminance and chrominance % components of an image contained in the matrices Y,U,V to the red, % green and blue intensities of the image contained in the matrices % R,G,B. % % The component matrices Y,U,V can be in either 4:1:1, 4:2:2 or 4:4:4 % format. For the 4:1:1 or 4:2:2 formats, the size of the color % component matrices are increased to the size of the luminance matrix % by pixel duplication. % % The conversion is performed using the matrix equations % % R = Y + 1.4030.*V % G = Y - 0.3440.*U - 0.7144.*V % B = Y + 1.7330.*U % % See also CIMAGE, RGB2IND. if nargin < 3 error('Compulsory input argument omitted.'); end if nargin > 3 error('Too many input arguments.'); end

[n,m] = size(y); [un,um] = s i z e(u) ; [vn,vm] = size(u); if un vn | um ~= vm error('Chrominance components must be equal size.'); end r = 0.*ones(n,m); g = 0.*ones(n,m); b = 0.*ones(n,m); ui = 0.*ones(n,m); vi = 0.*ones(n,m); u = u-128; v = v-12 8; if n*m/(un*um) == 4 vi(1:2:n,1:2:m) = v; vi(2:2:n,1:2:m) = v; vi(1:2:n,2:2:m) = v; vi(2:2:n,2:2:m) = v; ui(1:2:n,1:2:m) = u; ui(2:2:n,1:2:m) = U; ui(1:2:n,2:2:m) = u; ui(2:2:n,2:2:m) = u; elseif n*m/(un*um) == 2 vi(:,1:2:m) = V; vi(:,2:2:m) = v; ui(:,1:2:m) = u; ui(:,2:2:m) = u; elseif n*m/(un*um) == 1 vi - - V; ui = u; else error('This structure not supported by Yuv2rgb.'); end r = y+1.4030.*vi; g = y-0.344.*ui-0.7144.*vi; b = y+1.733.*ui;

[i,j] = find(r > 2 55) ; [ni,mi] = size(i); for k = l:ni r(i(k),j(k)) = 255; end [i,j] - find(r < 0); [ni,mi] = size{i); for k = l:ni r(i{k),j(k)) = 0; end [i,j] = find(g > 2 55); [ni,mi] = size(i); for k = l:ni g(i(k),j(k)) = 255; end [i,j] = find(g < 0); [ni,mi] = size(i); for k = l:ni g(i(k),j(k)) - 0; end [i, j] = find(b > 2 55) ; [ni,mi] = size(i); for k = l:ni b{i(k),j(k)) - 255; end [i,j] = find(b < 0); [ni,mi] = size(i); for k = l:ni b{i(k),j(k)) = 0; end %functio n zz = zig_zag (block) % function zz = zig_zag (block) % % description: % Puts elements of block into zig-zag scan order % % input: % block : 8x8 integer matrix % % output: % zz : 64 element integer array % % John Bateman 4/3/95, 18/5/95 % if size(block) ~= [8 8] log_message('error in function zig_zag: invalid block size') end % preallocate size zz_index = zeros(64,2); zz = zeros(1,64);

% set up zig-zag scan sequence zz_index = [1 1;1 2;2 1,-3 1;2 2; 1 3;1 4; 2 3;3 2; 4 1;5 1;4 2;3 3 ; 2 4;1 5;1 6; 2 5 ; 3 4 ; 4 3 ; 5 2 ; 6 1; 7 1; 6 2 ; 5 3 ; 4 4 ; 3 5 ; 2 6 ; 1 7 ; 1 8 ; 2 7 ; 3 6 ; 4 5 ; 5 4 ; 6 3 ; . . . 7 2;8 1;8 2; 7 3;6 4;5 5 7 4 6;3 7;2 8;3 8;4 7;5 6;6 5;7 4;8 3;8 4;7 5;... 6 6 ; 5 7 ; 4 8 ; 5 8 ; 6 7 ; 7 6 ; 8 5 ; 8 6 ; 7 7 ; 6 8 ; 7 8 ; 8 7 ; 8 8 ] ;

% rearrange into zig-zag scan order for i = 1:64 zz(i) = block(zz_index(i,1),zz_index(i,2)); end