Vector Quantization and Signal Compression the Kluwer International Series in Engineering and Computer Science

VECTOR QUANTIZATION AND SIGNAL COMPRESSION THE KLUWER INTERNATIONAL SERIES IN ENGINEERING AND COMPUTER SCIENCE COMMUNICATIONS AND INFORMATION THEORY Consulting Editor: Robert Gallager Other books in the series: Digital Communication. Edward A. Lee, David G. Messerschmitt ISBN: 0-89838-274-2 An Introduction to Cryptolog)'. Henk c.A. van Tilborg ISBN: 0-89838-271-8 Finite Fields for Computer Scientists and Engineers. Robert J. McEliece ISBN: 0-89838-191-6 An Introduction to Error Correcting Codes With Applications. Scott A. Vanstone and Paul C. van Oorschot ISBN: 0-7923-9017-2 Source Coding Theory.. Robert M. Gray ISBN: 0-7923-9048-2 Switching and TraffIC Theory for Integrated BroadbandNetworks. Joseph Y. Hui ISBN: 0-7923-9061-X Advances in Speech Coding, Bishnu Atal, Vladimir Cuperman and Allen Gersho ISBN: 0-7923-9091-1 Coding: An Algorithmic Approach, John B. Anderson and Seshadri Mohan ISBN: 0-7923-9210-8 Third Generation Wireless Information Networks, edited by Sanjiv Nanda and David J. Goodman ISBN: 0-7923-9128-3 VECTOR QUANTIZATION AND SIGNAL COMPRESSION by Allen Gersho University of California, Santa Barbara Robert M. Gray Stanford University ..... SPRINGER SCIENCE+BUSINESS" MEDIA, LLC Library of Congress Cataloging-in-Publication Data Gersho, A1len. Vector quantization and signal compression / by Allen Gersho, Robert M. Gray. p. cm. -- (K1uwer international series in engineering and computer science ; SECS 159) Includes bibliographical references and index. ISBN 978-1-4613-6612-6 ISBN 978-1-4615-3626-0 (eBook) DOI 10.1007/978-1-4615-3626-0 1. Signal processing--Digital techniques. 2. Data compression (Telecommunication) 3. Coding theory. 1. Gray, Robert M., 1943- . II. Title. III. Series. TK5102.5.G45 1991 621.382'2--dc20 91-28580 CIP This book was prepared with U-TEX and reproduced by Kluwer from camera-ready copy supplied by the authors. Copyright © 1992 Springer Science+Business Media New York Eighth Printing 2001. Originally published by Kluwer Academic Publishers,New York in 1992 Softcover reprint ofthe hardcover Ist edition 1992 AII rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, mechanical, photo-copying, recording, or otherwise, without the prior written permission of the publisher, Springer Science+Business Media, LLC. Printed on acid-free paper. to Roberta & Lolly Contents Preface xiii 1 Introduction 1 1.1 Signals, Coding, and Compression 1 1.2 Optimality....... 8 1.3 How to Use this Book. 12 1.4 Related Reading . 13 I Basic Tools 15 2 Random Processes and Linear Systems 17 2.1 Introduction .......... 17 2.2 Probability............ 18 2.3 Random Variables and Vectors. 23 2.4 Random Processes. 26 2.5 Expectation .......... 29 2.6 Linear Systems ........ 32 2.7 Stationary and Ergodic Properties. 35 2.8 Useful Processes 39 2.9 Problems 42 3 Sampling 49 3.1 Introduction 49 3.2 Periodic Sampling . 50 3.3 Noise in Sampling. 57 3.4 Practical Sampling Schemes 60 3.5 Sampling Jitter . 65 3.6 Multidimensional Sampling. 67 3.7 Problems ....... 78 VB viii CONTENTS 4 Linear Prediction 83 4.1 Introduction............ 83 4.2 Elementary Estimation Theory . 84 4.3 Finite-Memory Linear Prediction 93 4.4 Forward and Backward Prediction. 98 4.5 The Levinson-Durbin Algorithm .. 104 4.6 Linear Predictor Design from Empirical Data 108 4.7 Minimum Delay Property ..... 112 4.8 Predictability and Determinism . 115 4.9 Infinite Memory Linear Prediction. 117 4.10 Simulation of Random Processes. 125 4.11 Problems .............. 125 II Scalar Coding 131 5 Scalar Quantization I 133 5.1 Introduction ............ 133 5.2 Structure of a Quantizer ..... 138 5.3 Measuring Quantizer Performance. 142 5.4 The Uniform Quantizer ...... 151 5.5 Nonuniform Quantization and Companding 156 5.6 High Resolution: General Case. 161 5.7 Problems ............ 168 6 Scalar Quantization II 173 6.1 Introduction .............. 173 6.2 Conditions for Optimality ...... 173 6.3 High Resolution Optimal Companding 185 6.4 Quantizer Design Algorithms . 187 6.5 Implementation 194 6.6 Problems..... .. 198 7 Predictive Quantization 203 7.1 Introduction..... 203 7.2 Difference Quantization ....... 204 7.3 Closed-Loop Predictive Quantization 206 7.4 Delta Modulation 214 7.5 Problems ............... 220 CONTENTS IX 8 Bit Allocation and Transform Coding 225 8.1 Introduction .......... 225 8.2 The Problem of Bit Allocation . 226 8.3 Optimal Bit Allocation Results ... 228 8.4 Integer Constrained Allocation Techniques 233 8.5 Transform Coding . .. 235 8.6 Karhunen-Loeve Transform ....... 240 8.7 Performance Gain of Transform Coding. 243 8.8 Other Transforms 245 8.9 Sub-band Coding 246 8.10 Problems .. 252 9 Entropy Coding 259 9.1 Introduction ............... 259 9.2 Variable-Length Scalar Noiseless Coding 261 9.3 Prefix Codes . 269 9.4 Huffman Coding .... 271 9.5 Vector Entropy Coding 276 9.6 Arithmetic Coding .. 277 9.7 Universal and Adaptive Entropy Coding 284 9.8 Ziv-Lempel Coding ........ 288 9.9 Quantization and Entropy Coding . 295 9.10 Problems .............. 302 III Vector Coding 307 10 Vector Quantization I 309 10.1 Introduction.... 309 10.2 Structural Properties and Characterization. 317 10.3 Measuring Vector Quantizer Performance. 323 lOA Nearest Neighbor Quantizers ........ 327 10.5 Lattice Vector Quantizers ..... .... 335 10.6 High Resolution Distortion Approximations 338 10.7 Problems ................... 340 11 Vector Quantization II 345 11.1 Introduction ......... 345 11.2 Optimality Conditions for VQ 349 11.3 Vector Quantizer Design 358 11.4 Design Examples 372 11.5 Problems......... 401 x CONTENTS 12 Constrained Vector Quantization 407 12.1 Introduction .......... 407 12.2 Complexity and Storage Limitations 408 12.3 Structurally Constrained VQ . 409 12.4 Tree-Structured VQ . 410 12.5 Classified VQ ..... .. 423 12.6 Transform VQ . .... 424 12.7 Product Code Techniques 430 12.8 Partitioned VQ .. 434 12.9 Mean-Removed VQ 435 12.10 Shape-Gain VQ .. 441 12.11 Multistage VQ ... 451 12.12 Constrained Storage VQ 459 12.13 Hierarchical and Multiresolution VQ 461 12.14 Nonlinear Interpolative VQ .... 466 12.15 Lattice Codebook VQ . 470 12.16 Fast Nearest Neighbor Encoding. 479 12.17 Problems . 482 13 Predictive Vector Quantization 487 13.1 Introduction ......... 487 13.2 Predictive Vector Quantization 491 13.3 Vector Linear Prediction .... 496 13.4 Predictor Design from Empirical Data 504 13.5 Nonlinear Vector Prediction 506 13.6 Design Examples 509 13.7 Problems .......... 517 14 Finite-State Vector Quantization 519 14.1 Recursive Vector Quantizers ...... 519 14.2 Finite-State Vector Quantizers ..... 524 14.3 Labeled-States and Labeled-Transitions . 528 14.4 Encoder/Decoder Design .. 533 14.5 Next-State Function Design 537 14.6 Design Examples 545 14.7 Problems ....... 552 15 Tree and Trellis Encoding 555 15.1 Delayed Decision Encoder 555 15.2 Tree and Trellis Coding .. 557 15.3 Decoder Design ..... 568 15.4 Predictive Trellis Encoders 573 CONTENTS Xl 15.5 Other Design Techniques 584 15.6 Problems......... 585 16 Adaptive Vector Quantization 587 16.1 Introduction......... 587 16.2 Mean Adaptation . ... 590 16.3 Gain-Adaptive Vector Quantization 594 16.4 Switched Codebook Adaptation 602 16.5 Adaptive Bit Allocation ..... 605 16.6 Address VQ . 611 16.7 Progressive Code Vector Updating. 618 16.8 Adaptive Codebook Generation 620 16.9 Vector Excitation Coding .... 621 16.10 Problems ............. 628 17 Variable Rate Vector Quantization 631 17.1 Variable Rate Coding ............. 631 17.2 Variable Dimension VQ ............ 634 17.3 Alternative Approaches to Variable Rate VQ . 638 17.4 Pruned Tree-Structured VQ .... 640 17.5 The Generalized BFOS Algorithm . 645 17.6 Pruned Tree-Structured VQ 652 17.7 Entropy Coded VQ . 653 17.8 Greedy Tree Growing .. 654 17.9 Design Examples .... 656 17.10 Bit Allocation Revisited 677 17.11 Design Algorithms. 682 17.12 Problems ........ 688 Bibliography 691 Index 720 Preface Herb Caen, a popular columnist for the San Francisco Chronicle, recently quoted a Voice of America press release as saying that it was reorganizing in order to "eliminate duplication and redundancy." This quote both states a goal of data compression and illustrates its common need: the removal of duplication (or redundancy) can provide a more efficient representation of data and the quoted phrase is itself a candidate for such surgery. Not only can the number of words in the quote be reduced without losing informa tion, but the statement would actually be enhanced by such compression since it will no longer exemplify the wrong that the policy is supposed to correct. Here compression can streamline the phrase and minimize the em barassment while improving the English style. Compression in general is intended to provide efficient representations of data while preserving the essential information contained in the data. This book is devoted to the theory and practice of signal compression, i.e., data compression applied to signals such as speech, audio, images, and video signals (excluding other data types such as financial data or general purpose computer data). The emphasis is on the conversion of analog waveforms into efficient digital representations and on the compression of digital information into the fewest possible bits. Both operations should yield the highest possible reconstruction fidelity subject to constraints on the bit rate and implementation complexity. The conversion of signals into such efficient digital representations has several goals: • to minimize the communication capacity required for transmission of high quality signals such as speech and images or, equivalently, to get the best possible fidelity over an available digital communication channel, • to minimize the storage capacity required for saving such information in fast storage media and in archival data bases or, equivalently, to get the best possible quality for the largest amount of information XlIl XlV PREFACE stored in a given medium, • to provide the simplest possible accurate descriptions of a signal so as to minimize the subsequent complexity of signal processing algorithms such as classification, transformation, and encryption.

Load more