VLSI implementation of a high speed LZW data compressor by Robert Lyle Wall A thesis submitted in partial fulfillment of the requirements for the degree of Master of Science in Electrical Engineering Montana State University © Copyright by Robert Lyle Wall (1991) Abstract: The growing volume of data stored and transmitted by computer is creating an increasing demand for efficient means of reducing the size of this data, while retaining all or most of its information content. This process is known as data compression, and it is frequently classified by whether the data recovered by the decompression process is always exactly the same as the original (lossless) or is allowed to vary from the original somewhat (lossy). This discussion will concentrate on lossless data compression methods. Today, most lossless data compression is still being performed in software. There are a small number of integrated circuits available which implement compression algorithms directly in hardware, but their execution speed is fairly limited. A design is presented for a VLSI integrated circuit which will perform lossless data compression at speeds roughly an order of magnitude greater than those currently available. It is based on the Lempel-Ziv-Welch (LZW) algorithm and relies on a high-speed content-addressable memory (also known as associative memory) to provide its performance increase. The process by which this algorithm has been subdivided into hardware modules is described, and the implementation of various modules using VLSI standard cell design techniques is presented. The similarities between standard cell circuit design and software development are examined, and the applicability of established software development methodologies to this type of design process is considered. Simulation and timing analysis of the modules suggests that the fully assembled circuit will be capable of compressing data at the rate of ten million bytes per second, which is nearly five times that of commercially available hardware, and the decompression rate will be only slightly less. VLSI IMPLEMENTATION OF A HIGH SPEED LZW DATA COMPRESSOR by Robert Lyle Wall - I' A thesis submitted in partial fulfillment of the requirements for the degree of Master of Science in Electrical Engineering MONTANA STATE UNIVERSITY Bozeman, Montana August 1991 AJ31$ APPROVAL of a thesis submitted by Robert Lyle Wall This thesis has been read by each member of the thesis committee and has been found to be satisfactory regarding content, English usage, for­ mat, citations, bibliographic style, and consistency, and is ready for submission to the College of Graduate Studies. , /^7 9 / Date Chairperson, Graduate Committee Approved for the Major Department Date Approved for the College of Graduate ? / f f / Date o 7 ! Graduate Dda iii STATEMENT OF PERMISSION TO USE In presenting this thesis in partial fulfillment of the requirements. for a master's degree at Montana State University, I agree that the Li­ brary shall make it available to borrowers under rules of the Library* Brief quotations from this thesis are allowable without special permis­ sion, provided that accurate acknowledgment of source is made. Permission for extensive quotation from or reproduction of this thesis may be granted by my major professor, or in his/her absence, by the Dean of Libraries when, in the opinion of either, the proposed use of the material is for scholarly purposes. Any copying or use of the material in this thesis for financial gain shall not be allowed without my written permission. Signature o/ _____ .____ Date 0,^ 4 / 49/______________________ Q ' iv ACKNOWLEDGEMENTS I would especially like to thank Professor Kel Winters for the guidance, suggestions, and inspiration he has provided. I also wish to thank Reza Massaratx for his work on the project. Thanks also to Jaye Mathisen, for his help with the OCT tools and UNIX problems, and to Rick Spickelmier, Andrea Cassotto, and Luciano Lavagno of the University of California at Berkeley for their support of OCT. Finally, thanks to Pat Owsley and others at Advanced Hardware Architectures for funding this research project. TABLE OF CONTENTS Page 1. INTRODUCTION .......... I Data Compression Definitions ................................ I E n t r o p y ..................... 5 Common Data Compression Algorithms ....................... 6 Huffman Coding . ............... 6 Arithmetic Coding .... .................. 8 Dictionary-Based Algorithms . ............. ...... 11 Other Data Compression Algorithms.................. 12 Choosing an Algorithm for Hardware Implementation ...... 13 2. LZW COMPRESSION AND DECOMPRESSION ALGORITHMS . 15 The Basic LZ78 Compression Algorithm . ........ .... 15 The LZW Algorithms ............. .............. 17 Compression Algorithm . ............. 21 Decompression Algorithm ........ ............. 24 Adaptation for High-Speed Hardware Implementation ........ 27 Compressor String Table Search ....................... 28 Decompressor String Reversal .......... ...... 29 Hardware Design Requirements ........ .. ........... .. 29 3. CONTENT-ADDRESSABLE MEMORY . ................. 31 Basic Content-Addressable Memory Characteristics .......... 31 Commercially Available CAMS ........ ............ 32 Custom Dynamic C A M .......................................... 34 Synchronous Mode-sensitive Operation ^ . 35 Predefined String Initialization . ............ 36 Reset Operation..................... 37 Refresh Operation ............. .......... 37 Possible Performance Enhancements ^ ................... 38 4. STRING REVERSAL MECHANISM ................... 40 String Reversal Algorithm ....................... .*.... 40 Physical Implementation Considerations . ................. 45 5. INPUT AND OUTPUT BUFFERING . 49 Character Buffers . ............. .. ........... 49 Code Buffers .................. • 51 vi (- TABLE OF CONTENTS— Continued Page f) 6. HARDWARE DESIGN PROCEDURE ..................... .............. 53 -i Target Technology.............. 54 CAD T o o l s ..................... .............. .............. 54 Bdsyn ..................... .............. 54 MisII . ... 56 Bdnet . i . ............... .............. .. 56 Wolfe . ................... ........................... 57 Musa ............. 58 Design Methodology ...... .......... ............... 58 External Reference Specifications ................. 59 RTL Descriptions ................... 59 BDS and Bdnet Input Files ........ 61 The Logic Generation P r o c e s s .......... 62 Logic Simulation ................................ 63 Naming Conventions . ...................... 64 Finite Field Sequencers .............. ........ 64 Problem Decomposition ....................... 66 7. THE COMPRESSOR M O D U L E ............... ........... .. ; . 68 Module Design for Normal DCAM Operation................. 68 External Reference Specifications . ............. 69 RTL Description........... ............................. 69 Module Generation ....................... 12 Module Testing ................................... .. .74 Module Design for Enhanced DCAM Operation . ................. 75 External Reference Specifications .... 75 RTL Description...................... ................. 75 Module Generation........................... 77 Module Testing ............ 79 8. THE DECOMPRESSOR MODULE ...................................... 80 Module Design to Accompany First Compressor . .......... 80 External Reference Specifications .... ............. 80 RTL Description...................... ........... .. 81 Module Generation ................................ 84 i: Module T e s t i n g ....... ........................ 86 Module Design to Accompany Single-Cycle Compressor . 86 External Reference Specifications............... 87 RTL Description ................. 87 Module Generation ...................................... 88 Module Testing .......... 90 vii TABLE OF CONTENTS— Continued Page 9. MERGING THE COMPRESSOR AND DECOMPRESSOR ................. .. 91 Additional Functionality .................................... 93 Halt Signal H a n d l i n g ..................... 93 String Length Limiting ..................... 95 Complete Merged Module .. ........................... 96 10. THE STRING REVERSAL MODULE .................................... 97 Module Controller Design ............... ................. 97 External Reference Specifications ................. 97 RTL Description........................... 98 Module Generation ...................................... 99 Module Testing ........................................ 99 Addition of Musa RAM Models . ................... ............ 99 11. THE MASTER CONTROLLER MODULE .................................. 103 Module Design ......... ..................................... 103 External Reference Specifications ...................... 103 RTL Description ...... ............................. 104 Module Generation ....................... 106 Module Testing ........................................ 106 System Integration and Test . '............................. 106 12. MANAGING THE DESIGN PROCESS .................................. 107 13. CIRCUIT PERFORMANCE ESTIMATES . ........................ H O Timing Extraction Using MisII . .......................... HO Timing Information for LZW Modules .......................... 112 Compression Efficiency Measurement ............... ..... 115 14. CONCLUSIONS ... .................................. ............ 117 Further Research ............. ............................... 118 REFERENCES CITED . ................. ........ ............