University of Central Florida STARS Electronic Theses and Dissertations, 2004-2019 2005 Transform Based And Search Aware Text Compression Schemes And Compressed Domain Text Retrieval Nan Zhang University of Central Florida Part of the Computer Sciences Commons, and the Engineering Commons Find similar works at: https://stars.library.ucf.edu/etd University of Central Florida Libraries http://library.ucf.edu This Doctoral Dissertation (Open Access) is brought to you for free and open access by STARS. It has been accepted for inclusion in Electronic Theses and Dissertations, 2004-2019 by an authorized administrator of STARS. For more information, please contact [email protected]. STARS Citation Zhang, Nan, "Transform Based And Search Aware Text Compression Schemes And Compressed Domain Text Retrieval" (2005). Electronic Theses and Dissertations, 2004-2019. 419. https://stars.library.ucf.edu/etd/419 Transform Based and Search Aware Text Compression Schemes and Compressed Domain Text Retrieval by Nan Zhang B.S. Beijing Colloge of Economics, 1990 M.S. National University of Singapore, 1998 A dissertation submitted in partial ful¯llment of the requirements for the degree of Doctor of Philosophy in the School of Computer Science in the College of College of Engineering and Computer Science at the University of Central Florida Orlando, Florida Spring Term 2005 Major Professor: Amar Mukherjee °c 2005 by Nan Zhang Abstract In recent times, we have witnessed an unprecedented growth of textual information via the Internet, digital libraries and archival text in many applications. While a good fraction of this information is of transient interest, useful information of archival value will continue to accumulate. We need ways to manage, organize and transport this data from one point to the other on data communications links with limited bandwidth. We must also have means to speedily ¯nd the information we need from this huge mass of data. Sometimes, a single site may also contain large collections of data such as a library database, thereby requiring an e±cient search mechanism even to search within the local data. To facilitate the information retrieval, an emerging ad hoc standard for uncompressed text is XML which preprocesses the text by putting additional user de¯ned metadata such as DTD or hyperlinks to enable searching with better e±ciency and e®ectiveness. This increases the ¯le size considerably, underscoring the importance of applying text compression. On account of e±ciency (in terms of both space and time), there is a need to keep the data in compressed form for as much as possible. Text compression is concerned with techniques for representing the digital text data in alternate representations that takes less space. Not only does it help conserve the storage space for archival and online data, it also helps system performance by requiring less number iii of secondary storage (disk or CD Rom) accesses and improves the network transmission bandwidth utilization by reducing the transmission time. Unlike static images or video, there is no international standard for text compression, although compressed formats like .zip, .gz, .Z ¯les are increasingly being used. In general, data compression methods are classi¯ed as lossless or lossy. Lossless compression allows the original data to be recovered exactly. Although used primarily for text data, lossless compression algorithms are useful in special classes of images such as medical imaging, ¯nger print data, astronomical images and data bases containing mostly vital numerical data, tables and text information. Many lossy algorithms use lossless methods at the ¯nal stage of the encoding stage underscoring the importance of lossless methods for both lossy and lossless compression applications. In order to be able to e®ectively utilize the full potential of compression techniques for the future retrieval systems, we need e±cient information retrieval in the compressed domain. This means that techniques must be developed to search the compressed text without decompression or only with partial decompression independent of whether the search is done on the text or on some inversion table corresponding to a set of key words for the text. In this dissertation, we make the following contributions: ² Star family compression algorithms: We have proposed an approach to develop a re- versible transformation that can be applied to a source text that improves existing algorithm's ability to compress. We use a static dictionary to convert the English words into prede¯ned symbol sequences. These transformed sequences create addi- iv tional context information that is superior to the original text. Thus we achieve some compression at the preprocessing stage. We have a series of transforms which improve the performance. Star transform requires a static dictionary for a certain size. To avoid the considerable complexity of conversion, we employ the ternary tree data structure that e±ciently converts the words in the text to the words in the star dictionary in linear time. ² Exact and approximate pattern matching in Burrows-Wheeler transformed (BWT) ¯les: We proposed a method to extract the useful context information in linear time from the BWT transformed text. The auxiliary arrays obtained from BWT inverse transform brings logarithm search time. Meanwhile, approximate pattern matching can be performed based on the results of exact pattern matching to extract the possible candidate for the approximate pattern matching. Then fast verifying algorithm can be applied to those candidates which could be just small parts of the original text. We present algorithms for both k-mismatch and k-approximate pattern matching in BWT compressed text. A typical compression system based on BWT has Move-to-Front and Hu®man coding stages after the transformation. We propose a novel approach to replace the Move-to-Front stage in order to extend compressed domain search capability all the way to the entropy coding stage. A modi¯cation to the Move-to-Front makes it possible to randomly access any part of the compressed text without referring to the part before the access point. v ² Modi¯ed LZW algorithm that allows random access and partial decoding for the com- pressed text retrieval: Although many compression algorithms provide good compres- sion ratio and/or time complexity, LZW is the ¯rst one studied for the compressed pattern matching because of its simplicity and e±ciency. Modi¯cations on LZW algo- rithm provide the extra advantage for fast random access and partial decoding ability that is especially useful for text retrieval systems. Based on this algorithm, we can provide a dynamic hierarchical semantic structure for the text, so that the text search can be performed on the expected level of granularity. For example, user can choose to retrieve a single line, a paragraph, or a ¯le, etc. that contains the keywords. More importantly, we will show that parallel encoding and decoding algorithm is trivial with the modi¯ed LZW. Both encoding and decoding can be performed with multiple pro- cessors easily and encoding and decoding process are independent with respect to the number of processors. vi To my parents and my wife with love vii Acknowledgments I am grateful to all the help given during my doctoral research. First and foremost, I thank Dr. Amar Mukherjee for his guidance and supervision as my academic advisor and committee Chair. His passion to the knowledge deeply impressed me and will sure motivate my pursuing in the rest of my life. I thank my committee member Dr. Mostafa Bassiouni, Dr. Sheau-Dong Lang, and Dr. Huaxin You for their insightful reviews, comments, and other contributions. I also thank for the ¯nancial supports provided by NSF fund IIS-9977336, IIS-0207819, and IIS-0312724. The M5 group provides countless supplies to my research. It is a pleasure and encour- aging to discuss with my fellow graduate students, Tao Tao, Ravi Vijaya Satya, Robert Franceschini, Fauzia Awan, Weifeng Sun, Raja Iqabal, and Nitin Motgi. My research and publications also reflect their selfless assistance. I am glad to have the communications with my friends Biao Chen, Yubin Huang, Zhiguang Xu, Rong Wang, Ning Jiang, Jun Li, Ji Liu, Yi Wang, Feng Lu, Guoqiang Wang, Yixiao Yang, Dahai Guo, Jiaying Ni, Chun Huang, Chengya Liang, Dr. Karl Chai, and many other warmhearted schoolmates. They made my life in Orlando a happy one. Finally, I thank my parents Guoheng Zhang and Zhonghui Huang, my wife Liping Chen for their encouragement and love that never end. viii TABLE OF CONTENTS LIST OF TABLES ::::::::::::::::::::::::::::::::::: xiv LIST OF FIGURES :::::::::::::::::::::::::::::::::: xvi CHAPTER 1 MOTIVATION AND INTRODUCTION :::::::::::: 1 1.1 Motivation ..................................... 1 1.2 Some Background ................................. 7 1.2.1 Lossless Text Compression ........................ 7 1.2.2 Compressed Pattern Matching ...................... 14 1.2.3 Text Information Retrieval in Compressed Text . 17 1.3 Our Contribution ................................. 21 1.4 Contents of the Thesis .............................. 23 CHAPTER 2 REVIEW OF RELATED WORKS ::::::::::::::: 25 2.1 Classi¯cation of Lossless Compression Algorithms . 25 2.1.1 Statistical Methods ............................ 26 ix 2.1.2 Dictionary Methods ............................ 31 2.1.3 Transform Based Methods: The Burrows-Wheeler Transform (BWT) 32 2.1.4 Comparison of Performance of Compression Algorithms . 34 2.1.5 Transform Based Methods: Star (*)