<p>Telecommunications Industry Association (TIA) TR–30.3/01–07–0nn</p><p>COMMITTEE CONTRIBUTION Technical Committee TR–30.3 Baltimore, MD; 16 Jul 2001</p><p>SOURCE: Henderson Communications Laboratories </p><p>CONTRIBUTORS: Warren L. Henderson, Jr. 909.788.8849 909.788.9019 (fax) [email protected]</p><p>TITLE: Evaluation of Proposed Test Files PROJECT: PN-3509 DATE: 16 Jul 2001 DISTRIBUTION: TR-30.3 Meeting Attendees</p><p>ABSTRACT This contribution is an expansion of one submitted by the same author at the Boca Raton, Florida; 12 Jan 2000, meeting of this committee. Committee members have discussed methods for improving the relevance of files used for tests performed on a modems expected to be used on Internet connections. Proposed files and evaluations were presented at the January 2000 TR30.3 meeting. This paper contains a discussion of the files, the justification for their use and sample test results using the proposed files in both V.42bis and V.44 compression testing.</p><p>Copyright Statement The contributor grants a free, irrevocable license to the Telecommunications Industry Association (TIA) to incorporate text contained in this contribution and any modifications thereof in the creation of a TIA standards publication; to copyright in name any standards publication even though it may include portions of this contribution; and at TIA's sole discretion to permit others to reproduce in whole or in part the resulting TIA standards publication.</p><p>Intellectual Property Statement The individual preparing this contribution does not know of patents, the use of which may be essential to a standard resulting in whole or in part from this contribution.</p><p>0a9cbe162cf848f30f21b32630f11b70.doc 1 Telecommunications Industry Association (TIA) TR–30.3/01–07–0nn</p><p>1.0.0 INTRODUCTION The nature of the information transmitted through data modems has evolved to a point where performance evaluations using traditional file types has been questioned. This contribution is based on research conducted by A. Clark with the objective of identifying a set of files that would objectively evaluate lossless compression algorithms and is a review of files submitted. The files evaluated and proposed were selected within the TIA TR30.3 committee based on the considerations described herein.</p><p>2.0.0 The Concept The proposed files were presented as a contribution to ITU Study Group 16 – Question 8 and TIA working group TR30.1 for use in comparison testing of specific data compression algorithms and to TR30.3 for use in evaluating transmission products that implement compression schemes. The evaluation process followed procedures under discussion in TR30.3 as a part of PN3509 for evaluating modem throughput capacity as a function of file content (throughput vs file type–TFT). Individual file construction is based on the concept that Internet traffic is a complex construction of diverse content files. The committee selected a set of application files, including CAD, Database, Graphics, Spreadsheet and Word Processing file types. Included, as well, are files that represent languages other than English. In addition to specifically “Internet-type” files, the committee submits for consideration, one file from previous evaluation standards. File “4.tst”, a highly non-compressible file, has been part of the testing repertoire for many years and, as noted later in this contribution, serves a unique purpose in product evaluation. The selection of the proposed files was based on evaluating the Entropy and Compression Ratio values for all considered files. </p><p>3.0.0 The Files Individual file construction is based on the concept that Internet traffic is a complex construction of diverse content files. The files are created in two formats. The native form of each file is the comprehensive file content, constructed to meet all the stated goals for the file’s purpose. Native file sizes range between 1x106 bytes and 2 x106 bytes. To accommodate test equipment file size limitations, additional versions are submitted at 64 x103 bytes each. While the smaller versions closely reflect the function and performance of their native counterparts, it is recognized that test results will be somewhat skewed by algorithm learning of repeated string transmissions. 3.1.0 Text File - “TEXT.tst” Entropy = 4.98 Compression Ratio = 2.15:1 3.1.1 Text File - “TEXT64.tst”  A file containing strictly text information</p><p>3.2.0 Text File - “MAIL.tst” Entropy = 1.21 Compression Ratio = 7.59:1 3.2.1 Text File - “MAIL64.tst”  A highly compressible text file created from electronic mail.</p><p>3.3.0 Unix File - “EXE-U.tst” Entropy = 6.58 Compression Ratio = 1.44:1 3.3.1 Unix File - “EXE-U64.tst”  An executable file.</p><p>0a9cbe162cf848f30f21b32630f11b70.doc 2 Telecommunications Industry Association (TIA) TR–30.3/01–07–0nn</p><p>3.4.0 Web Page - “WEB-i.tst” Entropy = 5.32 Compression Ratio = 7.88:1 3.4.1 Web Page - “WEB-i64.tst”  A highly compressible HTM file created from a Web index page .</p><p>3.5.0 Web Page - “WEB-1.tst” Entropy = 5.55 Compression Ratio = 3.59:1 3.5.1 Web Page - “WEB-164.tst”  A moderately compressible HTM file created from a Web page of random data (random image files) merged between HTML files.</p><p>3.6.0 Web Page - “WEB-2.tst” Entropy = 4.62 Compression Ratio = 2.23:1 3.6.1 Web Page - “WEB-264.tst”  A mildly compressible HTM file created from a Web page of random data (random image files) merged between HTML files, larger than WEB-1.tst.</p><p>3.7.0 Text Printed - “Print1.tst” Entropy = 4.92 Compression Ratio = 2.75:1 3.7.1 Text Printed - “Print164.tst”  A mildly compressible file containing strictly text information, in postscript format.</p><p>3.8.0 Text Printed - “Print2.tst” Entropy = 4.37 Compression Ratio = 3.20:1 3.8.1 Text Printed - “Print264.tst”  A moderately compressible file containing strictly text information, in postscript format.</p><p>3.9.0 Language - “Lang1.tst” Entropy = 5.02 Compression Ratio = 1.90:1 3.9.1 Language - “Lang164.tst”  A slightly compressible file containing text information in Chinese.</p><p>3.10.0 Language - “Lang2.tst” Entropy = 4.49 Compression Ratio = 2.12:1 3.10.1 Language - “Lang264.tst”  A slightly compressible file containing text and HTM information in French.</p><p>3.11.0 Language - “Lang3.tst” Entropy = 4.70 Compression Ratio = 2.35:1 3.11.1 Language - “Lang364.tst”  A slightly compressible file containing text and HTM information in German.</p><p>3.12.0 Language - “Lang4.tst” Entropy = 2.95 Compression Ratio = 4.18:1 3.12.1 Language - “Lang464.tst”  A slightly compressible file containing text and HTM information in Hebrew.</p><p>3.13.0 Language - “Lang5.tst” Entropy = 4.50 Compression Ratio = 2.30:1 3.13.1 Language - “Lang564.tst”  A slightly compressible file containing text and HTM information in Nordic.</p><p>3.14.0 Language - “Lang6.tst” Entropy = 6.23 Compression Ratio = 2.37:1 3.14.1 Language - “Lang664.tst”  A slightly compressible file containing text and HTM information in Japanese.</p><p>0a9cbe162cf848f30f21b32630f11b70.doc 3 Telecommunications Industry Association (TIA) TR–30.3/01–07–0nn</p><p>3.15.0 Language - “Lang7.tst” Entropy = 6.30 Compression Ratio = 2.22:1 3.15.1 Language - “Lang764.tst”  A slightly compressible file containing text and HTM information in Chinese.</p><p>3.16.0 64–bit random - “4.tst” Entropy = 8.00 Compression Ratio = 0.998:1  The 4.TST file is taken from a 64–bit random–number generator, and its characteristics are typical for files compressed with Ziv–Lempel algorithms. This file is included as a tool for evaluating algorithms when uncompressible data is transmitted as well as for use in evaluating modem performance over various network models.</p><p>4.0.0 Test Station 4.1.0 Equipment & Software: Digital Network simulator from SPIRENT/Telecom Analysis Systems. Files larger than 64 Kbytes in length were transmitted in manual mode using the “HowFast” software, in the size and format as described herein. Transmission of the same files in automatic mode was prohibited by test equipment limitations. HCL created alternate test files by using the first 64 Kbytes from each of J. Heath’s files to accommodate the equipment limitations. Smaller files were tested using the Spirent Gemini-PC software. For these tests, each 64 Kbyte file was transmitted 16 times so that the total amount of transmitted data reached 1 Mbyte of each file. In accordance with current interim practice for throughput versus file type testing, the simulator was configured to network model i09d and local loop 3.</p><p>4.2.0 Test Systems: The test systems consisted of a single remote access server (RAS) from two manufacturers and one client modem capable of both V.42bis and V.44 compression. </p><p>4.3.0 Test Procedure: The client-modem was programmed to connect at a maximum speed of 48,000 bps and to disconnect if it could not connect at this speed. Since this is not a product evaluation, failures to connect were not considered in the average. Every average included 3 completed (successful) file transfer measurements if such a test was completed by the modem at all. Individual calls were placed for each file transmission measurement and torn down before beginning the next test. Testing included down-link measurements only. In the interest of performing a reasonable number of tests and still obtain enough data to reach a reasonable conclusion, except for the benchmark tests, up-link and bi-directional transmission directions were not carried out. 4.3.1 Benchmark, TSB-38 files (1.tst, 2.tst, 3.tst, 4.tst and 5.tst) – V.42bis mode As a benchmark, V.42bis tests were run with the original TSB-38 files (1.tst, 2.tst, 3.tst, 4.tst and 5.tst) automatically on 1 modems and 1 server. Tests of each file were performed 3 times with the average cps reported here. V.42bis tests were again run with the original TSB-38 files manually on 1 modems and 1 server. As before, tests of each file were repeated 3 times and the average cps throughput was compared with the software driven tests. 4.3.2 Observation The resultant difference between the two procedures was never less than 3.1% and never greater than 4.7%.</p><p>0a9cbe162cf848f30f21b32630f11b70.doc 4 Telecommunications Industry Association (TIA) TR–30.3/01–07–0nn</p><p>5.0.0 COMMENT and CONCLUSION In all, this report represents results of tests of several file types. The purpose of this round of tests was to evaluate file types for use in standardized testing. The only reportable conclusions that can be drawn here are: 5.1.0 Conclusion 1: File compatibility with tested systems With the exception of “TEXT.tst”, the test files, as proposed by A. Clark, do not appear to adversely change the relative performance of the tested systems when compared to existing test files. Therefore, since they are more representative of actual user files, further work towards standardization is justified. 5.2.0 Conclusion 2: File compatibility with existing equipment The test files, as proposed by A. Clark, are incompatible with existing test equipment and software in automated mode. Therefore, the basic test concept of using larger files that are representative of actual user data needs to be evaluated and adjusted, along with the test equipment (if necessary).</p><p>0a9cbe162cf848f30f21b32630f11b70.doc 5 Telecommunications Industry Association (TIA) TR–30.3/01–07–0nn</p><p>File Entropy V42bis V.44 1x30.tst 4.59 11,331 11,118 2x10.tst 4.38 11,503 13,640 3x06.tst 6.02 7,116 8,979 4x04.tst 6.67 5,559 7,147 5x16.tst 11,380 12,992 calgary/book1 4.53 11,320 11,697 calgary/paper1 4.98 11,435 11,340 calgary/pic 1.21 ? ? calgary/progc 5.20 11,561 13,464 canterbury/asyoulik.txt 4.81 9,344 9,344 canterbury/bible.txt 4.34 11,404 11,404 canterbury/cp.html 5.23 11,543 11,543 canterbury/E.coli 2.00 11,415 11,415 EXE-U64-S-TST 6.58 9,149 8,785 EXE-U-S-TST 7,523 Fax 64 S-TST 7.40 11,076 13,941 Fax S-TST 7.40 11,251 13,932 LANG1.tst 8,852 8,254 LANG2.tst 10,007 8,833 LANG3.tst 11,226 9,935 Print 1 S-TST 4.92 11,453 11,745 Print 164 S-TST 4.92 7,553 8,033 Print 2 S-TST 4.37 10,713 10,444 Print 264 S-TST 4.92 11,305 11,589 Text 200 S-TST 4.59 11,411 11,624 Text 64 S-TST 4.57 11,387 11,183 Web 1 S-TST 5.16 11,428 12,838 Web 1130 S-TST 5.16 10,838 12,228 Web 164 S-TST 5.16 10,476 11,810 Web 2 S-TST 5.32 11,229 8,695 Web 264 S-TST 5.32 10,944 12,204 Web i 64 S-TST 11,370 12,900 Web i S-TST 11,131 12,501</p><p>0a9cbe162cf848f30f21b32630f11b70.doc 6</p>