USOO7016547B1 (12) United States Patent (10) Patent No.: US 7,016,547 B1 Smirnov (45) Date of Patent: Mar. 21, 2006 (54) ADAPTIVE ENTROPY 5,504,591 A 4/1996 Dujari ENCODING/DECODING FORSCREEN 5,544.286 A 8/1996 Laney CAPTURE CONTENT 5,559,557 A 9/1996 Kato et al. 5,579,430 A 11/1996 Grill et al. (75)75 Inventor: Sergey Smirnov, Redmond, WA (US) 5,661,7555,654,706 A 8/1997 JeongVan de et Kerkhof al. (73) Assignee: Microsoft Corporation, Redmond, WA 5,717.821 A 2/1998 Tsutsui (US) (Continued) (*) Notice: Subject to any disclaimer, the term of this FOREIGN PATENT DOCUMENTS patent is extended or adjusted under 35 EP O5403SO 5/1993 U.S.C. 154(b) by 768 days. (Continued) (21)21) Appl. No.: 10/186,10/186,639 OTHER PUBLICATIONS (22) Filed: Jun. 28, 2002 Brandenburg, “ASPEC CODING”, AES 10" International Conference, pp. 81-90 (1991). (51) Int. Cl. G06K 9/36 (2006.01) (Continued) (52) U.S. Cl. ...................... 382/245; 382/166; 3. Primary Examiner-Anh Hong Do (74) Attorney, Agent, or Firm-Klarquist Sparkman, LLP (58) Field of Classification Search ................ 382/162, 382/166, 209, 218, 219, 173,244, 232,233, (57) ABSTRACT 382/234, 245; 341/87, 106,55, 67; 34.5/545, 34.5/538; 375/240.1, 241,253; 719/328; Adaptive entropy encoding and decoding techniques are 35 8/426.13 described. For example, a Screen capture encoder and See application file for complete Search history. decoder perform adaptive entropy encoding and decoding of (56) References Cited palettized Screen capture content in Screen capture video. The encoder Selects between different entropy encoding U.S. PATENT DOCUMENTS modes (such as an arithmetic coding mode and a combined run length/Huffman encoding mode, which allow the 4,698,672 A 10/1987 Chen 4,813,056 A 3/1989 Fedele encoder to emphasize bitrate reduction at Some times and 4,901,075 A 2/1990 Vogel encoding speed at other times). The run length encoding can 5,043,919 A 8/1991 Callaway et al. use run value Symbols adapted to common patterns of 5,089.818 A 2/1992 Mahieux et al. redundancy in the content. When the encoder detects Series 5,128,758 A 7/1992 Azadegan of pixels that could be encoded with different, alternative 5,179,442 A 1/1993 Azadegan runs, the encoder Selects between the alternative runs based 5,227,788 A 7/1993 Johnston upon efficiency criteria. The encoder also performs adaptive 5,266,941 A 11/1993 Akeley et al. Huffman encoding, efficiently parameterizing Huffman code 5,394,170 A 2/1995 Akeley et al. tables to reduce overall bitrate while largely preserving the 5,400,075 A 3/1995 Savatier compression gains of the adaptive Huffman encoding. A 5,457.495 A 10/1995 Hartung 5,461,421. A 10/1995 Moon decoder performs converse operations. 5,467,134 A 11/1995 Laney 5,481,553 A 1/1996 Suzuki 15 Claims, 19 Drawing Sheets 610 630 Run length encode runs for frame. Huffman encode results of run length encoding. 640 US 7,016,547 B1 Page 2 U.S. PATENT DOCUMENTS Proc. Data Compression Conference '92, IEEE Computer Society Press, pp. 52-62 (1992). 5,825,830 A 10/1998 Kopf Gill et al., “Creating High-Quality Content with Microsoft 5.835,144 A 11/1998 Matsumura Windows Media Encoder 7,” 4 pp. (2000). Downloaded 5,883,633. A 3/1999 Gill et al. from the World Wide Web on May 1, 2002). 5,982,437 A 11/1999 Okazaki Li et al., “Optimal Linear Interpolation Coding for Server 5.990,960 A 11/1999 Murakami Based Computing.” Proc. IEEE Int'l Conf. On Communica 5,995,670 A 11/1999 Zabinsky tions, 5 pp. (Apr.-May 2002). 6,002,439 A 12/1999 Murakami Matthias, “An Overview of Microsoft Windows Media 6,054,943 A * 4/2000 Lawrence .................... 341/87 Screen Technology,” 3 pp. (2000). Downloaded from the 6,097,759 A 8/2000 Murakami World Wide Web on May 1, 2002). 6,100,825 A 8/2000 Sedluk Palmer et al., “Shared Desktop: A Collaborative Tool for 6,111,914 A 8/2000 Bist Sharing 3-D Applications Among Different Window 6,148,109 A 11/2000 Boon Systems,” Digital Tech. Journal, v. 9, No. 3, pp. 42-49 6.215,910 B1 4/2001 Chaddha (1997). 6,223,162 B1 4/2001 Chen OPTX International, “OPTX Improves Technology-Based 6,226,407 B1 * 5/2001 Zabih et al. ................ 382/209 6,233,017 B1 5/2001 Chaddha Training with Screen WatchTM 3.0. Versatile Screen Capture 6.253,165 B1 6/2001 Malvar Software Adds High Color and Live Webcast Support,” 1 p., 6,292.588 B1 9/2001 Shen document marked Feb. 15, 2001 downloaded from the 6,300,888 B1 10/2001 Chen World Wide Web on Sep. 22, 2005). 6,304,928 B1 10/2001 Mairs et al. OPTX International, “OPTX International Marks One Year 6,337.881 B1 1/2002 Chaddha Anniversary of Screen Watch With Release of New 2.0 6,341,165 B1 1/2002 Gbur Version,” 1 p., document marked May 16, 2000 6,345,123 B1 2/2002 Boon downloaded from the World Wide Web on Sep. 22, 2005). 6,377,930 B1 4/2002 Chen OPTX International, “New Screen WatchTM 4.0 Click and 6,404.931 B1 6/2002 Chen StreamTM Wizard From OPTX International Makes 6,421,738 B1 7/2002 Ratan et al. ................ 719/328 Workplace Communication Effortless,” 1 p., document 6,477,280 B1 11/2002 Malvar marked Sep. 24, 2001, downloaded from the World Wide 6,542,631 B1 4/2003 Ishikawa .................... 382/162 6,573.915 B1* 6/2003 Sivan et al. ................ 71.5/781 Web on Sep. 22, 2005). 6,646,578 B1 11/2003 Au Schaar-Mitrea et al., “Hybrid Compression of Video with 6,721,700 B1 4/2004 Yin Graphics in DTV Communication Systems," IEEE Trans. On 6,728,317 B1 4/2004 Demos Consumer Electronics, pp. 1007-1017 (2000). 6,766.293 B1 7/2004 Herre Shamoon et al., “A Rapidly Adaptive LOSsleSS Compression 6,771,777 B1 8/2004 Gbur Algorithm for High Fidelity Audio Coding.” IEEE Data 2004/O136457 A1 7/2004 Funnell et al. Compression Conf. 1994, pp. 430-439 (Mar. 1994). TechSmith Corporation, “TechSmith Camtasia Screen FOREIGN PATENT DOCUMENTS Recorder SDK,” 2 pp. (2001). EP O910927 1/1998 TechSmith Corporation, “Camtasia Feature of the Week: EP O966793 9/1998 Quick Capture,” 2 pp. (Downloaded from the World Wide EP O931386 1/1999 Web on May 9, 2002; document dated Jan. 4, 2001). JP 5-292481 11/1993 TechSmith Corporation, “Camtasia Screen Recorder SDK JP T-274,171 10/1995 DLL API User Guide,” version 1.0, 66 pp. (2001). OTHER PUBLICATIONS TechSmith Corporation, “Camtasia v3.0.1-README. De Agostino et al., “Parallel Algorithms for Optimal Com TXT,”, 19 pp. (Jan. 2002). pression using Dictionaries with the Prefix Property,’” in * cited by examiner U.S. Patent Mar. 21, 2006 Sheet 1 of 19 US 7,016,547 B1 1.IVJOJCI 00[ OII ZZI U.S. Patent Mar. 21, 2006 Sheet 2 of 19 US 7,016,547 B1 Computing Environment 200 Communication Connection(s) 270 Input Device(s) 250 Processing Display Card 230 Unit 210 Memory 220 Output Device(s) 260 a in a - - - - a Storage 240 Software 280 Implementing Adaptive Screen Capture Entropy Encoder and/or Decoder U.S. Patent Mar. 21, 2006 Sheet 4 of 19 US 7,016,547 B1 U.S. Patent Mar. 21, 2006 Sheet S of 19 US 7,016,547 B1 00/ L9.InãIJ 009 0IS U.S. Patent Mar. 21, 2006 Sheet 6 of 19 US 7,016,547 B1 Figure 6 600 Start next frame. Run length encode runs Arithmetic encode data for frame. for frame. Huffman encode results of run length encoding. Figure 8 800 Start next frame. Huffman decode data for Arithmetic decode data runs for frame. for frame. Run length decode runs for frame. U.S. Patent Mar. 21, 2006 Sheet 7 of 19 US 7,016,547 B1 00ZI 0IZI OZZI 006 U.S. Patent Mar. 21, 2006 Sheet 8 of 19 US 7,016,547 B1 Figure 10 1000 Start next frame. Start Grow each extant run yes Start alternative by adding pixel. alternative run ? run(s). Each run ended ? yCS Encode pixels for Select longest run. selected run. Encode any skipped pixels. U.S. Patent US 7,016,547 B1 [0II eIIQun??+ U.S. Patent US 7,016,547 B1 [ qII9InãIA LEEEEEEEEEEEEE|× U.S. Patent Mar. 21, 2006 Sheet 11 0f 19 US 7,016,547 B1 ?I?InãIA U.S. Patent Mar. 21, 2006 Sheet 12 of 19 US 7,016,547 B1 Figure 15 1500 // Deterministically assign codes based on an incomplete code length table void GenerateVLCArray(CodeLencodeLen, intnReservedSyms, int vlcArray, intnSyms) { int iSym, len, nextAvail, nCtherSyms, *vlcArray++ = nSyms; for (iSym = 0; iSym < nSyms; iSym++) vlcArrayiSym * 2 + 1 = 0; // assign codes according to the code length table nextAvail = 0; len = 1; for (iSym = 0; iSym < n ReservedSyms; iSym++) { if (codeLeniSym).len = len) { nextAvail <= (codeLeniSym-len - len); len – codeLeniSym.len; vlcArraycodeLeniSym).sym *2 = nextAvail++, vlcArraycodeLeniSym).sym * 2 + 1 = len; Assert(nextAvail < (1 < len)); // assign (almost) equal length codes for symbols not in the table nOtherSyms = nSyms - nRescrvedSyms; int nExcessCodes; do lent---, nextAvail <= 1; nExcessCodes = (1 << len) - nextAvail - notherSyms; } while (nExcessCodes < 0); Assert(nExcessCodes < nOtherSyms); for (iSym = 0; iSym < nSyms, iSym-H) if (vlcArrayiSym * 2 + 1) = 0) { if (nExcessCodes) { // use shorter codes while they last vlcArrayiSym 2 = nextAvail D 1.