(19) United States (12) Patent Application Publication (10) Pub

US 20090310671A1 (19) United States (12) Patent Application Publication (10) Pub. No.: US 2009/0310671 A1 Reynolds et al. (43) Pub. Date: Dec. 17, 2009

(54) SYSTEM AND METHOD FOR COMPRESSING Related U.S. Application Data PORTIONS OF A MEDIA SIGNAL USING DIFFERENT CODECS (63) Continuation of application No. 10/692,106, ?led on Oct. 23, 2003, noW Pat. No. 7,599,434, Which is a continuation-in-part of application No. 10/256,866, (75) Inventors: Jodie L. Reynolds, Folsom, CA ?led on Sep. 26, 2002, noW Pat. No. 7,295,608. (US); Robert W. Ingraham, (60) Provisional application No. 60/325,483, ?led on Sep. Folsom, CA (US); Lawrence S. 26, 2001. Reznick, Sacramento, CA (US); John E. Wilkinson, Carmichael, Publication Classi?cation CA (US); Jack A. Prock, Folsom, (51) Int. Cl. CA (US); Danny L. Mabey, H04N 11/02 (2006.01) Farmington, UT (US) (52) U.S. Cl...... 375/240.01 (57) ABSTRACT Correspondence Address: STOEL RIVES LLP - SLC An input module obtains a media signal to be communicated 201 SOUTH MAIN STREET, SUITE 1100, ONE to a destination system, after Which an identi?cation module identi?es a plurality of scenes Within the media signal. A UTAH CENTER selection module automatically selects different codecs from SALT LAKE CITY, UT 84111 (US) a codec library to respectively compress at least tWo of the scenes. The codecs are automatically selected to produce a (73) Assignee: Interact Devices, Inc., Folsom, CA highest compression quality for the respective scenes accord (US) ing to a set of criteria Without exceeding a target data rate. A compression module then compresses the scenes using the automatically selected codecs, after Which an output module (21) Appl' NO‘: 12/544’102 delivers the compressed scenes to the destination system With an indication of Which codec Was used to compress each (22) Filed: Aug. 19, 2009 scene.

Compressed Mledla Signal 210 Source System

Network 114

204 5 29's 29's v ‘210 110a‘|| Scene 1 110b\Scene J, 2 110c\||Scene i 3“110c\ Scene 4 | | Codec1 | | Codec2 | | Codec1 | Codec3 |

Decompressed Media Signal 116 Destination System Patent Application Publication Dec. 17, 2009 Sheet 1 0f 11 US 2009/0310671 A1

@ 106

Original Media Signal 108 l Codec $110 $102 I Compressed Media Signal 112

114

l Compressed Media Signal 112

Codec $110 r104

| Decompressed Media Signal 116

\.—,J |||||||||||||||||||||| 118

FIG. 1 (Prior Art) Patent Application Publication Dec. 17, 2009 Sheet 2 0f 11 US 2009/0310671 A1

A 202 I S Original Media Signal 108 + I Scene1i 266-"\I) Scene2i I Scene3I Scene4i I

1103 Codec1 110b C0dec2 110 Codec1 110d C0dec3

Compressed Mledia Signal 210 Source System

Network 114

204 S 2(I8 2(I8 v ‘210 110a\II Scene1 110b\Scene2 110c\IIScene3II 110d\ Scene4 I I Codec1 I I Codec2 I I Codec1 I I Codec3 I

Decompressed Media Signal116 _ _ Destination System

FIG. 2 Patent Application Publication Dec. 17, 2009 Sheet 3 of 11 US 2009/0310671 A1

‘-‘ ’\——108 V Source System 202 Input Module $302 308 304 $108 206 3506 i“ Identi?cation i’ Selection Module Module 312 210 iv i“ Output ‘L Compression Module Module \ ~20s/210 310

114

Destination _,_204 System

FIG. 3 Patent Application Publication Dec. 17, 2009 Sheet 4 of 11 US 2009/0310671 A1 |\108 4' Source System 202 Input Module $302 308 304 #108 206 3206 110 i“ Identification i’ Selection Module Module 312 210 lv i“ Output 4L Compression Module Module \ woe/210 310

v Storage Device $0 \402

404

Destination $204 System

118

FIG. 4 Patent Application Publication Dec. 17, 2009 Sheet 5 of 11 US 2009/0310671 A1

301 Identification 596 Module Target Data Rate

v 206 Selection Module @ Scene I 504 . . 502 Ls Artl?cial < S I t d + Intelligence " A > eece Characteristics —> System 516 Codec Baseline \508 Critfria Snapshot v v 510 _|—> Comparison . Compressed ' Module Compresslon > Test Scene _|—> \ 514 MOdUIGA I =P

308

FIG. 5 Patent Application Publication Dec. 17, 2009 Sheet 6 0f 11 US 2009/0310671 A1

F108 Identi?cation 394 Module AI System w v 1206a 502b 602 110a Scene 5 5 S + 5052a Characteristics Codec 1 Characteristics ~ 5052C 1150b Characteristics 4—> Codec 2 502d 110c 5 5 Characteristics <—> Codec 3 502a 110a : 5 : : S 586 Characteristics Codec 1 Target Data A Rate

508 S

‘ Baseline _ ' Snapshot 510 12 v 5 110a 5s a ‘ Compression Compressed _ 14 Module Codec 1 + Test Scene 1 53 110a “ Comparison 1 gob 5152b —> Module —> C0dec1 Codec 2 _> Compressed I? I Test Scene 2 Cntena 516 Library 5120 1150c Compressed5 308 Codec 3 '’ Test Scene 3

FIG. 6 Patent Application Publication Dec. 17, 2009 Sheet 7 0f 11 US 2009/0310671 A1

Arti?cial Intelligence System 700 / Output Layer 708

Selected Characteristics 4 Codec 502 1 10 mud/0.7%.\\2//..m 762

FIG. 7 Patent Application Publication Dec. 17, 2009 Sheet 8 0f 11 US 2009/0310671 A1

tins Identi?cation 394 Module AI System w v f206a 502b 802 804a Scene 3 5 5 + 5082a Characteristics Settings 1 Characteristics 7 5052C 8054b Characteristics <—> Settings 2 502d 804C 5 S Characteristics <—> Settings 3 502a 804a 5 I S I Target596 Data A g ’--> Characteristics Settings11‘

Rate

508 5 Baseline _ ' Snapshot 510 12 v 5 824a 5S a Compression . Compressed _ Module Settmgs 1 + Test Scene 1 534 824a 1 Comparison . Codec 8054b 51% —> Module —> Se tt ings 1 ; Sewn 2 * Compressed _ i 110 gs Test Scene 2 Criteria' ' 1516 512c 8040 5 .5 Compressed _ Settings 3 ‘’ Test Scene 3

FIG. 8 Patent Application Publication Dec. 17, 2009 Sheet 9 0f 11 US 2009/0310671 A1

308 514 ‘I 8 Comparison Module

I Rank '- Codec1C0 PSNR Licensing Cost .

1 QXTZ 35 0.02334

2 TDLM 36 0.06400

3 ZH35 38 0.14538

4 RPEG 15 0

5 ZFRM 5 0

FIG. 9 Patent Application Publication Dec. 17, 2009 Sheet 10 of 11 US 2009/0310671 A1

506 106 S 512 kbps

V Source System 202 Input Module $302 308 306 $108 206 V S Identification Selection Module i, Module 210 l Output 4L Compression Module Module “208/21 0

128 kbps

FIG. 10 Patent Application Publication Dec. 17, 2009 Sheet 11 0f 11 US 2009/0310671 A1

206 110a I J1 102 110b Codec 1 1104a 11 b Codec 2 ? 502a 502b T 5 5 Characteristics <- —+» Characteristics Q 7m

50205 j 502d5 Characteristics <- -i» Characteristics 110c5 i i 110d5 Codec 3 11040 1104d _— Codec 4

FIG. 11 US 2009/0310671 A1 Dec. 17, 2009

SYSTEM AND METHOD FOR COMPRESSING Selecting the Wrong codec at the outset can be a costly mis PORTIONS OF A MEDIA SIGNAL USING take in terms of the bandWidth required to transmit or store the DIFFERENT CODECS media signal.

CROSS-REFERENCE TO RELATED BRIEF DESCRIPTION OF THE DRAWINGS APPLICATIONS [0009] FIG. 1 is a block diagram of a conventional commu [0001] This application is a continuation of US. patent nication system using data compression; application Ser. No. 10/692,106, ?led Oct. 23, 2003, Which is [0010] FIG. 2 is a block diagram of a communication sys a continuation-in-part of US. patent application Ser. No. tem using multiple codecs for compressing portions of a 10/256,866, ?led Sep. 26, 2002, Which claims the bene?t of media signal according to an embodiment of the invention; Provisional Application No. 60/325,483, ?led Sep. 26, 2001. [0011] FIG. 3 is a detailedblock diagram of a source system All of the foregoing applications are incorporated herein by according to a ?rst embodiment of the invention; reference. [0012] FIG. 4 is a detailedblock diagram of a source system according to a second embodiment of the invention; TECHNICAL FIELD [0013] FIG. 5 is a detailed block diagram of a selection [0002] The present invention relates generally to the ?eld of module; data compression. More speci?cally, the present invention [0014] FIG. 6 is a data How diagram ofa process for auto relates to techniques for optimizing the compression of video matically selecting a codec; [0015] FIG. 7 is a detailed block diagram of an arti?cial and audio signals. intelligence system; BACKGROUND OF THE INVENTION [0016] FIG. 8 is a data How diagram ofa process for auto matically selecting settings for a codec; [0003] In the communication age, bandWidth is money. [0017] FIG. 9 is a block diagram ofa comparison module Video and audio signals (hereinafter “media signals”) con shoWing the introduction of a licensing cost factor; sume enormous amounts of bandWidth depending on the [0018] FIG. 10 is a block diagram ofa process for modify desired transmission quality. As a result, data compression is ing a target data rate; and playing an increasingly important role in communication. [0019] FIG. 11 shoWs a video frame subdivided into a plu [0004] Conventionally, the parties to a communication rality of sub-frames. decide on a particular codec (compressor/decompressor) for compressing and decompressing media signals. A Wide vari DETAILED DESCRIPTION ety of codecs are available. General classi?cations of codecs include discrete cosine transfer (DCT) or “block” codecs, [0020] Reference is noW made to the ?gures in Which like fractal codecs, and Wavelet codecs. reference numerals refer to like elements. For clarity, the ?rst [0005] Some codecs are “lossless,” meaning that no data is digit of a reference numeral indicates the ?gure number in lost during the compression process. A compressed media Which the corresponding element is ?rst used. signal, after being received and decompressed by a lossless [0021] In the folloWing description, numerous speci?c codec, is identical to the original. HoWever, most commer details of programming, softWare modules, user selections, cially-available codecs are “lossy” and result in some degra netWork transactions, database queries, database structures, dation of the original media signal. etc., are provided for a thorough understanding of the [0006] For lossy codecs, compression “quality” (i.e., hoW embodiments of the invention. HoWever, those skilled in the similar a compressed media signal is to the original after art Will recogniZe that the invention can be practiced Without decompression) varies substantially from codec to codec, and one or more of the speci?c details, or With other methods, may depend, for instance, on the amount of available band components, materials, etc. Width, the quality of the communication line, characteristics [0022] In some cases, Well-knoWn structures, materials, or of the media signal, etc. Another compression metric, i.e., operations are not shoWn or described in detail in order to performance, relates to the amount of bandWidth required to avoid obscuring aspects of the invention. Furthermore, the transmit the compressed signal as opposed to the original described features, structures, or characteristics may be com signal. Typically, lossy codecs result in better performance bined in any suitable manner in one or more embodiments. than lossless codecs, Which is Why they are preferred in most [0023] FIG. 1 is a block diagram of a conventional system applications. 100 for communicating media signals from a source system [0007] Codec designers generally attempt to fashion 102 to a destination system 104. The source and destination codecs that produce high quality compressed output across a systems 102, 104 may be variously embodied, for example, as Wide range of operating parameters. Although some codecs, personal computers (PCs), cable or satellite set-top boxes such as MPEG-2, have gained Widespread acceptance (STBs), or video-enabled portable devices, such as personal because of their general usefulness, no codec is ideally suited digital assistants (PDAs) or cellular telephones. to all purposes. Each codec has individual strengths and [0024] Within the source system 102, a video camera 106 or Weaknesses. other device captures an original media signal 108. A codec [0008] Conventionally, the same codec is used to compress (compressor/decompressor) 110 processes the original media and decompress a media signal during the entire communi signal 108 to create a compressed media signal 112, Which cation session or uniformly across a storage medium (e.g., may be delivered to the destination system 104 via a netWork DVD). HoWever, a media signal is not a static quantity. A 114, such as a local area netWork (LAN) or the Internet. video signal, for example, may change substantially from Alternatively, the compressed media signal 112 could be scene to scene. Likewise, the available bandWidth or line Written to a storage medium, such as a CD, DVD, ?ash quality may change during the course of a communication. memory device, or the like. US 2009/0310671 A1 Dec. 17, 2009

[0025] At the destination system 104, the same codec 110 For example, one codec 110a may be a DCT codec, While processes the compressed media signal 112 received through another codec 110!) is a fractal codec, and yet another codec the netWork 114 to generate a decompressed media signal 1100 is a Wavelet codec. 116. The destination system 104 then presents the decom [0031] Unlike conventional systems 100, the system 200 of pressed media signal 116 on a display device 118, such as a FIG. 2 automatically selects, from the available codecs 110, a particular codec 110 best suited to compressing each scene television or computer monitor. 206. Details of the selection process are described in greater [0026] Conventionally, the source system 102 uses a single detail beloW. Brie?y, hoWever, the system 200 “remembers” codec 110 to process the entire media signal 108 during a Which codecs 110 are used for scenes 206 having particular communication session or for a particular storage medium. characteristics. If a subsequent scene 206 is determined to HoWever, as noted above, a media signal is not a static quan have the same characteristics, the same codec 110 is used. tity. Video signals may change substantially from scene to HoWever, if a scene 206 is found to have substantially differ scene. A single codec, Which may function Well under certain ent characteristics from those previously observed, the sys tem 200 tests various codecs 110 on the scene 206 and selects conditions, may not fare so Well under different conditions. the codec 110 producing the highest compression quality Changes in available bandWidth, line conditions, or charac (i.e., hoW similar the compressed media signal 210 is to the teristics of the media signal, itself, may drastically change the original signal 108 after decompression) for a particular tar compression quality to the point that a different codec may do get data rate. much better. In certain cases, a content developer may be able [0032] In addition, the source system 202 reports to the to manually specify a change of codec 110 Within a media destination system 204 Which codec 110 Was used to com signal 108 Where, for instance, the content developer knoWs press each scene 206. As illustrated, this may be accom that one codec 110 may be superior to another codec 110. plished by associating codec identi?ers 208 With each scene HoWever, this requires signi?cant human effort and cannot be 206 in the resulting compressed media signal 210. The codec performed in real time. identi?ers 208 may precede each scene 206, as shoWn, or could be sent as a block at some point during the transmission. [0027] FIG. 2 is a block diagram of an alternative system The precise format of the codec identi?ers 208 is not crucial 200 for communicating media signals from a source system to the invention and may be implemented using standard data 202 to a destination system 204 according to an embodiment structures knoWn to those of skill in the art. of the present invention. As before, the source system 202 [0033] The destination system 204 uses the codec identi? receives an original media signal 108 captured by a video ers 208 to select the appropriate codecs 110 for decompress camera 106 or other suitable device. ing the respective scenes 206. The resulting decompressed media signal 116 may then be presented on the display device [0028] HoWever, unlike the system 100 of FIG. 1, the 118, as previously described. depicted system 200 is not limited to using a single codec 110 [0034] FIG. 3 illustrates additional details of the source during a communication session or for a particular storage system 202. In one embodiment, an input module 302 medium. Rather, as described in greater detail beloW, each receives the original media signal 108 from the video camera scene 206 or segment of the original media signal 108 may be 106 or other source device. An identi?cation module 304 compressed using one of a plurality of codecs 110. A scene divides the original media signal 108 into scenes 206 and 206 may include one or more frames of the original media identi?es various characteristics (not shoWn) of each scene signal 108. In the case of video signals, a frame refers to a 206, as described in greater detail beloW. single image in a sequence of images. More generally, hoW [0035] Thereafter, for each scene 206, a selection module ever, a frame refers to a packet of information used for com 306 uses the characteristics (or the scene 206 itself) to select munication. the optimal codec 110 from a codec library 308. As used herein, “optimal” means producing the highest compression [0029] As used herein, a scene 206 may correspond to a quality for the compressed media signal 210 at a particular ?xed segment of the media signal 108, e.g., tWo seconds of target data rate (among those codecs 110 Within the codec audio/video or a ?xed number of frames. In other embodi library 308). ments, hoWever, a scene 206 may be de?ned by characteris [0036] In one embodiment, a user may specify a particular tics of the original media signal 108, i.e., a scene 206 may target data rate, i.e., 128 kilobits per second (kbps). Altema include tWo or more frames sharing similar characteristics. tively, the target data rate may be determined by the available When one or more characteristics of the original media signal bandWidth or in light of other constraints. 108 changes beyond a preset threshold, the source system 202 [0037] The codec library 308 may include a Wide variety of may detect the beginning of a neW scene 206. Thus, While the codecs 110. Examples of possible video codecs 110 are pro video camera 106 focuses on a static object, a scene 206 may vided in the folloWing table. In addition, various audio-only last until the camera 106, the object, or both are moved. codecs may be provided, such as MPEG Audio Layer 3 [0030] As illustrated, tWo adjacent scenes 206 Within the (MP3), MPEG-4 Structured Audio (MP4-SA), CCITT same media signal 108 may be compressed using different u-LaW, Ogg Vorbis, and AC3. Of course, other presently codecs 110. The codecs 110 may be of the same general type, available or yet-to-be-developed codecs 110 may be used e.g., discrete cosine transform (DCT), or of different types. Within the scope of the invention.

TABLE 1

FOURCC Name OWner FOURCC Name OWner

3IV1 3ivx 3IVX MPG4 MPEG-4 Microsoft 3IV2 3ivx 3IVX MPGI MPEG Sigma Designs US 2009/0310671 A1 Dec. 17, 2009 3

TABLE l-continued

FOURCC Name OWner FOURCC Name OWner

AASC Autodesk Animator Autodesk MRCA Mrcodec FAST codec Multimedia ADV1 WaveCodec Loronix MRLE Microsoft RLE Microsoft ADV] Avid M-JPEG Avid Technology MSVC Microsoft Microsoft Video 1 AEMI Array VideoONE Array MSZH AVImsZh Kenji Oshima MPEG1 —I Capture Microsystems AFLI Autodesk Animator Autodesk MTX1 Matrox codec through MTX9 AFLC Autodesk Animator Autodesk MV12 codec AMPG Array VideoONE Array MWV1 AWare Motion AWare Inc. MPEG Microsystems Wavelets ANIM RDX Intel nAVI AP41 AngelPotion AngelPotion NTN1 Video Nogatech De?nitive Compression 1 ASV1 Asus Video Asus NVDS NVidia NVidia Texture Format ASV2 Asus Video (2) Asus NVHS NVidia NVidia Texture Format ASVX Asus Video 2.0 Asus NHVU NVidia NVidia Texture Format AUR2 Aura 2 Codec-YUV Auravision NVSO-NVSS NVidia 422 AURA Aura 1 Codec-YUV Auravision NVTO-NVTS NVidia 41 1 AVRn Avid M-JPEG Avid Technology PDVC DVC codec I-O Data Device, Inc. BINK Bink Video RAD Game Tools PGVV Radius Video Radius Vision BT20 Prosumer Video Conexant PHMO Photomotion IBM BTCV Composite Video Conexant PIM1 Pegasus Imaging Codec BWIO BroadWay MPEG Data Translation PIM2 Pegasus Imaging Capture/Compression CC12 YUV12 Codec Intel PIMJ Lossless .IPEG Pegasus Imaging CDVC Canopus DV Codec Canopus PIXL Video XL Pinnacle Systems CFCC DPS Perception Digital Processing PVEZ PoWerEZ Horizons Systems Technology CGDI Camcorder Video Microsoft PVMM PacketVideo PacketVideo Corporation Corporation MPEG-4 CHAM Caviara Champagne Winnov PVW2 Pegasus Pegasus Imaging Wavelet Compression CMYK Uncompressed Colorgraph qpeq QPEG 1.1 Q-Team CMYK CJPG WebCam 1 PEG Creative Labs QPEG QPEG Q-Team CPLA YUV 4:2:0 Weitek raW RaW RGB CRAM Microsoft Video 1 Microsoft RGBT 32 bit support Computer Concepts CVID Cinepak ProvidenZa & RLE Run Length Microsoft Boekelheide Encoder CWLT Color WLT DIB Microsoft RLE4 4bpp Run Microsoft Length Encoder CYUV Creative YUV Creative Labs RLES 8bpp Run Microsoft Length Encoder CYUY ATI Technologies RMP4 MPEG-4 AS Sigma Designs Pro?le Codec D261 H.261 DEC RT21 Real Time Intel Video 2.1 D263 H.263 DEC rv20 RealVideo G2 Real DIV3 DivX MPEG-4 DivX rv30 RealVideo 8 Real DIV4 DivX MPEG-4 DivX RVX RDX Intel DIV5 DivX MPEG-4 DivX s422 VideoCap Tekram C210 International YUV Codec US 2009/0310671 A1 Dec. 17, 2009

TABLE l-continued

FOURCC Name Owner FOURCC Name Owner

DIVX DivX OpenDivX SAN3 DivX 3 divX DivX SDCC Digital Camera Sun Codec Communications DMBI Rainbow Runner MatroX SEDG Samsung Samsung hardware MPEG-4 compression DMB2 Rainbow Runner MatroX SFMC Surface Fitting CrystalNet hardware Method compression DSVD DV Codec SMSC Proprietary Radius codec DUCK TrueMotion S Duck Corporation SMSD Proprietary Radius codec dv25 DVCPRO MatroX smsv Wavelet Video WorldConnect (corporate site) dv50 DVCPRO5O MatroX SP54 SunPlus dvsd Pinnacle Systems SPIG Spigot Radius DVE2 DVE-2 InSo? SQZ2 VXTreme Microsoft Videoconferencing Video Codec Codec V2 DVXI DVXIOOOSP Video Lucent SVIO Video R1 Sorenson Media Decoder DVX2 DVX2000S Video Lucent STVA ST CMOS ST Decoder Imager Data Microelectronics DVX3 DVX3000S Video Lucent STVB ST CMOS ST Decoder Imager Data Microelectronics DX50 DivX MPEG-4 DivX STVC ST CMOS ST version 5 Imager Data Microelectronics (Bunched) DXTn DirectX Compressed Microsoft STVX ST CMOS ST Texture Imager Data Microelectronics DXTC DirectX Texture Microsoft STVY ST CMOS ST Compression Imager Data Microelectronics ELKO Elsa Quick Codec Elsa SVQI Sorenson Sorenson Media Video EKQO Elsa Quick Codec Elsa TLMS Motion TeraLogic Intraframe Codec ESCP Escape Eidos Technologies TLST Motion TeraLogic Intraframe Codec ETVI eTreppid Video eTreppid TM2O TrueMotion Duck Codec Technologies 2.0 Corporation ETV2 eTreppid Video eTreppid TM2X TrueMotion Duck Codec Technologies 2X Corporation ETVC eTreppid Video eTreppid TMIC Motion TeraLogic Codec Technologies Intraframe Codec FLIP Field Encoded D-Vision TMOT TrueMotion S Horizons Motion JPEG Technology FRWA Forward Motion SoftLab-Nsk TR20 TrueMotion Duck .IPEG with alpha RT 2.0 Corporation channel FRWD Forward Motion SoftLab-Nsk TSCC TechSmith Techsmith Corp. .IPEG Screen Capture Codec FVF1 Fractal Video Frame Iterated Systems TVIO Tecomac Low- Tecomac, Inc. Bit Rate Codec GLZW Motion LZW [email protected] TV] P Pinnacle/Truevision GPEG Motion JPEG [email protected] TVMJ Pinnacle/Truevision GWLT Greyscale WLT DIB Microsoft TY2C Trident Trident Decompression Microsystems H260 ITU H.26n Intel TY2N Trident through Microsystems H269 HFYU Huffman Lossless TYON Trident Codec Microsystems HMCR Rendition Motion Rendition UCOD ClearVideo eMajiX.com Compensation Format HMRR Rendition Motion Rendition ULTI Ultimotion IBM Corp. Compensation Format US 2009/0310671 A1 Dec. 17, 2009 5

TABLE l-continued

FOURCC Name Owner FOURCC Name Owner i2 63 ITU H.263 Intel V261 Lucent Lucent VX2000S IAN Indeo 4 Codec Intel V655 YUV 4:2:2 Vitec Multimedia ICLB CellB InSoft VCR1 ATI Video ATI Videoconferencing Codec 1 Technologies Codec IGOR Power DVD VCR2 ATI Video ATI Codec 2 Technologies I] PG Intergraph I PEG Intergraph VCK3-9 ATI Video ATI Codecs Technologies ILVC Layered Video Intel VDCT VideoMaker Vitec Multimedia Pro DIB ILVR ITU H.263+ Codec VDOM VDOWave VDONet IPDV Giga AVI DV Codec I-O Data Device, VDOW VDOLive VDONet Inc. IR21 Indeo 2.1 Intel VDTZ VideoTiZer Darim Vision Co. YUV Codec IRAW Intel Uncompressed Intel VGPX VideoGramPix Alaris UYUV IV3O Indeo 3 Ligos VIFP VFAPI Codec through IV39 IV32 Indeo 3.2 Ligos VIDS Vitec Multimedia IV4O Indeo Interactive Ligos VIVO Vivo H.263 Vivo Software through IV49 IV5O Indeo Interactive Ligos VIXL Video XL Pinnacle Systems .IBYR Kensington VLV1 VideoLogic .IPEG .IPEG Still Image Microsoft VP3O VP3 On2 .IPGL J PEG Light VP31 VP3 On2 L261 Lead H.26 Lead Technologies vssv VSS Video Vanguard Software Solutions L263 Lead H.263 Lead Technologies VX1K VXIOOOS Lucent Video Codec LCMW Motion CMW Codec Lead Technologies VX2K VX2000S Lucent Video Codec LEAD LEAD Video Codec Lead Technologies VXSP VXIOOOSP Lucent Video Codec LGRY Grayscale Image Lead Technologies VYU9 ATIYUV ATI Technologies Ljp g LEAD MI PEG Lead Technologies VYUY ATI YUV ATI Codec Technologies LZO1 Lempel-Ziv- Markus Oberhumer WBVC W9960 Winbond Oberhumer Codec Electronics M263 H.263 Microsoft WHAM Microsoft Microsoft Video 1 M261 H.261 Microsoft WINX Winnov Winnov Software Compression M452 MPEG-4 Microsoft WIPG Winbond (automatic WMP I PEG download) MC12 Motion ATI Technologies WNV1 Winnov Winnov Compensation Hardware Format Compression MCAM Motion ATI Technologies x263 Xirlink Compensation Format MJ2C Motion JPEG 2000 Morgan XVID XVID MPEG-4 XVID Multimedia mJPG Motion .IPEG IBM XLVO XL Video NetXL Inc. including Huffman Decoder Tables MJPG Motion JPEG XMPG XING MPEG XING Corporation MMES MPEG-2 ES Matrox XWVO-XWV9 XiWave Video XiWave Codec MP2A Eval download Media Excel XXAN Origin MP2T Eval download Media Excel Y411 YUV 4:1 :1 Microsoft MP2V Eval download Media Excel Y41P Brooktree Conexant US 2009/0310671 A1 Dec. 17, 2009 6

TABLE l-continued

FOURCC Name Owner FOURCC Name Owner

MP42 MPEG-4 Microsoft Y8 Grayscale (automatic WMP video download) MP43 MPEG-4 Microsoft YC12 YUV 12 codec Intel (automatic WMP download) MP4A Eval download Media Excel YUVS Caviar YUVS Winnov MP4S MPEG-4 Microsoft YUY2 Raw, Microsoft (automatic WMP uncompressed download) YUV 4:2:2 MP4T Eval download Media Excel YUYV Canopus MP4V Eval download Media Excel ZLIB MPEG MPEG ZPEG Video Zipper Metheus MPG4 MPEG-4 Microsoft ZyGo ZyGoVideo ZyGo Digital (automatic WMP download)

[0038] Those of skill in the art will recognize that many of motion characteristics, color characteristics, YUV signal the above-described codecs may be deemed “generalist” characteristics, color grouping characteristics, color dither codecs in that they achieve a high compression quality for a ing characteristics, color shifting characteristics, lighting wide variety of media signals and conditions. However, other characteristics, and contrast characteristics. Those of skill in codecs may be deemed “specialist” codecs because they com the art will recognize that a wide variety of other character press certain types of media signals well or compress many istics of a scene 206 may be identi?ed within the scope of the types of media signals well under certain conditions. Provid invention. ing a codec library 308 that includes a variety of both gener [0043] Motion is composed of vectors resulting from object alist and specialist codecs, including codecs of different fami detection. Relevant motion characteristics may include, for lies, typically results in the best overall compression quality example, the number of objects, the size of the objects, the for a compressed media signal 210. speed of the objects, and the direction of motion of the [0039] Referring again to FIG. 3, after a codec 110 is objects. selected for a scene 206, a compression module 310 com [0044] With respect to color, each pixel typically has a presses the scene 206 using the selected codec 110. An output range of values for red, green, blue, and intensity. Relevant module 312 receives the resulting compressed media signal color characteristics may include how the ranges of values 210 and, in one embodiment, adds codec identi?ers 208 to change through the frame set, whether some colors occur indicate which codecs 110 were used to compress each scene more frequently than other colors (selection), whether some 206. In other embodiments, the codec identi?ers 208 may be color groupings shift within the frame set, whether differ added by the compression module 3 1 0 or at other points in the ences between one grouping and another vary greatly across compression process. The output module 312 then delivers the frame set (contrast). the compressed media signal 210 to the destination system [0045] In one embodiment, an arti?cial intelligence (AI) 204 via the network 114. system 504, such as a neural network or expert system, [0040] The embodiment of FIG. 3 is primarily applicable to receives the characteristics 502 of the scene 206, as well as a streaming media applications, including video conferencing. target data rate 506 for the compressed media signal 210. The In an alternative embodiment, as depicted in FIG. 4, the AI system 504 then determines whether a codec 110 exists in output module 312 may be coupled to a storage device 402, the library 308 that has previously been found to optimally such as CD or DVD recorder, ?ash card writer, or the like. As compress a scene 206 with the given characteristics 502 at the depicted, the compressed media signal 210 (and codec iden target data rate 506. As explained below, the AI system 504 ti?ers 208) may be stored on an appropriate storage medium may be conceptualized as “storing” associations between sets 404, which is physically delivered to the destination system of characteristics 502 and optimal codecs 110. If an associa 204. In such an embodiment, the destination system 204 tion is found, the selection module 306 outputs the codec 110 would include a media reader (not shown) for reading the (or an indication thereof) as the “selected” codec 110. compressed media signal 210 from the storage medium 404. [0046] In many cases, a scene 206 having the speci?ed [0041] Unlike conventional media compression tech characteristics 502 may not have been previously encoun niques, the original media signal 108 is not compressed using tered. Accordingly, the selection module 306 makes a copy of a single codec (i.e., MPEG-2 as in DVDs). Rather, each scene the scene 206, referred to herein as a baseline snapshot 508, 206 is automatically compressed using the best codec 110 which serves as a reference point for determining compres selected from a codec library 308 for that scene 206. Using the sion quality. above-described technique, between 10 to 12 hours of DVD [0047] Thereafter, a compression module 510 tests differ quality video may be stored on a single recordable DVD. ent codecs 110 from the codec library 308 on the scene 206. [0042] FIG. 5 illustrates additional details of the selection In one embodiment, the compression module 510 is also the module 306. As noted above, the identi?cation module 304 compression module 310 of FIG. 3. As depicted, the com receives the original media signal 108 and identi?es indi pression module 510 compresses the scene 206 using differ vidual scenes 206, as well as characteristics 502 of each scene ent codecs 110 at the target data rate 506 to produce multiple 206. The characteristics 502 may include, for instance, compressed test scenes 512. US 2009/0310671 A1 Dec. 17, 2009

[0048] The codecs 110 may be tested sequentially, at ran skilled in the art Will recogniZe that these entities may be dom, or in other Ways, and all of the codecs 110 in the library represented by codes, hashes, or other identi?ers in various need not be tested. In one embodiment, input from the AI implementations. system 504 may assist With selecting a subset of the codecs [0056] Assuming that no such association 602 is found, a 110 from the library 308 for testing. In some cases, a time baseline snapshot 508 of the scene 20611 is taken. In addition, limit may be imposed for codec testing in order to facilitate the compression module 510 compresses the scene 20611 at real-time compression. Thus, When the time limit is reached, the target data rate 506 using a number of different codecs no additional compressed test scenes 512 are generated. 110a-c from the codec library 308 to create a plurality of [0049] In one embodiment, a comparison module 514 com compressed test scenes 512a-c. These test scenes 512a-c are pares the compression quality of each compressed test scene then compared against the baseline snapshot 508 according to 512 With the baseline snapshot 508 according to a set of a set of criteria 516, e.g., PSNR. criteria 516. The criteria 516 may be based on a comparison of [0057] Suppose that the compressed test scene 512a pro Peak Signal to Noise Ratios (PSNRs), Which may be calcu duced by one codec 110a (“Codec l”) results in the highest lated, for an M>

[0063] Referring to FIG. 8, the invention is not limited to ?rst three codecs 110 produce output of similar quality. HoW embodiments in Which different codecs 110 are used to ever, the codec 110 With the highest PSNR score is more than respectively encode different scenes 206 of an original media tWo times more expensive than the codec 110 With the next signal 108. As illustrated, a single codec 110 may be used in highest PSNR score, Which is, itself, almost three times more one embodiment. However, different settings 804 (param expensive than the codec 110 With the third highest PSNR eters) for the codec 110 may be automatically selected in score. In one con?guration, the comparison module 510 much the same Way that different codecs 110 Were selected in Would select the codec 110 With the third highest PSNR score the preceding embodiments. due to its much loWer licensing cost 902. [0064] As used herein, codec settings 804 refer to standard [0072] In other embodiments, the comparison module 514 parameters such as the motion estimation method, the GOP may create a composite score (not shoWn) based on the PSNR siZe (keyframe interval), types of transforms (e.g., DCT vs. score, the licensing cost 902, and other possible factors. In Wavelet), noise reduction for luminance or chrominance, still other embodiments, the comparison module 514 may decoder deblocking level, preprocessing/postprocessing ?l calculate an anticipated cost (not shoWn) for the entire trans ters (such as sharpening and denoising), etc. mission and seek to minimize that cost over all of the codec [0065] As before, suppose that the identi?cation module selection decisions. Hence, the comparison module 514 304 ?nds a scene 206a having a given set of characteristics might select a more expensive codec 110 for certain scenes 50211. In one embodiment, the AI system 504 searches an 206, Where a substantial increase in quality is realiZed, While association 802 betWeen the characteristics 502a and one or selecting less expensive codecs 110 for other scenes. more settings 80411 for the codec 110. [0073] Referring to FIG. 10, a user of the source system 202 [0066] Assume that no such association 802 is found. In may specify a particular target data rate 506, e.g., 512 kbps, one con?guration, a baseline snapshot 508 of the scene 20611 for video communication. HoWever, there is no guarantee that is taken. In addition, the compression module 510 com the destination system 204 may be able to process data that presses the scene 20611 at the target data rate 506 using the quickly. Moreover, there is no guarantee that the network 114 same codec 110 but With different settings 804a-c. The result Will alWays provide the same amount of bandWidth. As a ing compressed test scenes 512a-c are then compared against result, there may be a need to periodically change the target the baseline snapshot 508 according to a set of criteria 516, data rate 506 Within the selection module 306 of the source e.g., PSNR. system 202, since the target data rate 506 Will affect Which [0067] Suppose that the compressed test scene 512a pro codecs 110 are selected for various scenes 206. duced by one group of settings 804a (“Settings 1”) results in [0074] For example, as shoWn in FIG. 10, the destination the highest compression quality, e.g., the highest PSNR. In system 204 may be embodied as a video-enabled cellular such a case, the comparison module 514 Would inform the AI telephone. Typically, the bandWidth over cellular netWorks system 504, so that an association 802 could be made betWeen 114 is limited. Similarly, the processing poWer of a cellular the characteristics 50211 of the scene 206a and the selected telephone is substantially less than that of a personal com group of settings 804a. Accordingly, if a scene 206 having the puter or dedicated video conferencing system. same characteristics 50211 is encountered in the future, the AI [0075] Thus, although the user of the source system 202 system 504 could simply identify the optimal settings 804a speci?es a target data rate 506 of 512 kbps, the destination Without the need for retesting. system 204 and/or netWork 114 may not be up to the chal [0068] In still other embodiments, the AI system 504 may lenge. In one embodiment, in response to receiving a connec search for both different codecs 110 and different codec set tion request, the destination system 204 provides the source tings 804 based on a given set of characteristics 502. Like system 202 With a modi?ed target data rate 1002, e.g., 128 Wise, the compression module 510 may generate compressed kpbs. The modi?ed rate 1002 may be communicated to the test scenes 512 based on combinations of different codecs 110 source system 202 using any standard data structure or tech and different settings 804. The comparison module 514 may nique. Thereafter, depending on the con?guration, the target then select the best combination of codec 110 and settings 804 data rate 506 may be replaced by the modi?ed rate 1002. for a given scene 206. [0076] In certain embodiments, an actual data rate is not [0069] In one embodiment, as shoWn in FIG. 9, the com communicated. Rather, a message is sent specifying one or parison module 514 may consider other factors in addition to more constraints or capabilities of the destination system 204 (or in lieu of) compression quality in determining Which or netWork 114, in Which case it Would be up to the source codec 110 and/or settings 804 to automatically select for a system 202 to revise the target data rate 506 as appropriate. A particular scene 206. For instance, the use of certain codecs technique of altering the target data rate 506 in response to 110 may incur licensing costs 902 based on patents or other various conditions is referred to herein as “dynamic stream intellectual property rights. The licensing costs 902 may be mg.” tied to the number of times the codec 110 is used, the amount [0077] In one embodiment, dynamic streaming may be of data compressed using the codec 110, or in other Ways. employed Where no speci?c message is sent by destination [0070] While one codec 110 may provide an exceptionally system 204. The source system 202 may use latency calcula high compression quality (e. g., PSNR), its licensing cost 902 tions, requests to resend lost packets, etc., to dynamically may exceed the value of the transmission and Would not be determine the target data rate 506 for purposes of codec cost justi?ed. Indications of the licensing costs 902 for vari and/or parameter selection. ous codecs 110 may be stored Within the codec library 308 or [0078] In one con?guration, as shoWn in FIG. 11, video at other locations accessible by the comparison module 514. frames 1102 Within a scene 206 may be subdivided into a [0071] In one embodiment, the licensing costs 902 are con plurality of sub-frames 1104. While the depicted video frame sidered only When a number of the top codecs 110 produce 1102 is subdivided into four sub-frames 1104a-d of equal similar results, e.g., the compression qualities differ by no siZe, the invention is not limited in this respect. For instance, more than a threshold amount. In the example of FIG. 9, the a video frame 1102 may be subdivided into any number of