(12) Patent Application Publication (10) Pub. No.: US 2007/0016476A1 Hoffberg Et Al

US 20070016476A1 (19) United States (12) Patent Application Publication (10) Pub. No.: US 2007/0016476A1 Hoffberg et al. (43) Pub. Date: Jan. 18, 2007

(54) INTERNET APPLIANCE SYSTEMAND Continuation of application No. 1 1/363.413, filed on METHOD Feb. 27, 2006. Continuation of application No. 1 1/363.431, filed on (75) Inventors: Steven Hoffberg, West Harrison, NY Feb. 27, 2006, which is a continuation of application (US); Linda Hoffberg-Borghesani, No. 10/693,759, filed on Oct. 24, 2003, now Pat. No. Acton, MA (US) 7.006,881, which is a continuation of application No. 10/162,079, filed on Jun. 3, 2002, now Pat. No. Correspondence Address: 6,640,145, which is a continuation of application No. MILDE & HOFFBERG, LLP 09/241,135, filed on Feb. 1, 1999, now Pat. No. 1O BANK STREET 6,400,996. SUTE 460 WHITE PLAINS, NY 10606 (US) Publication Classification (73) Assignee: BLANDING HOVENWEEP, LLC, Las (51) Int. C. Vegas, NV (US) G06Q 30/00 (2006.01) (52) U.S. Cl...... 705/14 (21) Appl. No.: 11/467,915 (22) Filed: Aug. 28, 2006 (57) ABSTRACT Related U.S. Application Data A system comprising a user-analysis component that exam (63) Continuation of application No. 1 1/363.393, filed on ines at least one user action and content associated with the Feb. 27, 2006. action, and an advertisement delivery component that deter Continuation of application No. 1 1/363,411, filed on mines at least one advertisement to present to a user based Feb. 27, 2006. at least in part on the user action, and one or more of the Continuation of application No. 1 1/363,412, filed on content associated with the user action and a user profile Feb. 27, 2006. derived from user actions.

1701

Input User Identification

1702

Personalized Startup Based on User Profile

Update Present MOSt Update Preference Likely Choices Preference Matrix TO USer Matrix

Confirmation Screen Patent Application Publication Jan. 18, 2007 Sheet 1 of 30 US 2007/0016476 A1

POWER ON

CLOCKSET = Search Time Fig. 1 PROGRAM

PROGRAM 2

NeW Interface Interface Patent Application Publication Jan. 18, 2007 Sheet 2 of 30 US 2007/0016476A1

N Total Time Total - Tr C) E------! TZZZZZZZZZZZZZZZZZ ZZZZZZZZZZZZZZZZZZZZZ,------^i zZzZZZZZZZZZZZZZZ?º—--- ºZZZZZZZZZZZZZZZZZZZZZZZZZF– zzzzzZZZZZZZZZZZZZ?º ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ ZZZZZZZZZZZZZZZZZZZZZZ------|co --ZZZZZZZZZZZZZZZZZZZZ–I-II ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ"-"T-TOEON! ZZZZZZZZZZZZZZZ)----rr)I-II] ZZZZZZZZZZZZZZZZZZZZ)----

Total Time Total - Tr

T’ZZZZZZZZZZZZZZZZZZZZZ,TOE) ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZF?------"ºZºZZZZZZZZZZZZZZZZŒ ºZZZZZZZZZZZZZZZZTOE — ºZzzzZZZZZZZ[-] —ZZZZZZZZZC[I] TZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ-TOE-º! "ZZZZZZZZZZZZZZZZZZZZZZZZ!—– --ZZZZZZZZZZZZcnI? --ZZZZZZZZZZZZZZZZZZZL-I]

Ljë C tical Steps

Fig. 5 Patent Application Publication Jan. 18, 2007 Sheet 5 of 30 US 2007/0016476 A1

N Existing Interface New Interface

ZZZZZZZZZZZZZZZZZZZZZ------|– ZZZZZZZZZZZZZZdº[- ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ•------|co ZZZZZZZZZZZZZ(ºF- ZZZZZZZZZZZZZZZZZZZZZ------Iun ZZZZZZZZZZZZZZZZZZZZZZZZtº|----ko ZZZZZZZZZZZZZZ|----loo ZZZZZZZZZZZZZZ?º[-] ØZZZZZZZZZZ-- ZZZZZZZZZZZZZZZZZ)----TILL!;-) ZZZZZZZZZZZZZZZZZZZZZZZZZ!"IL]^] ZZZZZZZZZZZZZZZZ)----I?

S NØ &&&&&&&&& Clock Set ZZZZZZZZZ,?a-1 &&&&&#={ Ø"SºØ Patent Application Publication Jan. 18, 2007 Sheet 7 of 30 US 2007/0016476 A1

N Max Std

Clock Set Pl P2 ZZZZZZZZZZZZZZZZZZZZZZÑºn Critical Steps ZØØ. Fig. 8 Patent Application Publication Jan. 18, 2007 Sheet 8 of 30 US 2007/0016476A1

Existing interface New Interface JOQUITINJOsessòIdÃ©?I

ZZZZZZZZZZZZZZZZZ–|------| I-III I vn------| «>|------isT??, ---- oF||| OLA!--+ (N.TOELI L^[(…] coTIJ ZZZZZZZZZZZZZZZZZZZZZZZ?º zZZZZZZZZZZZZZZZZZÁ.’ (ºZZZZZZZZZZZZZZZZZZZZTILL ZZZZZZZZZZZZZZZZZZZZZZZZZZ ZZZZZZZZZZZZZZZZZZZZZZ ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ ZZZZZZZZZZZZ4° TºZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ ZZZZZZZZZZZZZZZZZZZZZZ!-- TºZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ. ZZZZZZZZZZZZZZZZZZZZZ)----ºn-I? TºZZZZZZZZZZZZZZZZZZZZZZZZZZZ

Theoretical Actual

Existing Interface New Interface

Theoretical 2 Actual

Existing Interface New Interface Times for the Minimum, Theoretical and Actual Keypresses Patent Application Publication Jan. 18, 2007 Sheet 10 of 30 US 2007/0016476 A1

(XXX:tax XX

(XXXX,

%XXXXXXXXX & X &OX

N& S&OXXXXX

Fig. 12a

Clock Set (106)

NeW Interface Fig. 12b Patent Application Publication Jan. 18, 2007 Sheet 11 of 30 US 2007/0016476 A1

Fig. 13

Clock Set (106)

? N P1 (168)

Fig. 14 Patent Application Publication Jan. 18, 2007 Sheet 12 of 30 US 2007/0016476 A1

Display Error Verify Instruction MeSSage AS Being Consistent Yes Append Instruction to Program Sequence

Analyze Program Sequence to Predict Next Action NO

Display Predicted Program Next Action. On Yes Complete Display

Yes Ser Accept NO Confirmation Screen Manual Input Option

Display Complete Program Sequence tO USer

User Accept NO

Yes Fig. 15 Patent Application Publication Jan. 18, 2007 Sheet 13 of 30 US 2007/0016476 A1

Instruction

Assign Identifier 1 = X

Verify

Delete Instruction in Confirmation Screen Yes NO

YeS

Fig. 16 Patent Application Publication Jan. 18, 2007 Sheet 14 of 30 US 2007/0016476 A1

1701 Input User Identification

1702

Personalized Startup Based on User Profile

Update Present MOSt Update Preference Likely Choices Preference Matrix TO USer Matrix

Manual Input Option Accept Confirmation SCreen

Fig. 17 Patent Application Publication Jan. 18, 2007 Sheet 15 of 30 US 2007/0016476 A1

1807

Feature User LeVel Extractor Control

Endpoint InStruction Instruction Extractor Input

Controlled Device Authorization

Fig. 18 Patent Application Publication Jan. 18, 2007 Sheet 16 of 30 US 2007/0016476 A1

Tape Start Time Duration Channel Representative Identifier Scenes

1902 1903 1904.905 1906 1907 1908. 1909 90 91 s

Identifying Entry End Time Date Descriptive Error Correcting Header Identifier Information Codes

Fig. 19 Patent Application Publication Jan. 18, 2007 Sheet 17 of 30 US 2007/0016476 A1

2005

a Digital

Digital Information Read/Write Detecting/ Electronics Formatting T Tape Amplifier/ Circuit ape Read/Record Demodulator/ Head Modulator

N -7 Analo () 2004 2006 Read/Writeg Electronics 2OO 2002

Tape Drive MOtOr

Fig. 20 Patent Application Publication Jan. 18, 2007 Sheet 18 of 30 US 2007/0016476 A1

2. 28 217 2.04 2116 R N E. Feature Speed Linl Extractor Path 2.05 Optimization

Ouality of 2O6 Selection

Past User 2107 History

a 208 ProgrammingCurrent Status 5s

Number of e 2109 E. OrS S. Number of s 210 Cancels Cl

211 E.Level Choice Usert rt

Current Display User Level Help Level Simplification 2O3 Patent Application Publication Jan. 18, 2007 Sheet 19 of 30 US 2007/0016476 A1

22O1 22O2 2203 2204 2205 2207 2208 Video Frame Pre- Feature Transf ) Input Buffer filtering Extractor SO Correlator

2205 22O7 2209 22 ( Transform Correlator Adaptive s Weighting-Identifier --Output NetWOrk Template Intermediate Database 2207 Processing Transform Correlator 226 2205- "I Patent Application Publication Jan. 18, 2007 Sheet 20 of 30 US 2007/0016476 A1

2303 2301 s 2306

SensOr Microprocessor Control Plant Array Data Serializer

2302a Pattern Interface & Recognition Programming System Console

2304 2305

Fig. 23 Patent Application Publication Jan. 18, 2007 Sheet 21 of 30 US 2007/0016476 A1

24Ol 2402 Input DeVice

2405

S Plant OnScreen Programming Interface

User History Prospective & 243 Database & RealTime Event Preferences Characterization Local Database 2412 24O7 Preference & Event Telecommunication Signal Correlator Module Analysis 2414 2410 2409 Videotext & Character On-Line Signal Recognition Database Receiver

241. 2408

Fig. 24 Patent Application Publication Jan. 18, 2007 Sheet 22 of 30 US 2007/0016476 A1

2503 204

SERis Storage CompressionData 25 O. Encryption/ Decryption IntermediateStorage Control S. 2509 20g, S Tuner inside-Storage Plant VCR

s 2507 2508 Pattern USer Recognition Interface 7-7 2505 2506 Fig. 25 Patent Application Publication Jan. 18, 2007 Sheet 23 of 30 US 2007/0016476 A1

26O2 2601 2603

SensOr Control Plant

Intermediate 2604 E. SenSOr Data -26ll Storage LS 2605 On-Screen Transform InterfaceDisplay Engine L? 2612

2606 Program Memory

Characterization

Template 2607-1 NetWOrk Input 2609

Pattern Storage Correlator Memory Tu-2608

Fig. 26 Patent Application Publication Jan. 18, 2007 Sheet 24 of 30 US 2007/0016476 A1

Store Templates

Generate Image DOmains

Create Mapped Ranges

Assign Identifiers

Select Matching Ranges for Each Domain

Represent Image as Identifiers

Select Template That Matches Image From Combination of Identifiers

Stop

Fig. 27 Patent Application Publication Jan. 18, 2007 Sheet 25 of 30 US 2007/0016476 A1

Input Templates

Store Templates

Input Image Data

Store Image Data

Create Domains of Stored Image Data Create Mapping Ranges of Image Data PrOCeSS a Subset Range

YeS Assign Identifiers

Select Mapping Range by Criteria Represent Image As Identified Select Matching Template

Fig. 28 Patent Application Publication Jan. 18, 2007 Sheet 26 of 30 US 2007/0016476 A1

Input Templates Store Templates Input Motion Image Data StOre Motion Image Data Create DOmains of Stored Motion Image Data Create Mapping Ranges of Data With Corresponding Data Process a Subset Range

Yes Assign Identifiers

Compute Motion Vector Select Mapping Range by Criteria Represent Motion Vectors Scene Change Detection Select Matching Template Stop Fig. 29 Patent Application Publication Jan. 18, 2007 Sheet 27 of 30 US 2007/0016476A1

3001 Input 3006 Image Transform and Subtract

?o 3007 : Around1 Window Object SEError of Interest

3008 Create First Range Mapping Region

3009

Compare Codes With Templates

3004

Subtract 3010

Match

Probability

Transform and Subtract Maximization

302 Output Recognition Fig. 30 Patent Application Publication Jan. 18, 2007 Sheet 28 of 30 US 2007/0016476 A1

Start 31 OO

Adaptive Prediction System 31 O1

Contextual Output Predicted Analysis Choices 3104 3106

User User

Feedback input interface

3105 Control 31 O7

Memory 3111

Control Signal Output 3109 311 O

Fig. 31 Patent Application Publication Jan. 18, 2007 Sheet 29 of 30 US 2007/0016476 A1

EXTERNAL MACHINE 3212 TELEPHONY 3214, ENVIRONMENTAL CONTROL 3215, PHYSICAL SECURITY 3216, ALARM 3217, BRESER AUTOMOBILE 3218, HUMAN VIDEO SEERs 3220 See USER E. 32O5 CONSUMER ELECTRONICS 3221, THERMOSTAT3222, APPLIANCE 3223, ILLUMINATION 3224, FLOW CONTROL 3225, PUMP 3226.

DATA PACKET COMMUNICATION INTERFACE 3202 LOCAL AREA MEDIA COMMUNICATION NETWORK 32.13.1 WRELESS INTERFACE 32O3 NETWORK 32.11

INTELLIGENT SERVER 32O1 CRYPTO-GRAPHIC PROCESSOR 32O7

AUDIO AND/OR VIDEO DGITAL MEMORY DATA 3209 3204 METADATA 321 O Fig. 32 Patent Application Publication Jan. 18, 2007 Sheet 30 of 30 US 2007/0016476 A1

PROVIDE PACKETDATA COMMUNICATIONS INTERFACE 3301

PROVIDE MEDIA COMMUNICATION INTERFACE 33O2

PROVIDE DIGITAL MEMORY 3303

GENERATE VIRTUAL NTERFACE 3304

COMMUNICATE AUDIO AND/OR VIDEO DATA THROUGH MEDIA COMMUNICATION INTERFACE 3305

STORE COMMUNICATED AUDIO AND/OR VIDEO DATA 3306

ENFORCE A SET OF EXTERNALLY SUPPLIED RESTRICTIONS 3307 Fig. 33 US 2007/0016476 A1 Jan. 18, 2007

INTERNET APPLIANCE SYSTEMAND METHOD herein by reference to avoid duplication of prior art teach ings. However, the disclosure herein is not meant to be RELATED APPLICATIONS limiting as to the knowledge of a person of ordinary skill in 0001. The present application is a continuation of U.S. the art. Recitation hereinbelow of these teachings or refer Ser. No. 1 1/363,393 filed on Feb. 27, 2006, U.S. Ser. No. ence to these teachings is not meant to imply that the 11/363,411 filed on Feb. 27, 2006, U.S. Ser. No. 1 1/363,412 inventors hereof were necessarily in any way involved in filed on Feb. 27, 2006, U.S. Ser. No. 1 1/363,413 filed on these references, nor that the particular improvements and Feb. 27, 2006, U.S. Ser. No. 1 1/363,431 filed on Feb. 27, claimed inventions recited herein were made or conceived 2006, each of which is Pending and claim the benefit of U.S. after the publication of these references. Thus, prior art cited Ser. No. 10/693,759 filed on Oct. 24, 2003 (U.S. Pat. No. herein is intended to (1) disclose information related to the 7,006,881). The present application claims benefit of prior application published before the filing hereof; (2) define the ity through U.S. Ser. Nos. 10/693,759 to 10/162,079 filed on problem in the art to which the present invention is directed, Jun. 03, 2002 (U.S. Pat. No. 6,640,145) and U.S. Ser. No. (3) define prior art methods of solving various problems also 09/241,135 filed on Feb. 01, 1999 (U.S. Pat. No. 6,400.996). addressed by the present invention, (4) define the state of the Each of the above applications is expressly incorporated art with respect to methods disclosed or referenced herein, and/or (5) detail technologies used to implement methods or herein by reference. apparatus in accordance with the present invention. COPYRIGHT RETENTION NOTICE 0007 Human Interface 0002 A portion of the disclosure of this patent document 0008 Aspects of the present invention provide an and appendices contain material which is Subject to copy advanced user interface. The Subject of man-machine inter right protection. The copyright owner has no objection to the faces has been studied for many years, and indeed the entire facsimile reproduction by anyone of this patent document or field of ergonomics and human factors engineering revolves the patent disclosure, as it appears in the U.S. Patent and around optimization of human-machine interfaces. Typi Trademark Office patent file or records, but otherwise cally, the optimization scheme optimizes the mechanical reserves all copyright rights whatsoever. elements of a design, or seeks to provide a universally optimized interface. Thus, a single user interface is typically FIELD OF THE INVENTION provided for a system. In fact, some systems provide a 0003) The present invention relates to the field of adap variety of interfaces, for example, novice, intermediate and tive systems, and more particularly systems and methods advanced, to provide differing balances between available which are adaptive to a human user input and/or a data control and presented complexity. Further, adaptive and/or environment, as well as applications for Such systems and responsive human-machine computer interfaces are now methods. More particularly, embodiments of the invention well known. However, a typical problem presented is defin involve, for example, consumer electronics, personal com ing a self-consistent and useful (i.e., an improvement over a puters, control systems, and professional assistance systems. well-designed static interface) theory for altering the inter face. Therefore, even where, in a given application, a theory BACKGROUND OF THE INVENTION exists, the theory is typically not generalizable to other 0004 The prior art is rich in various systems and methods applications. Therefore, one aspect of the present invention for data analysis, as well as various systems and methods is to provide such a theory by which adaptive and/or relating to useful endeavors. In general, most existing sys responsive user interfaces may be constructed and deployed. tems and methods provide concrete functions, which have a 0009. In a particular application, the user interface defined response to a defined stimulus. Such systems, while according to the present invention is applied to general embodying the “wisdom of the designer, have a particular purpose-type computer systems, for example, personal com shortcoming in that their capabilities are static. puters. One aspect of the present invention thus relates to a programmable device that comprises a menu-driven inter 0005 Intelligent or learning systems are also known. face in which the user enters information using a direct These systems are limited by the particular paradigm manipulation input device. Such a type of interface scheme employed, and rarely are the learning algorithms general. In is disclosed in Verplank, William L., “Graphics in Human fact, while the generic theory and systems which learn are Computer Communication: Principles of Graphical User well known, the application of Such systems to particular Interface Design. Xerox Office Systems. See the references problems requires both a detailed description of the prob cited therein: Foley, J. D., Wallace, V. L., Chan, P., “The lem, as well as knowledge of the input and output spaces. Human Factor of Computer Graphics Interaction Tech Even once these factors are known, a Substantial tuning niques”, IEEE CG&A, November 1984, pp. 13-48: Koch, effort may be necessary to enable acceptable operation. H., “Ergonomische Betrachtung von Schreibtastaturen', 0006 Therefore, the present invention builds upon the Humane Production, 1. pp. 12-15 (1985); Norman, D.A., prior art, which defines various problems to be addressed, Fisher, D., “Why Alphabetic Keyboards Are Not Easy To intelligent systems and methods, tuning paradigms and user Use: Keyboard Layout Doesn't Much Matter, Human Fac interfaces. Therefore, as set forth below, and in the attached tors 24(5), pp. 509-519 (1982); Perspectives: High Technol appendix of references (including abstracts), incorporated ogy 2, 1985; Knowlton, K. “Virtual Pushbuttons as a Means herein by reference, a significant number of references detail of Person-Machine Interaction'. Proc. Conf. Computer fundamental technologies which may be improved accord Graphics, Pattern Recognition and Data Structure, Beverly ing to the present invention, or incorporated together to form Hills, Calif., May 1975, pp. 350-352: “Machine Now Reads, a part of the present invention. To the some extent, these enters Information 25 Times Faster Than Human Keyboard technologies are disclosed and are expressly incorporated Operators’, Information Display 9, p. 18 (1981); "Scanner US 2007/0016476 A1 Jan. 18, 2007

Converts Materials to Electronic Files for PCs, IEEE motion: height, length, depth, roll, pitch and yaw. Such a CG&A, December 1984, p. 76; “New Beetle Cursor Direc device may be useful to assist a user in inputting a complex tor Escapes All Surface Constraints'. Information Display description of an object, by providing Substantially more 10, p. 12, 1984; Lu, C. “Computer Pointing Devices: Living degrees of freedom sensing than minimally required by a With Mice'. High Technology, January 1984, pp. 61-65; standard graphic user interface. The many degrees of free “Finger Painting. Information Display 12, p. 18, 1981; dom available thus provide suitable input for various types Kraiss, K F., “Neuere Methoden der Interaktion an der of systems, such as “Virtual Reality” or which track a Schnittstelle Mensch-Maschine', Z. F. Arbeitswissenschaft, moving object, where many degrees of freedom and a high 2, pp. 65-70, 1978; Hirzinger, G., Landzettel, K. “Sensory degree of input accuracy is required. The Hallpot, a device Feedback Structures for Robots with Supervised Learning, which pivots a magnet about a Hall effect sensor to produce IEEE Conf. on Robotics and Automation, St. Louis, March angular orientation information, a pair of which may be used 1985; Horgan, H., “Medical Electronics', IEEE Spectrum, to provide information about two axes of displacement, January 1984, pp. 90-93. available from Elweco, Inc, Willougbhy, Ohio, may also be 0010. A menu based remote control-contained display employed as an input device. device is disclosed in Platte, Oberjatzas, and Voessing, “A 0013 User input devices may be broken down into a New Intelligent Remote Control Unit for Consumer Elec number of categories: direct inputs, i.e. touch-screen and tronic Device', IEEE Transactions on Consumer Electron light pen; indirect inputs, i.e. trackball, joystick, mouse, ics, Vol. CE–31, No.1, February 1985, 59-68. touch-tablet, bar code scanner (see, e.g., Atkinson, Terry, 0.011) A directional or direct manipulation-type sensor “VCR Programming: Making Life Easier Using Bar based infrared remote control is disclosed in Zeisel, Tomas, Codes'), keyboard, and multi-function keys; and interactive Tomaszewski, "An Interactive Menu-Driven Remote Con input, i.e. Voice activation/instructions (see, e.g., Rosch, trol Unit for TV-Receivers and VC-Recorders, IEEE Trans Winn L., “Voice Recognition: Understanding the Master's actions on Consumer Electronics, Vol. 34, No. 3, 814-818 Voice, PC Magazine, Oct. 27, 1987, 261-308); and eye (1988), which relates to a control for programming with the tracker and data Suit/data glove (see, e.g. Tello, Ernest R. West German Videotext system. This implementation differs “Between Man And Machine, Byte, September 1988, 288 from the Videotext programming system than described in 293; products of EXOS, Inc.; Data Glove). Each of the Bensch, U., VPV “VIDEOTEXT PROGRAMS VIDEO aforementioned input devices has advantages and disadvan RECORDER, IEEE Transactions on Consumer Electron tages, which are known in the art. ics, Vol. 34, No. 3, 788-792 (1988), which describes the 0014 Studies suggest that a “direct manipulation' style system of Video Program System Signal Transmitters, in of interface has advantages for menu selection tasks. This which the VCR is programmed by entering a code for the type of interface provides visual objects on a display screen, Video Program System signal, which is emitted by televi which can be manipulated by “pointing and "clicking on sion stations in West Germany. Each separate program has them. For example, the popular Graphical User Interfaces a unique identifier code, transmitted at the beginning of the (“GUIs), such as Macintosh and Microsoft Windows, and program, so that a user need only enter the code for the others known in the art, use a direct manipulation style program, and the VCR will monitor the channel for the code interface. A device Such as a touch-screen, with a more transmission, and begin recording when the code is received, natural selection technique, is technically preferable to the regardless of schedule changes. The Videotext Programs direct manipulation method. However, the accuracy limita Recorder (VPV) disclosed does not intelligently interpret the tions and relatively high cost make other inputs more transmission, rather the system reads the transmitted code as commercially practical. Further, for extended interactive a literal label, without any analysis or determination of a use, touchscreens are not a panacea for office productivity classification of the program type. applications. In addition, the user must be within arms 0012 Known manual input devices include the trackball, length of the touch-screen display. In a cursor positioning mouse, and joystick. In addition, other devices are known, task, Albert (1982) found the trackball to be the most including the so-called "J-cursor or “mousekey' which accurate pointing device and the touch-screen to be the least embeds a two (x,y) or three (x,y,p) axis pressure sensor in a accurate when compared with other input devices such as button conformed to a finger, present in a general purpose the light pen, joystick, data tablet, trackball, and keyboard. keyboard; a keyboard joystick of the type described in Epps (1986) found both the mouse and trackball to be Electronic Engineering Times, Oct. 28, 1991, p. 62, “IBM somewhat faster than both the touch-pad and joystick, but he Points a New Way'; a so-called "isobar” which provides a concluded that there were no significant performance dif two axis input by optical sensors (0. X), a two and one half ferences between the mouse and trackball as compared with axis (x, y, digital input) input device. Such as a mouse or a the touch-pad and joystick. “felix device, infrared, acoustic, etc., position sensors for 0015. It is noted that in text-based applications, an input determining the position of a finger or pointer on a display device that is accessible, without the necessity of moving the screen (touch-screen input) or on a touch surface, e.g., users hands from the keyboard, may be preferred. Thus, for “GlidePoint’ (ALPS/Cirque), goniometer input (angle posi example, Electronic Engineering Times (EET), Oct. 28, tion, Such as human joint position detector), etc. Many of 1991, p. 62, discloses a miniature joystick incorporated into Such suitable devices are Summarized in Kraiss, K. F., the functional area of the keyboard. This miniature joystick “Alternative Input Devices For Human Computer Interac has been successfully incorporated into a number of laptop tion’. Forschunginstitut Für Anthropotecahnik, Werthhoven, computers. F. R. Germany. Another device, which may also be suitable is the GyroPoint, available from Gyration Inc., which pro 0016. The following references are also relevant to the vides 2-D or 3-D input information in up to six axes of interface aspects of the present invention: p. Hoffberg, Linda US 2007/0016476 A1 Jan. 18, 2007

I, “AN IMPROVED HUMAN FACTORED INTERFACE 0037 Carlson, Mark A., “Design Goals for an Effective FOR PROGRAMMABLE DEVICES: A CASE STUDY OF User Interface', Electro/82 Proc., Mar. 1, 2001-Mar. 4, THE VCR' Master's Thesis, Tufts University (Master of 2004. Sciences in Engineering Design, November, 1990). 0038 Carlson, Mark A., “Design Goals for an Effective 0017 “Bar Code Programs VCR, Design News, Feb. 1, User Interface'. Human Interfacing with Instruments, Ses 1988, 26. sion 3. 0018) “How to find the best value in VCRs', Consumer 0039) Carroll, Paul B., “High Tech Gear Draws Cries of Reports, March 1988, 135-141. “Uncle', Wall Street Journal, Apr. 27, 1988, 29. 0019 “Low-Cost VCRs: More For Less', Consumer 0040 Cobb, Nathan, “I don't get it', Boston Sunday Reports, March 1990, 168-172. Globe Magazine, Mar. 25, 1990, 23-29. 0020 “Nielsen Views VCRs', Television Digest, Jun. 23, 0041) Davis, Fred, “The Great Look-and-Feel Debate”, 1988, 15. A+, 5:9-11 (July 1987). 0021) “The Highs and Lows of Nielsen Homevideo 0042 Dehning, Waltraud, Essig Heidrun, and Maass, Index’, Marketing & Media Decisions, November 1985, Susanne, The Adaptation of Virtual Man-Computer Inter 84-86+. faces to User Requirements in Dialogs, Germany: Springer 0022. “The Quest for User Friendly', U.S. News & Verlag, 1981. World Report, Jun. 13, 1988 54-56. 0043 Ehrenreich, S. L., “Computer Abbreviations—Evi 0023 “The Smart House: Human Factors in Home Auto dence and Synthesis'. Human Factors, 27(2): 143-155 (April mation, Human Factors in Practice, December 1990, 1-36. 1985). 0024 “VCR, Camcorder Trends”, Television Digest, Vol. 0044) Friedman, M. B., “An Eye Gaze Controlled Key 29:16 (Mar. 20, 1989). board, Proc. 2nd International Conference on Rehabilita 0025) “VCR's: A Look At The Top Of The Line”, Con tion Engineering, 1984, 446-447. sumer Reports, March 1989, 167-170. 0045 Gilfoil, D., and Mauro, C. L., “Integrating Human 0026) “VHS Videocassette Recorders”. Consumer Guide, Factors and Design: Matching Human Factors Methods up 1990, 17-20. to Product Development', C.L. Mauro Assoc., Inc., 1-7. 0046 Gould, John D. Boies, Stephen J., Meluson, Anto 0027 Abedini, Kamran, “An Ergonomically-improved nia, Rasammy, Marwan, and Vosburgh, Ann Marie, “Entry Remote Control Unit Design”, Interface 87 Proc., 375-380. and Selection Methods For Specifying Dates'. Human Fac 0028 Abedini, Kamran, and Hadad, George, "Guidelines tors, 32(2): 199-214 (April 1989). For Designing Better VCRs', Report No. IME 462, Feb. 4, 0047 Green, Lee, “Thermo Tech: Here’s a common 1987. sense guide to the new thinking thermostats. Popular 0029 Bensch, U. “VPV VIDEOTEXT PROGRAMS Mechanics, October 1985, 155-159. VIDEORECORDER, IEEE Transactions on Consumer 0048 Grudin, Jonathan, “The Case Against User Inter Electronics, 34(3):788-792. face Consistency'. MCC Technical Report Number ACA 0030) Berger, Ivan, “Secrets of the Universals”, Video, HI-002-89, January 1989. February 1989, 45-47+. 0049 Harvey, Michael G., and Rothe, James T., “Video 0.031) Beringer, D. B., “A Comparative Evaluation of Cassette Recorders: Their Impact on Viewers and Advertis Calculator Watch Data Entry Technologies: Keyboards to ers”, Journal of Advertising, 25:19-29 (December/January Chalkboards”, Applied Ergonomics, December 1985, 275 1985). 278. 0050 Hawkins, William J., “Super Remotes”, Popular 0032) Bier, E. A. et al. “MMM: A User Interface Archi Science, February 1989, 76-77. tecture for Shared Editors on a Single Screen,” Proc. ACM Symposium on User Interface Software and Technology, 0051 Henke, Lucy L., and Donohue. Thomas R., “Func Nov. 11-13, 1991, p. 79. tional Displacement of Traditional TV Viewing by VCR Owners”, Journal of Advertising Research, 29:18-24 (April 0033 Bishop, Edward W., and Guinness, G. Victor Jr., May 1989). “Human Factors Interaction with Industrial Design”. Human Factors, 8(4):279-289 (August 1966). 0.052 Hoban, Phoebe, “Stacking the Decks”, New York, Feb. 16, 1987, 20:14. 0034 Brown, Edward, “Human Factors Concepts For Management'. Proc. Human Factors Society, 1973, 372 0053. Howard, Bill, “Point and Shoot Devices”, PC 375. Magazine, 6:95-97 (August 1987). 0035 Bulkeley, Debra, “The Smartest House 0054 Jane Pauley Special, NBC TV News Transcript, in America”. Design News, Oct. 19, 1987, 56-61. Jul. 17, 1990, 10:00 PM. 0036) Card, Stuart K, “A Method for Calculating Perfor 0055 Kolson, Ann, “Computer wimps drown in a raging mance times for Users of Interactive Computing Systems’. sea of technology’. The Hartford Courant, May 24, 1989, IEEE, 1979, 653-658. B1. US 2007/0016476 A1 Jan. 18, 2007

0056 Kreifeldt, J. G., “A Methodology For Consumer 0075 Sperling, Barbara Bied, Tullis Thomas S., “Are Product Safety Analysis”. The 3rd National Symposium on You a Better Mouser or Trackballer'? A Comparison of Human Factors in Industrial Design in Consumer Products, Cursor Positioning Performance'. An Interactive/Poster August 1982, 175-184. Session at the CHI+GI’87 Graphics Interface and Human 0057 Kreifeldt, John, “Human Factors Approach to Factors in Computing Systems Conference. Medical Instrument Design. Electro/82 Proc., Mar. 3, 2001 0.076 Streeter, L. A., Ackroff, J. M., and Taylor, G. A. Mar. 3, 2006. “On Abbreviating Command Names'. The Bell System 0.058 Kuocheng, Andy Poing, and Ellingstad, Vernon S., Technical Journal, 62(6): 1807-1826 (July/August 1983). “Touch Tablet and Touch Input', Interface 87, 327. 0077 Swanson, David, and Kopfenstein, Bruce, “How to 0059) Ledgard, Henry, Singer, Andrew, and Whiteside, Forecast VCR Penetration, American Demographic, John, Directions in Human Factors for Interactive Systems, December 1987, 44-45. New York, Springer-Verlag, 1981. 0078 Tello, Ernest R. “Between Man And Machine”, 0060 Lee, Eric, and MacGregor, James, “Minimizing Byte, September 1988, 288-293. User Search Time Menu Retrieval Systems”, Human Fac 0079 Thomas, John, C., and Schneider, Michael L., tors, 27(2): 157-162 (April 1986). Human Factors in Computer Systems, New Jersey, Ablex 0061 Leon, Carol Boyd, “Selling Through the VCR', Publ. Co., 1984. American Demographics, December 1987, 40-43. 0080 Trachtenberg, Jeffrey A., “How do we confuse 0062 Long, John, “The Effect of Display Format on the thee? Let us count the ways. Forbes, Mar. 21, 1988, Direct Entry of Numerical Information by Pointing, Human 159-16O. Factors, 26(1):3-17 (February 1984). 0081) Tyldesley, D. A., “Employing Usability Engineer 0063. Mantei, Marilyn M., and Teorey, Toby J., “Cost/ ing in the Development of Office Products'. The Computer Benefit Analysis for Incorporating Human Factors in the Journal”, 31(5):431-436 (1988). Software Lifecycle'. Association for Computing Machinery, 0082 Verplank, William L., “Graphics in Human-Com 1988. puter Communication: Principles of Graphical User-Inter 0064. Meads, Jon A., “Friendly or Frivolous', Datama face Design. Xerox Office Systems. tion, Apr. 4, 1988, 98-100. 0.083 Voyt, Carlton F., “PLC’s Learn New Languages', 0065) Moore, T. G. and Dartnall, “Human Factors of a Design News, Jan. 2, 1989, 78. Microelectronic Product: The Central Heating Timer/Pro 0084. Whitefield, A. “Human Factors Aspects of Pointing grammer. Applied Ergonomics, 1983, 13(1): 15-23. as an Input Technique in Interactive Computer Systems’. 0.066 Norman, DonaldA, "Infuriating By Design, Psy Applied Ergonomics, June 1986, 97-104. chology Today, 22(3):52-56 (March 1988). 0085 Wiedenbeck, Susan, Lambert, Robin, and Scholtz, 0067. Norman, DonaldA, The Psychology of Everyday Jean, “Using Protocol Analysis to Study the User Interface'. Bulletin of the American Society for Information Science, Things, New York, Basic Book, Inc. 1988. June/July 1989, 25-26. 0068 Platte, Hans-Joachim, Oberjatzas, Gunter, and Voessing, Walter, “A New Intelligent Remote Control Unit 0.086 Wilke, William, “Easy Operation of Instruments by for Consumer Electronic Device', IEEE Transactions on Both Man and Machine. Electro/82 Proc., Mar. 2, 2001 Consumer Electronics, Vol. CE–31(1):59-68 (February Mar. 2, 2004. 1985). 0087 Yoder, Stephen Kreider, “U.S. Inventors Thrive at 0069 Rogus, John G. and Armstrong, Richard, “Use of Electronics Show”, The Wall Street Journal, Jan. 10, 1990, Human Engineering Standards in Design”. Human Factors, B1. 19(1): 15-23 (February 1977). 0088 Zeisel, Gunter, Tomas, Philippe, Tomaszewski, 0070 Rosch, Winn L., “Voice Recognition: Understand Peter, “An Interactive Menu-Driven Remote Control Unit ing the Master’s Voice', PC Magazine, Oct. 27, 1987, for TV-Receivers and VC-Recorders’, IEEE Transactions on 261-3O8. Consumer Electronics, 34(3):814-818. 0071 Sarver, Carleton, “A Perfect Friendship', High 0089 Agent Technologies Fidelity, 39:42-49 (May 1989). 0090 Presently well known human computer interfaces include so-called agent technology, in which the computer 0072 Schmitt, Lee, “Let’s Discuss Programmable Con interface learns a task defined (inherently or explicitly) by trollers’, Modern Machine Shop, May 1987, 90-99. the user and Subsequently executes the task. Such systems 0.073 Schniederman, Ben, Designing the User Interface: are available from Firefly (www.firefly.com), and are com Strategies for Effective Human-Computer Interaction, Read mercially present in Some on-line commerce systems. Such ing, Mass. Addison-Wesley, 1987. as Amazon.com (www.amazon.com). See: 0074 Smith, Sidney J., and Mosier, Jane N., Guidelines 0.091 “ABI WHAP. Web Hypertext Applications Proces for Designing User Interface Software, Bedford, Mass. sor, alphabase.com/abi3/whapinfo.html#profiling, (1996, MITRE, 1986. Jul. 11). US 2007/0016476 A1 Jan. 18, 2007

0092 "AdForce Feature Set”, www.imgis.com/in 0113 Garretson, R., “IBM Adds Drawing Assistant dex.html/core/p2—2html (1997, Apr. 11). Design Tool to Graphics Series', PC Week, Aug. 13, 1985, 0093. “IPRO,” www.ipro.com, Internet profiles Corpora vol. 2, Issue 32, p. 8. tion Home and other Web Pages (1996, Jul. 11). 0114 Gessler, S. and Kotulla A., “PDAs as mobile 0094) “Media Planning is Redefined in a New Era of WWW browsers.” Computer Networks and ISDN Systems, Online Advertising.” PR Newswire, (1996, Feb. 5). vol. 28, No. 1-2, pp. 53-59 (December 1995). 0115) Glinert-Stevens, Susan, “Microsoft Publisher: 0.095 “My Yahoo! news summary for My Yahoo! Desktop Wizardry'.PC Sources, February 1992, vol. 3, Issue Quotes', my..yahoo.com, (1997, Jan. 27). 2, p. 357. 0096 “NetGravity Announces Adserver 2.1, www.net 0116 Goldberg, Cheryl, “IBM Drawing Assistant: gravity.com/news/pressrel/launch21.html (1997, Apr. 11). Graphics for the EGA, PC Magazine, Dec. 24, 1985, vol. 0097. “Netscape & NetGravity: Any Questions?”, 4, Issue 26, p. 255. www.netgravity.com/, (1996, Jul. 11). 0.117) Hendrix, Gary G. and Walter, Brett A. “The Intel 0098 “Network Site Main”, www.doubleclick.net/ ligent Assistant: Technical Considerations Involved in frames/general/nets2set.htm (1997, Apr. 11). Designing Q&A's Natural-language Interface'. Byte Maga zine, December 1987, vol. 12, Issue 14, p. 251. 0099) “Real Media,” www.realmedia.com/index.html, (1996, Jul. 11). 0118 Hoffman, D. L. et al., “A New Marketing Paradigm for Electronic Commerce.” (1996, Feb. 19), 0100 “The Front Page', live...excite.com/?aBb (1997, www2000.ogsm. Vanderbilt.edu novak/new.marketing para Jan. 27) and (1997, Apr. 11). digm.html. 0101) “The Pointcast Network,” http:/www.pointcast 0119) Information describing BroadVision One-to-One .com, (1996, Spring). Application System: “Overview.’ p. 1: Further Resources on One-To-One Marketing, p. 1; BroadVision Unleashes the 0102) “The Power of PenPoint, Canet al., 1991, p. 39, Power of the Internet with Personalized Marketing and Chapter 13, pp. 258-260. Selling, pp. 1-3; Frequently Asked Questions, pp. 1-3: 0103) “Welcome to Lycos,” www.lycos.com, (1997, Jan. Products, p. 1: BroadVision One-To-One(TM.), pp. 1-2; 27). Dynamic Command Center, p. 1; Architecture that Scales, pp. 1-2; Technology, pp. 1; Creating a New Medium for 0104 Abatemarco, Fred, “From the Editor', Popular Marketing and Selling BroadVision One-To-One and the Science, September 1992, p. 4 World Wide Web a White Paper, pp. 1-15; www.broadvi 0105 Berniker, M., “Nielsen plans Internet Service.” sion.com (1996, January-March). Broadcasting & Cable, 125(30):34 (1995, Jul. 24). 0120) Jones, R., “Digital's World-Wide Web server: A 0106 Berry, Deanne, et al. In an Apr. 10, 1990 news case study, Computer Networks and ISDN Systems, vol. release, Symantec announced a new version of MORE 27, No. 2, pp. 297-306 (November 1994). (TM). 0121 McFadden, M., “TheWeb and the Cookie Mon 0107 Betts, M., “Sentry cuts access to naughty bits.” ster.” Digital Age, (1996, August). Computers and Security, Vol. 14, No. 7, p. 615 (1995). 0.122 Nadoli, Gajanana and Biegel, John, “Intelligent 0108 Boy, Guy A., Intelligent Assistant Systems, Har Agents in the Simulation of Manufacturing Systems. Proc. court Brace Jovanovich, 1991, uses the term “Intelligent SCS Multiconference on A and Simulation, 1989. Assistant Systems’. 0123) Nilsson, BA, “Microsoft Publisher is an Honorable Start for DTP Beginners’, Computer Shopper, February 0109) Bussey, H. E., et al., “Service Architecture, Proto 1992, vol. 12, Issue 2, p. 426, evaluates Microsoft Publisher type Description, and Network Implications of a Personal and Page Wizard. ized Information Grazing Service.” IEEE Multiple Facets of Integration Conference Proc., vol. 3, No. Conf. 9, Jun. 3, 0.124 O'Connor, Rory J., “Apple Banking on Newton's 1990, pp. 1046-1053. Brain', San Jose Mercury News, Wednesday, Apr. 22, 1992. 0110 Donnelley, J. E., “WWW media distribution via 0.125 Ohsawa, I. and Yonezawa, A., “A Computational Hopewise Reliabe Multicast,” Computer Networks and Model of an Intelligent Agent Who Talks with a Person’, ISDN Systems, vol. 27, No. 6, pp. 81-788 (April 1995). Research Reports on Information Sciences, Series C, April 0111 Edwards, John R., “Q&A: Integrated Software with 1989, No. 92, pp. 1-18. Macros and an Intelligent Assistant. Byte Magazine, Janu 0.126 Pazzani, M. et al., “Learning from hotlists and ary 1986, vol. 11, Issue 1, pp. 120-122, critiques the Intel coldlists: Towards a WWW Information Filtering and Seek ligent Assistant by Symantec Corporation. ing Agent.” Proc. International Conference on Tools with Artificial Intelligence, January 1995, pp. 492-495. 0112 Elofson, G. and Konsynski, B., “Delegation Tech nologies: Environmental Scanning with Intelligent Agents'. 0127 Poor, Alfred, “Microsoft Publisher, PC Magazine, Journal of Management Information Systems, Summer Nov. 26, 1991, vol. 10, Issue 20, p. 40, evaluates Microsoft 1991, vol. 8, Issue 1, pp. 37-62. Publisher. US 2007/0016476 A1 Jan. 18, 2007

0128 PRNewswire, information concerning the Point 0146 K. J. Astrom and B. Wittenmark, “Adaptive Con Cast Network (PCN) (1996, Feb. 13) p. 213. trol. Addison-Wesley Publishing Company (1989) pp. 105 0129 Raggett, D., “A review of the HTML +document 215. format. Computer Networks and ISDN Systems, vol. 27. 0147 K. J. Astrom, T. Hagglund, “Automatic Tuning of No. 2, pp. 35-145 (November 1994). PID Controllers’, Instrument Society of America, Research 0130 Rampe, Dan, et al. In a Jan. 9, 1989 news release, Triangle Park, N.C. (1988) pp. 105-132. Claris Corporation announced two products, SmartForm 0148 R. W. Haines, “HVAC Systems Design Hand Designer and SmartForm Assistant, which provide “Intelli book’, TAB Professional and Reference Books, Blue Ridge gent Assistance'. Such as custom help messages, choice Summit, Pa. (1988) pp. 170-177. lists, and data-entry validation and formatting. 0149 S. M. Pandit & S. M. Wu, “Timer Series & System 0131) Ratcliffe, Mitch and Gore, Andrew, “Intelligent Analysis with Applications”, John Wiley & Sons, Inc., NY Agents take U.S. Bows., MacWeek, Mar. 2, 1992, vol. 6, (1983) pp. 200-205. No. 9, p. 1. O150 T. W. Kraus 7 T. J. Myron, “Self-Tuning PID 0132) Sharif Heger, A. and Koen, B.V., “KNOWBOT: an Controller Uses Pattern Recognition Approach”. Control Adaptive Data Base Interface'. Nuclear Science and Engi Engineering, pp. 106-111, June 1984. neering, February 1991, vol. 107, No. 2, pp. 142-157. 0151. Pattern Recognition 0.133 Soviero, Marcelle M., “Your World According to 0152 Another aspect of some embodiments of the inven Newton’, Popular Science, September 1992, pp. 45-49. tion relates to signal analysis and complex pattern recogni 0134. Upendra Shardanand, “Social Information Filter tion. This aspect encompasses analysis of any data set ing for Music Recommendation September 1994, pp. 1-93, presented to the system: internal, user interface, or the Massachusetts Institute of Technology, Thesis. environment in which it operates. While semantic, optical 0135 Weber, Thomas E., “Software Lets Marketers Tar and audio analysis systems are known, the invention is by no get Web Abs.” The Wall Street Journal, Apr. 21, 1997 means limited to these types of data. 0153. Pattern recognition involves examining a complex 0136 Weiman, Liza and Moran, Tom, “A Step toward the data set to determine similarities (in its broadest context) Future', Macworld, August 1992, pp. 129-131. with other data sets, typically data sets which have been 0137 Yan, T. W. and Garcia-Molina, H., “SIFT A Tool previously characterized. These data sets may comprise for Wide-Area Information Dissemination,” Paper presented multivariate inputs, sequences in time or other dimension, or at the USENIX Technical Conference, New Orleans, La. a combination of both multivariate data sets with multiple (1995, January), pp. 177-186. dimensions. 0138) Industrial Controls 0154) The following cited patents and publications are relevant to pattern recognition and control aspects of the 0139 Industrial control systems are well known. Typi present invention, and are herein expressly incorporated by cally, a dedicated reliable hardware module controls a task reference: using a conventional algorithm, with a low level user interface. These devices are programmable, and therfore a O155 U.S. Pat. No. 5,067,163, incorporated herein by high level software program may be provided to translate reference, discloses a method for determining a desired user instructions into the low level commands, and to image signal range from an image having a single back analyze any return data. See, U.S. Pat. No. 5,506,768, ground, in particular a radiation image Such as a medical expressly incoporated herein by reference. See, also: X-ray. This reference teaches basic image enhancement techniques. 0140 A. B. Corripio, “Tuning of Industrial Control Sys tems”. Instrument Society of America, Research Triangle 0156 U.S. Pat. No. 5,068,664, incorporated herein by Park, N.C. (1990) pp. 65-81. reference, discloses a method and device for recognizing a target among a plurality of known targets, by using a 0141 C. J. Harris & S. A. Billings, “Self-Tuning and probability based recognition system. This patent document Adaptive Control: Theory and Applications'. Peter Peregri cites a number of other references, which are relevant to the nus LTD (1981) pp. 20-33. problem of image recognition: 0142. C. Rohrer & Clay Nesler, “Self-Tuning Using a 0157 Appriou, A., “Interet des theories de l’incertain en Pattern Recognition Approach”, Johnson Controls, Inc., fusion de donnees’, Colloque International sur le Radar Research Brief 228 (Jun. 13, 1986). Paris, 24-28 avril 1989. 0143 D. E. Seborg, T. F. Edgar, & D. A. Mellichamp, 0158. Appriou, A., “Procedured aide a la decision multi “Process Dynamics and Control, John Wiley & Sons, NY informateurs. Applications a la classification multi-capteurs (1989) pp. 294-307, 538-541. decibles', Symposium de l'Avionics Panel (AGARD) Tur 0144) E. H. Bristol & T. W. Kraus, “Life with Pattern quie, 25-29 avril 1988. Adaptation’. Proc. 1984 American Control Conference, pp. 0159). Arrow, K. J., “Social choice and individual 888-892, San Diego, Calif. (1984). valves”, John Wiley and Sons Inc. (1963). 0145 Francis Schied, “Shaum's Outline Series-Theory & 0.160 Bellman, R. E., L. A. Zadeh, "Decision making in Problems of Numerical Analysis, McGraw-Hill Book Co., a fuzzy environment. Management Science, 17(4) (Decem NY (1968) pp. 236, 237, 243, 244, 261. ber 1970). US 2007/0016476 A1 Jan. 18, 2007

0161 Bhatnagar, R. K. L. N. Kamal, “Handling uncertain 0.178 Roy, B., "Classements et choix en presence de information: a review of numeric and non-numeric meth points de Vue multiples, R.I.R.O.-2eme annee-no. 8, pp. ods’. Uncertainty in Artificial Intelligence, L. N. Kamal and 57-75 (1968). J. F. Lemmer, Eds. (1986). 0.179 Roy, B., “Electre III: un algorithme de classements 0162 Blair, D., R. Pollack, "La logique du choix collec fonde Sur une representation floue des preferences en pres tif Pour la Science (1983). ence de criteres multiples’. Cahiers du CERO, 2001): 3-24 0163 Chao, J. J. E. Drakopoulos, C. C. Lee, “An evi (1978). dential reasoning approach to distributed multiple hypoth 0180 Scharlic, A., “Decider sur plusieurs criteres. Pan esis detection', Proc. 20th Conference on decision and orama de laide a la decision multicritere” Presses Polytech control, Los Angeles, Calif., December 1987. niques Romandes (1985). 0164 Dempster, A. P., “A generalization of Bayesian 0181 Shafer, G., “A mathematical theory of evidence'. inference'. Journal of the Royal Statistical Society, Vol. 30. Princeton University Press, Princeton, N.J. (1976). Series B (1968). 0182 Sugeno, M.. “Theory of fuzzy integrals and its 0165 Dempster, A. P. “Upper and lower probabilities applications”, Tokyo Institute of Technology (1974). induced by a multivalued mapping. Annals of mathematical Statistics, no. 38 (1967). 0183 Vannicola et al., “Applications of Knowledge based Systems to Surveillance', Proc. 1988 IEEE National Radar 0166 Dubois, D., “Modeles mathematiques de l'imprecis Conference, 20-21 April 1988, pp. 157-164. et de l’incertain en Vue d'applications aux techniques d'aide a la decision”. Doctoral Thesis, University of Grenoble 0.184 Yager, R. R., “Entropy and specificity in a math (1983). ematical theory of Evidence'. Int. J. General Systems, 9:249-260 (1983). 0167 Dubois, D., N. Prade, “Combination of uncertainty with belief functions: a reexamination', Proc. 9th Interna 0185. Zadeh, L.A., “Fuzzy sets as a basis for a theory of tional Joint Conference on Artificial Intelligence, Los Ange possibility', Fuzzy sets and Systems 1:3-28 (1978). les (1985). 0186 Zadeh, L. A., “Fuzzy sets’, Information and Con 0168 Dubois, D., N. Prade, "Fuzzy sets and systems trol, 8:338-353 (1965). Theory and applications”. Academic Press, New York 0187 Zadeh, L. A., “Probability measures of fuzzy (1980). events”, Journal of Mathematical Analysis and Applications, 0169) Du ois, D., N. Prade, “Theorie des possibilites: 23:421-427 (1968). application a la representation des connaissances en infor 0188 U.S. Pat. No. 5,067,161, incorporated herein by matique', Masson, Paris (1985). reference, relates to a video image pattern recognition sys 0170 Duda, R. O., P. E. Hart, M. J. Nilsson, “Subjective tem, which recognizes objects in near real time. Bayesian methods for rule-based inference systems, Tech 0189 U.S. Pat. Nos. 4,817,176 and 4,802,230, both nical Note 124-Artificial Intelligence Center-SRI Interna incorporated herein by reference, relate to harmonic trans tional. form methods of pattern matching of an undetermined 0171 Fua, P. V., “Using probability density functions in pattern to known patterns, and are useful in the pattern the framework of evidential reasoning Uncertainty in knowl recognition method of the present invention. U.S. Pat. No. edge based systems”. B. Bouchon, R. R. Yager, Eds. 4,998.286, incorporated herein by reference, relates to a Springer Verlag (1987). harmonic transform method for comparing multidimen sional images, such as color images, and is useful in the 0172 Ishizuka, M., “Inference methods based on present pattern recognition methods. extended Dempster and Shafer's theory for problems with uncertainty/fuzziness”, New Generation Computing, 1:159 0.190 U.S. Pat. No. 5,067,166, incorporated herein by 168 (1983), Ohmsha, Ltd, and Springer Verlag. reference, relates to a pattern recognition system, in which a local optimum match between subsets of candidate refer 0173 Jeffrey, R. J., “The logic of decision”, The Univer ence label sequences and candidate templates. It is clear that sity of Chicago Press, Ltd., London (1983) (2nd Ed.). this method is useful in the pattern recognition aspects of the present invention. It is also clear that the interface and 0174 Kaufmann, A., “Introduction a la theorie des sous control system of the present invention are useful adjuncts to ensembles flous”, Vol. 1, 2 et 3-Masson-Paris (1975). the method disclosed in U.S. Pat. No. 5,067,166. 0175 Keeney, R. L., B. Raiffa, “Decisions with multiple 0191 U.S. Pat. No. 5,048,095, incorporated herein by objectives: Preferences and value tradeoffs', John Wiley and reference, relates to the use of a genetic learning algorithm Sons, New York (1976). to adaptively segment images, which is an initial stage in 0176 Ksienski et al., “Low Frequency Approach to Tar image recognition. This patent has a software listing for this get Identification', Proc. IEEE. 63(12):1651-1660 (Decem method. It is clear that this method is useful in the pattern ber 1975). recognition aspects of the present invention. It is also clear that the interface and control system of the present invention 0177 Kyburg, H. E., “Bayesian and non Bayesian evi are useful adjuncts to the method disclosed in U.S. Pat. No. dential updating. Artificial Intelligence 31:271-293 (1987). 5,048,095. US 2007/0016476 A1 Jan. 18, 2007

0192 Fractal-Based Image Processing 0204 A. Jacquin, “A Fractal Theory of Iterated Markov 0193 Fractals are a relatively new field of science and Operators with Applications to Digital Image Coding, PhD technology that relate to the study of order and chaos. While Thesis, Georgia Tech, 1989. the field of fractals is now very dense, a number of relevant 0205 A. Jacquin, “Image Coding Based on a Fractal principles are applicable. First, when the coordinate axes of Theory of Iterated Contractive Image Transformations a space are not independent, and are related by a recursive p. 18, January 1992 (Vol 1 Issue 1) of IEEE Trans on Image algorithm, then the space is considered to have a fractional Processing. dimensionality. One characteristic of Such systems is that a mapping of Such spaces tends to have self-similarity on a 0206 A. Jacquin, Fractal image coding based on a number of scales. Interestingly, natural systems have also theory of iterated contractive image transformations, Proc. been observed to have self-similarity over several orders of SPIE Visual Communications and Image Processing, 1990, magnitude, although as presently believed, not over an pages 227-239. unlimited range of scales. Therefore, one theory holds that 0207 A. E. Jacquin, A novel fractal block-coding tech images of natural objects may be efficiently described by nique for digital images, Proc. ICASSP 1990. iterated function systems (IFS), which provide a series of parameters for a generic formula or algorithm, which, when 0208 Baldwin, William, “Just the Bare Facts, Please”, the process is reversed, is visually similar to the starting Forbes Magazine, Dec. 12, 1988. image. Since the “noise' of the expanded data is masked by the “natural appearance of the result, visually acceptable 0209 Barnsley et al., “A Better Way to Compress image compression may be provided at relatively high Images”. Byte Magazine, January 1988, pp. 213-225. compression ratios. This theory remains the Subject of 0210 Barnsley et al., “Chaotic Compression’, Computer significant debate, and, for example, wavelet algorithm Graphics World, November 1987. advocates claim Superior results for a more general set of starting images. It is noted that, on a mathematical level. 0211 Barnsley et al., “Harnessing Chaos For Images wavelets and fractal theories have some common threads. Synthesis’, Computer Graphics, 22(4): 131-140 (August, 1988). 0194 According to a particular embodiment of the inven tion, the expression of an image as an ordered set of 0212 Barnsley et al., “Hidden Variable Fractal Interpo coefficients of an algorithm, wherein the coefficients relate lation Functions”. School of Mathematics, Georgia Institute to elements of defined variation in scale, and the resulting set of Technology, Atlanta, Ga. 30332, July 1986. of coefficients is related to the underlying image morphol ogy, is exploited in order to provide a means for pattern 0213 Barnsley, M. F., “Fractals Everywhere', Academic analysis and recognition without requiring decompression to Press, Boston, Mass., 1988. an orthogonal coordinate space. 0214 Barnsley, M. F., and Demko, S., “Iterated Function 0195 U.S. Pat. Nos. 5,065.447, and 4.941,193, both Systems and The Global Construction of Fractals'. Proc. R. incorporated herein by reference, relate to the compression Soc. Lond. A 99:243-275 (1985). of image data by using fractal transforms. These are dis 0215 Barnsley, M. F., Ervin, V., Hardin, D., Lancaster, J., cussed in detail below. U.S. Pat. No. 5,065,447 cites a “Solution of an Inverse Problem for Fractals and Other number of references, relevant to the use of fractals in image Sets”, Proc. Natl. Acad. Sci. U.S.A., 83:1975-1977 (April processing: 1986). 0196) U.S. Pat. No. 4,831,659. 0216 Beaumont J. M. “Image data compression using 0197) “A New Class of Markov Processes for Image fractal techniques’’. British Telecom Technological Journal Encoding. School of Mathematics, Georgia Inst. of Tech 9(4):93-108 (1991). nology (1988), pp. 14-32. 0217 Byte Magazine, January 1988, supra, cites: 0198 “Construction of Fractal Objects with Iterated 0218. D. S. Mazel, Fractal Modeling of Time-Series Function Systems, Siggraph 85 Proc., 19(3):271-278 Data, PhD Thesis, Georgia Tech, 1991. (One dimensional, (1985). not pictures). 0199 “Data Compression: Pnting by Numbers”, The 0219 Derra, Skip, “Researchers Use Fractal Geometry, Economist, May 21, 1988. ... Research and Development Magazine, March 1988. 0200 "Fractal Geometry-Understanding Chaos', Geor 0220 Elton, J., “An Ergodic Theorem for Iterated Maps', gia Tech Alumni Magazine, p. 16 (Spring 1986). Journal of Ergodic Theory and Dynamical Systems, 7 0201 “Fractal Modelling of Biological Structures, Per (1987). spectives in Biological Dynamics and Theoretical Medicine, Koslow, Mandell, Shlesinger, eds. Annals of New York 0221 Fisher Y. "Fractal image compression . Siggraph Academy of Sciences, vol. 504, 179-194 (date unknown). 92. 0202 "Fractal Modelling of Real World Images, Lecture 0222 Fractal Image Compression Michael F. Barnsley Notes for Fractals: Introduction, Basics and Perspectives'. and Lyman P. Hurd ISBN 0-86720-457-5, ca. 250 pp. Siggraph (1987). 0223 Fractal Image Compression: Theory and Applica 0203 "Fractals-A Geometry of Nature', Georgia Institute tion, Yuval Fisher (ed.), Springer Verlag, N.Y., 1995. ISBN of Technology Research Horizons, p. 9 (Spring 1986). number 0-387-94211-4. US 2007/0016476 A1 Jan. 18, 2007

0224 Fractal Modelling of Biological Structures, School may also be processed by processes such as afline trans of Mathematics, Georgia Institute of Technology (date forms. Then, for each domain block, the mapped range block unknown). which is most similar to the domain block is determined, and the address of that range block and the processes the blocks 0225 G. E. Oien, S. Lepsoy & T A. Ramstad, An inner were subjected to, are combined as an identifier which is product space approach to image coding by contractive appended to a list of identifiers for other domain blocks. The transformations, Proc. ICASSP 1991, pp 2773-2776. list of identifiers for all domain blocks is called a fractal 0226 Gleick, James, “Making a New Science', pp. 215, transform and constitutes a compressed representation of the 239, date unknown. input image. To decompress the fractal transform and recover the input image, an arbitrary input image is formed 0227 Graf S, “Barnsley's Scheme for the Fractal Encod into range blocks and the range blocks processed in a ing of Images, Journal Of Complexity, V8, 72-78 (1992). manner specified by the identifiers to form a representation 0228 Jacobs, E. W., Y. Fisher and R. D. Boss. “Image of the original input image. Compression: A study of the Iterated Transform Method. 0242 “Image Compression Using Fractals and Wave Signal Processing 29, (1992) 25-263. lets”. Final Report for the Phase II Contract Sponsored by 0229 M. Barnsley, L. Anson, “Graphics Compression the Office of Naval Research, Contract No. N00014-91-C- Technology, Sun World, October 1991, pp. 42-52. 0117, Netrologic Inc., San Diego, Calif. (Jun. 2, 1993), relates to various methods of compressing image data, 0230 M. F. Barnsley, A. Jacquin, F. Malassenet, L. including fractals and wavelets. This method may also be Reuter & A. D. Sloan, Harnessing chaos for image synthe applicable in pattern recognition applications. This reference sis, Computer Graphics, Vol 22 no 4 pp 131-140, 1988. provides theory and comparative analysis of compression 0231 M. F. Barnsley, A. E. Jacquin, Application of schemes. recurrent iterated function systems to images. Visual 0243 A fractal-processing method based image extrac Comm. and Image Processing, vol SPIE-1001, 1988. tion method is described in Kim, D. H., Caulfield, H. J.; 0232) Mandelbrot, B., “The Fractal Geometry of Nature', Jannson, T.: Kostrzewski, A: Savant, G., “Optical fractal W.H. Freeman & Co., San Francisco, Calif., 1982, 1977. image processor for noise-embedded targets detection'. Proc. SPIE The International Society for Optical Engi 0233 Monro D M and Dudbridge F. “Fractal block neering, Vol. 2026, p. 144-9 (1993) (SPIE Conf: Photonics coding of images, Electronics Letters 28(11):1053-1054 for Processors, Neural Networks, and Memories 12-15 July (1992). 1993, San Diego, Calif., USA). According to this paper, a 0234 Monro D. M. & Dudbridge F. Fractal approxima fractal dimensionality measurement and analysis-based tion of image blocks, Proc ICASSP 92, pp. III: 485-488. automatic target recognition (ATR) is described. The ATR is a multi-step procedure, based on fractal image processing, 0235 Monro D. M. A hybrid fractal transform, Proc and can simultaneously perform preprocessing, interest ICASSP 93, pp. V: 169-72. locating, segmenting, feature extracting, and classifying. 0236 Monro D. M., Wilson D., Nicholls J. A. High speed See also, Cheong, C. K.: Aizawa, K.; Saito, T.; Hatori, M., image coding with the Bath Fractal Transform, IEEE Inter “Adaptive edge detection with fractal dimension'. Transac national Symposium on Multimedia Technologies tions of the Institute of Electronics Information and Com Southampton, April 1993. munication Engineers D-II, J76D-II(11): 2459-63 (1993); Hayes, H.I.; Solka, J. L.; Priebe, C. E.; “Parallel computa 0237 Peterson, Ivars, “Packing It In-Fractals.”. Science tion of fractal dimension', Proc. SPIE The International News, 131(18):283-285 (May 2, 1987). Society for Optical Engineering, 1962:219-30 (1993), 0238 S. A. Hollatz. “Digital image compression with Priebe, C. E.; Solka, J. L.; Rogers, G. W., “Discriminant two-dimensional affine fractal interpolation functions', analysis in aerial images using fractal based features'. Proc. Department of Mathematics and Statistics, University of SPIE The International Society for Optical Engineering, Minnesota-Duluth, Technical Report 91-2. (a nuts-and-bolts 1962:196-208(1993). See also, Anson, L., “Fractal Image how-to-do-it paper on the technique). Compression', Byte, October 1993, pp. 195-202, “Fractal 0239 Stark, J., “Iterated function systems as neural net Compression Goes On-Line', Byte, September 1993. works, Neural Networks, Vol 4, pp 679-690, Pergamon 0244 Methods employing other than fractal-based algo Press, 1991. rithms may also be used. See, e.g., Liu, Y., “Pattern recog nition using Hilbert space'. Proc. SPIE The International 0240 Vrscay, Edward R. “Iterated Function Systems: Society for Optical Engineering, 1825:63-77 (1992), which Theory, Applications, and the Inverse Problem. Fractal describes a learning approach, the Hilbert learning. This Geometry and Analysis, J. Belair and S. Dubuc (eds.) approach is similar to Fractal learning, but the Fractal part Kluwer Academic, 1991. 405-468. is replaced by Hilbert space. Like the Fractal learning, the 0241 U.S. Pat. No. 5,347,600, incorporated herein by first stage is to encode an image to a small vector in the reference, relates to a method and apparatus for compression internal space of a learning system. The next stage is to and decompression of digital image data, using fractal quantize the internal parameter space. The internal space of methods. According to this method, digital image data is a Hilbert learning system is defined as follows: a pattern can automatically processed by dividing Stored image data into be interpreted as a representation of a vector in a Hilbert domain blocks and range blocks. The range blocks are space. Any vectors in a Hilbert space can be expanded. If a Subjected to processes such as a shrinking process to obtain vector happens to be in a subspace of a Hilbert space where mapped range blocks. The range blocks or domain blocks the dimension L of the subspace is low (order of 10), the US 2007/0016476 A1 Jan. 18, 2007

vector can be specified by its norm, an L-vector, and the Xie: Brady, M., “Fractal dimension image for texture seg Hermitian operator which spans the Hilbert space, estab mentation', IRCV '92. Second International Conference on lishing a mapping from an image space to the internal space Automation, Robotics and Computer Vision, p. CV-4.3/1-5 P. This mapping converts an input image to a 4-tuple: t in vol.1, (1992). According to this reference, the fractal dimen P=(Norm, T. N. L-vector), where T is an operator parameter sion and its change over boundaries of different homoge space, N is a set of integers which specifies the boundary neous textured regions is analyzed and used to segment condition. The encoding is implemented by mapping an textures in infrared aerial images. Based on the fractal input pattern into a point in its internal space. The system dimension, different textures map into different fractal uses local search algorithm, i.e., the system adjusts its dimension image features, such that there is Smooth varia internal data locally. The search is first conducted for an tion within a single homogeneous texture but sharp variation operator in a parameter space of operators, then an error at texture boundaries. Since the fractal dimension remains function delta (t) is computed. The algorithm stops at a local unchanged under linear transformation, this method is minimum of delta (t). Finally, the input training set divides robust for dismissing effects caused by lighting and other the internal space by a quantization procedure. See also, Liu, extrinsic factors. Morphology is the only tool used in the Y., “Extensions of fractal theory”, Proc. SPIE The Inter implementation of the whole process: texture feature extrac national Society for Optical Engineering, 1966:255 tion, texture segmentation and boundary detection. This 68(1993). makes possible parallel implementations of each stage of the 0245 Fractal methods may be used for pattern recogni process. tion. See, Sadjadi, F., “Experiments in the use of fractal in 0249 Rahmati, M.; Hassebrook, L.G., “Intensity- and computer pattern recognition, Proc. SPIE The Interna distortion-invariant pattern recognition with complex linear tional Society for Optical Engineering, 1960:214-22(1993). morphology’. Pattern Recognition, 27 (4):549-68(1994) According to this reference, man-made objects in infrared relates to a unified model based pattern recognition approach and millimeter wave (MMW) radar imagery may be recog is introduced which can be formulated into a variety of nized using fractal-based methods. The technique is based techniques to be used for a variety of applications. In this on estimation of the fractal dimensions of sequential blocks approach, complex phasor addition and cancellation are of an image of a scene and slicing of the histogram of the incorporated into the design of filter(s) to perform implicit fractal dimensions computed by Fourier regression. The logical operations using linear correlation operators. These technique is shown to be effective for the detection of implicit logical operations are Suitable to implement high tactical military vehicles in IR, and of airport attributes in level gray scale morphological transformations of input MMW radar imagery. images. In this way non-linear decision boundaries are 0246. In addition to spatial self-similarity, temporal self effectively projected into the input signal space yet the similarity may also be analyzed using fractal methods. See, mathematical simplicity of linear filter designs is main Reusens, E., “Sequence coding based on the fractal theory of tained. This approach is applied to the automatic distortion iterated transformations systems”. Proc. SPIE The Inter and intensity-invariant object recognition problem. A set of national Society for Optical Engineering, 2094(pt. 1): 132 shape operators or complex filters is introduced which are 40(1993). This reference describes a scheme based on the logically structured into a filter bank architecture to accom iterated functions systems theory which relies on a 3D plish the distortion and intensity-invariant system. This approach in which the sequence is adaptively partitioned. synthesized complex filter bank is optimally sensitive to Each partition block can be coded either by using the spatial fractal noise representing natural Scenery. The sensitivity is self similarities or by exploiting temporal redundancies. optimized for a specific fractal parameter range using the 0247 Fractal compression methods may be used for Fisher discriminant. The output responses of the proposed Video data for transmission. See, Hurtgen, B.: Buttgen, P. system are shown for target, clutter, and pseudo-target inputs “Fractal approach to low rate video coding'. Proc. SPIE to represent its discrimination and generalization capability The International Society for Optical Engineering, in the presence of distortion and intensity variations. Its 2094(pt. 1):120-31(1993). This reference relates to a method performance is demonstrated with realistic scenery as well for fast encoding and decoding of image sequences on the as synthesized inputs. basis of fractal coding theory and the hybrid coding concept. 0250 Sprinzak, J.; Werman, M., “Affine point matching, The DPCM-loop accounts for statistical dependencies of Pattern Recognition Letters, 15(4):337-9(1994), relates to a natural image sequences in the temporal direction. Those pattern recognition method. A fundamental problem of pat regions of the original image where the prediction, i.e. tern recognition, in general, is recognizing and locating motion estimation and compensation, fails are encoded objects within a given scene. The image of an object may using an advanced fractal coding scheme, Suitable for still have been distorted by different geometric transformations images, and whose introduction instead of the commonly Such as translation, rotation, Scaling, general affine transfor used Discrete Cosine Transform (DCT) -based coding is mation or perspective projection. The recognition task advantageous especially at very low bit rates (8-64 kbit/s). involves finding a transformation that Superimposes the In order to increase reconstruction quality, encoding speed model on its instance in the image. This reference proposes and compression ratio, some additional features such as an improved method of Superimposing the model. hierarchical codebook search and multilevel block segmen tation may be employed. This hybrid technique may be used 0251 Temporal Image Analysis in conjunction with the present adaptive interface or other 0252) Temporal image analysis is a well known field. features of the present invention. This field holds substantial interest at present for two 0248 Fractal methods may be used to segment an image reasons. First, by temporal analysis of a series of two into objects having various surface textures. See, Zhi-Yan dimensional images, objects and object planes may be US 2007/0016476 A1 Jan. 18, 2007

defined, which provide basis for efficient yet general algo in this manner. Further, it is clear that this recognition feature rithms for video compression, such as the Motion Picture may form an integral part of certain embodiments of the Experts Group (MPEG) series of standards. Second, tem present invention. It is also clear that the various features of poral analysis has applications in signal analysis for an the present invention would be applicable as an adjunct to understanding and analysis of the signal itself. the various elements of the system disclosed in U.S. Pat. No. 0253 U.S. Pat. No. 5,280,530, incorporated herein by 5,063,603. reference, relates to a method and apparatus for tracking a 0256 U.S. Pat. No. 5,067,160, incorporated herein by moving object in a scene, for example the face of a person reference, relates to a motion-pattern recognition apparatus, in videophone applications, comprises forming an initial having adaptive capabilities. The apparatus recognizes a template of the face, extracting a mask outlining the face, motion of an object that is moving and is hidden in an image dividing the template into a plurality (for example sixteen) signal, and discriminates the object from the background Sub-templates, searching the next frame to find a match with within the signal. The apparatus has an image-forming unit the template, searching the next frame to find a match with comprising non-linear oscillators, which forms an image of each of the Sub-templates, determining the displacements of the motion of the object in accordance with an adjacent each of the Sub-templates with respect to the template, using mutual-interference-rule, on the basis of the image signal. A the displacements to determine affine transform coefficients memory unit, comprising non-linear oscillators, stores con and performing an affine transform to produce an updated ceptualized meanings of several motions. A retrieval unit template and updated mask. retrieves a conceptualized meaning close to the motion image of the object. An altering unit alters the rule, on the 0254 U.S. Pat. No. 5,214,504 relates to a moving video basis of the conceptualized meaning. The image forming image estimation system, based on an original video image unit, memory unit, retrieval unit and altering unit form a of time n and time n+1, the centroid, the principal axis of holonic-loop. Successive alterations of the rules by the inertia, the moment about the principal axis of inertia and the altering unit within the holonic loop change an ambiguous moment about the axis perpendicular to the principal axis of image formed in the image forming unit into a distinct inertia are obtained. By using this information, an affine image. U.S. Pat. No. 5,067,160 cites the following refer transformation for transforming the original video image at ences, which are relevant to the task of discriminating a time n to the original video image at time n+1 is obtained. moving object in a background: Based on the infinitesimal transformation (A), {e, and e''} obtained by making the affine transformation con 0257 U.S. Pat. No. 4,710,964. tinuous with regard to time is executed on the original video 0258 Shimizu et al., “Principle of Holonic Computer and image at time n and time n+1. The results are synthesized to perform an interpolation between the frames. e''} is Holovision', Journal of the Institute of Electronics, Infor applied to the original video system time n+1. The video mation and Communication, 70(9):921-930 (1987). image after time n+1 is thereby protected. 0259 Omata et al., “Holonic Model of Motion Percep tion', IEICE Technical Reports, Mar. 26, 1988, pp. 339-346. 0255 U.S. Pat. No. 5,063,603, incorporated herein by reference, relates to a dynamic method for recognizing 0260 Ohsuga et al., “Entrainment of Two Coupled van objects and image processing system therefor. This refer der Pol Oscillators by an External Oscillation', Biological ence discloses a method of distinguishing between different Cybernetics, 51:225-239 (1985). members of a class of images, such as human beings. A time 0261 U.S. Pat. No. 5,065.440, incorporated herein by series of Successive relatively high-resolution frames of reference, relates to a pattern recognition apparatus, which image data, any frame of which may or may not include a compensates for, and is thus insensitive to pattern shifting, graphical representation of one or more predetermined spe thus being useful for decomposing an image or sequence of cific members (e.g., particular known persons) of a given images, into various structural features and recognizing the generic class (e.g. human beings), is examined in order to features. U.S. Pat. No. 5,065.440 cites the following refer recognize the identity of a specific member; if that members ences, incorporated herein by reference, which are also image is included in the time series. The frames of image relevant to the present invention: U.S. Pat. Nos. 4.543,660, data may be examined in real time at various resolutions, 4,630,308, 4,677,680, 4,809,341, 4,864,629, 4,872,024 and starting with a relatively low resolution, to detect whether 4,905,296. Some earlier-occurring frame includes any of a group of image features possessed by an image of a member of the 0262 Recent analyses of fractal image compression tech given class. The image location of a detected image feature niques have tended to imply that, other than in special is stored and then used in a later-occurring, higher resolution circumstances, other image compression methods are “bet frame to direct the examination only to the image region of ter” than a Barnsley-type image compression system, due to the stored location in order to (1) verify the detection of the the poor performance of compression processors and lower aforesaid image feature, and (2) detect one or more other of than expected compression ratios. Further, Statements attrib the group of image features, if any is present in that image uted to Barnsley have indicated that the Barnsley technique region of the frame being examined. By repeating this type is not truly a “fractal technique, but rather a vector quan of examination for later and later occurring frames, the tization process which employs a recursive library. Never accumulated detected features can first reliably recognize theless, these techniques and analyses have their advantages. the detected image region to be an image of a generic object As stated hereinbelow, the fact that the codes representing of the given class, and later can reliably recognize the the compressed image are hierarchical represents a particu detected image region to be an image of a certain specific lar facet exploited by the present invention. member of the given class. Thus, a human identity recog 0263. Another factor which makes fractal methods and nition feature of the present invention may be implemented analysis relevant to the present invention is the theoretical US 2007/0016476 A1 Jan. 18, 2007 relation to optical image processing and holography. Thus, 0277 Lippmann, R. P., “An Introduction to Computing while Such optical systems may presently be cumbersome with Neural Nets', IEEE ASSP Magazine, 4(2):4-22 (April and economically unfeasible, and their implementation in 1987). software models slow, these techniques nevertheless hold 0278 Rumelhart, D. E., et al., Parallel Distr. Proc.: promise and present distinct advantages. Explorations in Microstructure of Cognition, vol. 1, 1986, 0264 Biometric Analysis “Learning Internal Representations by Error Propagation'. 0265 Biometric analysis comprises the study of the dif pp. 318-362. ferences between various organisms, typically of the same 0279 U.S. Pat. Nos. 5,048,100, 5,063,601 and 5,060,278, species. Thus, the intraspecies variations become the basis all incorporated herein by reference, also relate to neural for differentiation and identification. In practice, there are network adaptive pattern recognition methods and appara many applications for biometric analysis systems, for tuses. It is clear that the methods of U.S. Pat. Nos. 5,048, example in security applications, these allow identification 100, 5,060,278 and 5,063,601 may be used to perform the of a particular human. adaptive pattern recognition functions of the present inven tion. More general neural networks are disclosed in U.S. Pat. 0266 U.S. Pat. No. 5,055,658, incorporated herein by Nos. 5,040,134 and 5,058,184, both incorporated herein be reference, relates to a security system employing digitized reference, which provide background on the use of neural personal characteristics, such as Voice. The following ref networks. In particular, U.S. Pat. No. 5,058,184 relates to the erences are cited: use of the apparatus in information processing and feature 0267) “Voice Recognition and Speech Processing, Ele detection applications. ktor Electronics, September 1985, pp. 56-57. 0280 U.S. Pat. No. 5,058,180, incorporated herein by 0268) Naik et al., “High Performance Speaker Verifica reference, relates to neural network apparatus and method tion.”, ICASSP 86, Tokyo, CH2243-4/86/0000-0881, IEEE for pattern recognition, and is thus relevant to the intelligent 1986, pp. 881-884. pattern recognition functions of the present invention. This patent cites the following documents of interest: 0269. Shinan et al., “The Effects of Voice Disguise.”, ICASSP 86, Tokyo, CH2243-4/86/0000-0885, IEEE 1986, 0281 U.S. Pat. Nos. 4,876,731 and 4,914,708. pp. 885-888. 0282 Carpenter, G. A. S. Grossberg, “The Art of Adap 0270 Parts of this system relating to speaker recognition tive Pattern Recognition by a Self-Organizing Neural Net may be used to implement a voice recognition system of the work.” IEEE Computer, March 1988, pp. 77-88. present invention for determining an actor or performer in a 0283 Computer Visions, Graphics, and Image Process broadcast. ing 1987, 37:54-115. 0271 Neural Networks 0284 Grossberg, S., G. Carpenter, “A Massively Parallel 0272 Neural networks are a particular type of data analy Architecture for a Self-Organizing Neural Pattern Recogni sis tool. There are characterized by the fact that the network tion Machine.” Computer Vision, Graphics, and Image Pro is represented by a set of “weights’, which are typically cessing (1987, 37, 54-115), pp. 252-315. scalar values, which are derived by a formula which is 0285 Gullichsen E. E. Chang, “Pattern Classification by designed to reduce the error between a data pattern repre Neural Network: An Experiment System for Icon Recogni senting a known State and the network’s prediction of that tion,” ICNN Proceeding on Neural Networks, March 1987, state. These networks, when provided with sufficient com pp. IV-725-32. plexity and an appropriate training set, may be quite sensi tive and precise. Further, the data pattern may be arbitrarily 0286) Jackel, L. D., H. P. Graf, J. S. Denker, D. Hend complex (although the computing power required to evalu erson and I. Guyon, “An Application of Neural Net Chips: ate the output will also grow) and therefore these systems Handwritten Digit Recognition.” ICNN Proceeding, 1988, may be employed for video and other complex pattern pp. 11-107-15. analysis. 0287 Lippman, R. P., “An Introduction to Computing 0273 U.S. Pat. No. 5,067,164, incorporated herein by with Neural Nets.” IEEE ASSP Magazine, April 1987, pp. reference, relates to a hierarchical constrained automatic 4-22. learning neural network for character recognition, and thus 0288 Pawlicki, T. F., D. S. Lee, J. J. Hull and S. N. represents an example of a trainable neural network for Srihari, “Neural Network Models and their Application to pattern recognition, which discloses methods which are Handwritten Digit Recognition.” ICNN Proceeding, 1988, useful for the present invention. This Patent cites various pp. 11-63-70. references of interest: 0289 Chao, T.-H.; Hegblom, E.; Lau, B.: Stoner, W. W. 0274 U.S. Pat. Nos. 4,760,604, 4,774,677 and 4,897,811. Miceli, W. J., “Optoelectronically implemented neural net work with a wavelet preprocessor, Proc. SPIE The Inter 0275) LeCun, Y., Connectionism in Perspective, R. Pfei national Society for Optical Engineering, 2026:472 fer, Z. Schreter, F. Fogelman, L. Steels, (Eds.), 1989, “Gen 82(1993), relates to an optoelectronic neural network based eralization and Network Design Strategies’, pp. 143-55. upon the Neocognitron paradigm which has been imple 0276 LeCun, Y., et al., “Handwritten Digit Recognition: mented and Successfully demonstrated for automatic target Applications of Neural.'', IEEE Comm. Magazine, pp. 41-46 recognition for both focal plane array imageries and range (November 1989). Doppler radar signatures. A particular feature of this neural US 2007/0016476 A1 Jan. 18, 2007

network architectural design is the use of a shift-invariant 0299 Rhodes, W., "Acousto-Optic Signal Processing: multichannel Fourier optical correlation as a building block Convolution and Correlation, Proc. IEEE, 69(1):65-79 for iterative multilayer processing. A bipolar neural weights (January 1981). holographic synthesis technique was utilized to implement both the excitatory and inhibitory neural functions and 0300 Vander Lugt, A., “Signal Detection. By Complex increase its discrimination capability. In order to further Spatial Filtering, IEEE Transactions On Information increase the optoelectronic Neocognitron's self-organization Theory, IT-10, 2:139-145 (April 1964). processing ability, a wavelet preprocessor was employed for 0301 U.S. Pat. Nos. 5,159,474 and 5,063,602, expressly feature extraction preprocessing (orientation, size, location, incorporated herein by reference, also relate to optical image etc.). A multichannel optoelectronic wavelet processor using correlators. Also of interest is Li, H. Y. Y. Qiao and D. an e-beam complex-valued wavelet filter is also described. Psaltis, Applied Optics (April, 1993). See also, Bains, S., 0290 Neural networks are important tools for extracting “Trained Neural Network Recognizes Faces”. Laser Focus patterns from complex input sets. These systems do not World, June, 1993, pp. 26-28; Bagley, H. & Sloan, J., require human comprehension of the pattern in order to be “Optical Processing: Ready For Machine Vision?, Photo useful, although human understanding of the nature of the nics Spectra, August 1993, pp. 101-106. problem is helpful in designing the neural network system, 0302 Optical pattern recognition has been especially as is known in the art. Feedback to the neural network is applied to two dimensional patterns. In an optical pattern integral to the training process. Thus, a set of inputs is recognition system, an image is correlated with a set of mapped to a desired output range, with the network mini known image patterns represented on a hologram, and the mizing an "error” for the training data set. Neural networks product is a pattern according to a correlation between the may differ based on the computation of the "error, the input pattern and the provided known patterns. Because this optimization process, the method of altering the network to is an optical technique, it is performed nearly instanta minimize the error, and the internal topology. Such factors neously, and the output information can be reentered into an are known in the art. electronic digital computer through optical transducers known in the art. Such a system is described in Casasent, D., 0291. Optical Pattern Recognition Photonics Spectra, November 1991, pp. 134-140. The ref 0292 Optical image processing holds a number of advan erences cited therein provide further details of the theory and tages. First, images are typically optical by their nature, and practice of Such a system: Lendaris, G. G., and Stanely, G. therefore processing by this means may (but not always) L., “Diffraction Pattern Sampling for Automatic Target avoid a data conversion. Second, many optical image pro Recognition, Proc. IEEE 58:198-205 (1979); Ballard, D. cessing schemes are inherently or easily performed in par H., and Brown, C. M. ComputerVision, Prentice Hall, allel, improving throughput. Third, optical circuits typically Englewood Cliffs, N.J. (1982); Optical Engineering 28:5 have response times shorter than electronic circuits, allow (May 1988) (Special Issue on product inspection), Richards ing potentially short cycle times. While many optical phe J., and Casasent, D., “Real Time Hough Transform for nomena may be modeled using electronic computers, appro Industrial Inspection” Proc. SPIE Technical Symposium, priate applications for optical computing, such as pattern Boston 1989 1192:2-21 (1989); Maragos, P., “Tutorial recognition, hold promise for high speed in systems of Advances in Morphological Image Processing Optical acceptable complexity. Engineering 26:7:623-632 (1987); Casasent, D., and Tescher, A., Eds. “Hybrid Image and Signal Processing II’. 0293 U.S. Pat. No. 5,060.282, incorporated herein by Proc. SPIE Technical Symposium, April 1990, Orlando Fla. reference, relates to an optical pattern recognition architec 1297 (1990); Ravichandran, G. and Casasent, D., “Noise ture implementing the mean-square error correlation algo and Discrimination Performance of the MINACE Optical rithm. This method allows an optical computing function to Correlation Filter'. Proc. SPIE Technical Symposium, April perform pattern recognition functions. U.S. Pat. No. 5,060, 1990, Orlando Fla., 1471 (1990); Weshsler, H. Ed., “Neural 282 cites the following references, which are relevant to Nets For Human and Machine Perception'. Academic Press, optical pattern recognition: New York (1991). 0294 Kellman, P., “Time Integrating Optical Signal Pro 0303 By employing volume holographic images, the cessing, Ph.D. Dissertation, Stanford University, 1979, pp. same types of paradigms may be applied to three dimen 51-55. Sional images. 0295) Molley, P., “Implementing the Difference-Squared 0304 Query by Image Content Error Algorithm Using Acousto-Optic Processor, SPIE, 0305 Query by image content, a phrase coined by IBM 1098:232-239, (1989). researchers, relates to a system for retrieving image data 0296 Molley, P., et al., “A High Dynamic Range stored in a database on the basis of the colors, textures, Acousto-Optic Image Correlator for Real-Time Pattern Rec morphology or objects contained within the image. There ognition”, SPIE, 938:55-65 (1988). fore, the system characterizes the stored images to generate a metadata index, which can then be searched. Unindexed 0297 Psaltis, D., “Incoherent Electro-Optic Image Cor searching is also possible. relator, Optical Engineering, 23(1):12-15 (January/Febru 0306 A number of query by image content systems are ary 1984). known, including both still and moving image systems, for 0298 Psaltis, D., “Two-Dimensional Optical Processing example from IBM (QBIC), Apple (Photobook), Belmont Using One-Dimensional Input Devices'. Proc. IEEE, Research Inc. (Steve Gallant), BrainTech Inc.; Center for 72(7):962-974 (July 1984). Intelligent Information Retrieval (Umass Amherst), Virage, US 2007/0016476 A1 Jan. 18, 2007

Inc., Informix Software, Inc. (Illustra), Islip Media, Inc., 0320 "Color Indexing, Swain et al., International J. Magnifi, Numinous Technologies, Columbia University Computer Vision, Vol. 7, No. 1, 1991, pp. 11-32. VisualSeek/WebSeek (Chang et al., John R. Smith), Monet (CWI and UVA), Visual Computing Laboratory, UC San 0321) “Content Oriented Visual Interface Using Video Diego (ImageCREP, White and Jain). See also, ISO/IEC Icons for Visual Database Systems”. Tonomura et al., Jour MPEG-7 literature. nal of Visual Languages and Computing (1990) 1, pp. 183-198. 0307 See, Jacobs, et al., “Fast Multiresolution Image 0322 “Developing Power Tools for Video Indexing and Querying, Department of Computer Science, University of Retrieval'. Zhang et al., Proc. SPIE Conference on Storage Washington, Seattle Wash. and Retrieval for Image and Video Databases, San Jose, 0308 U.S. Pat. No. 5,655,117, expressly incorporated Calif., 1994. herein by reference, relates to a method and apparatus for 0323) “Image Information Systems: Where Do We Go indexing multimedia information streams for content-based From Here?, Chang et al., IEEE Transactions on Knowl retrieval. See also: edge and Data Engineering, vol. 4, No. 5, October 1992, pp. 0309 Gong et al., “An Image Database System with 431-442. Content Capturing and Fast Image Indexing Abilities'. 0324 “Image Processing on Compressed Data for Large PROC of the International Conference on Multimedia Com Video Databases, Arman et al., Proc. First ACM Interna puting and Systems, pp. 121-130 May 19, 1994. tional Conference on Multimedia, Anaheim, Calif., 1-6 Aug. 0310 Hongjiang, et al., Digital Libraries, "AVideo Data 1993, pp. 267-272. base System for Digital Libraries', pp. 253-264, May 1994. 0325 “Image Processing on Encoded Video Sequences'. 0311 S. Abe andY. Tonomura, Systems and Computers in Arman et al., ACM Multimedia Systems Journal, to appear Japan, vol. 24, No. 7, "Scene Retrieval Method Using 1994. Temporal Condition Changes', pp. 92-101, 1993. 0326 “Impact: An Interactive Natural-Motion-Picture 0312 Salomon et al. “Using Guides to Explore Multi Dedicated Multimedia Authoring System’. Ueda et al., Proc. media Databases, PROC of the Twenty-Second Annual Human Factors in Computing Systems (CHI 91), New Hawaii International Conference on System Sciences. Vol. Orleans, La., Apr. 27-May 2, 1991, pp. 343-350. IV, 3-6 January 1989, pp. 3-12 vol. 4. Jan. 6, 1989. 0327 “MPEG: A Video Compression Standard for Mul 0313 Stevens, “Next Generation Network and Operating timedia Applications. Le Gall, Communications of the System Requirements for Continuous Time Media', in Her ACM, vol. 34, No. 4, April 1991, pp. 47-58. rtwich (Ed.), Network and Operating System Support for 0328 “News On-Demand for Multimedia Networks', Digital Audio and Vdeo, pp. 197-208, November 1991. Miller et al., ACM International Conference on Multimedia, 0314 U.S. Pat. No. 5,606,655, expressly incorporated Anaheim, Calif., 1-6, Aug. 1993, pp. 383-392. herein by reference, relates to a method for representing 0329 “Overview of the px64 kbit/s Video Coding Stan contents of a single video shot using frames. The method dard, Liou, Communications of the ACM, Vol. 34, No. 4, provides a representative frame (Rframe) for a group of April 1991, pp. 60-63. frames in a video sequence, selecting a reference frame from 0330) “Pattern Recognition by Moment Invariants”, Hu the group of frames and storing the reference frame in a computer memory. This system defines a peripheral motion et al., Proc. IRE, vol. 49, 1961, p. 1428. tracking region along an edge of the reference frame and 0331 “Pattern Recognition Experiments in the Mandala Successively tracks movement of boundary pixels in the Cosine Domain. Hsu et al., IEEE Trans. on Pattern Analysis tracking region, symbolizing any of the length of the shot and Machine Intelligence, vol. PAMI-5, No. 5, September and the presence of any caption. See, also: 1983, pp. 512-520. 0315) “A Magnifier Tool for Video Data”, Mills et al., 0332 “The JPEG Still Picture Compression Standard”, Proc. ACM Computer Human Interface (CHI), May 3-7, Wallace, Communications of the ACM, Vol. 34, No. 4, April 1992, pp. 93-98. 1991, pp. 31-44. 0333) “The Revised Fundamental Theorem of Moment 0316 “A New Family of Algorithms for Manipulating Invariants’. Reiss, IEEE Transactions on Pattern Analysis Compressed Images”, Smith et al., IEEE Computer Graph and Machine Intelligence, vol. 13, No. 8, August 1991, pp. ics and Applications, 1993. 830-834. 0317 “Anatomy of a Color Histogram, Novak et al., 0334) “VideoMAP and VideoSpacelcon: Tools for Anato Proceeding of Computer Vision and Pattern Recognition, mizing Video Content, Tonomura et al., Inter CHI’93 Champaign, Ill., June 1992, pp. 599–605. Conference Proc., Amsterdam, The Netherlands, 24-29 Apr., 0318 “Automatic Structure Visualization for Vdeo Edit 1993, pp. 131-136. ing'. Ueda et al., InterCHI’93 Conference Proc., Amster 0335) “Visual Pattern Recognition by Moment Invari dam, The Netherlands, 24-29 Apr. 1993, pp. 137-141. ants, IRE Trans. Inform. Theory, vol. 8, February 1962, pp. 0319) “Automatic Video Indexing and Full-Video Search 179-187. for Object Appearances', Nagasaka et al., Proc. IFIP TC2/ 0336 “Watch Grab–Arrange See: Thinking with WG2.6 Second Working Conference on Visual Database Motion Images via Streams and Collages, Elliott, Ph.D. Systems, North Holland, Sep. 30–Oct. 3, 1991, pp. 113-127. Thesis, MIT, February 1993. US 2007/0016476 A1 Jan. 18, 2007

0337 Gonzalez et al., Digital Image Processing. Addi 0357) “Virage Visual Information Retrieval, Virage son-Wesley, Readings, Mass., 1977. Incorporated, home page. 0338 Rosenfeld et al., Digital Picture Processing, Aca 0358) “Virage Products”, Virage Incorporated web site, demic Press, Orlando, Fla., 1982. pp. 1-2. 0339 Book entitled Discrete Cosine Transform—Algo 0359) “Visual Information Retrieval: AVirage Perspective rithms, Advantages, Applications, by Rao et al., Academic Revision 3”, Virage Inc. website 1995 pp. 1-13. Press, Inc., 1990. 0360 “Workshop Report: NSF ARPA Workshop on 0340 Fu, Sequential Methods in Pattern Recognition and Visual Information Management Systems''. Virage Incorpo Machine Learning, Academic, NY, N.Y. 1968. rated web. site, pp. 1-15. 0341 C.-C. J. Kuo (ed), “Multimedia Storage and 0361 A. D. Bimbo, et al, "3-D Visual Query Language Archiving Systems”, SPIE Proc. Vol. 2916 (Nov. 18-Nov. for Image Databases. Journal Of Visual Languages & 22, 1996). Computing, 1992, pp. 257-271. 0342 U.S. Pat. No. 5,600,775, expressly incorporated 0362 A. E. Cawkell, “Current Activities in Image Pro herein by reference, relates to a method and apparatus for cessing Part III: Indexing Image Collections”. CRITique, annotating full motion video and other indexed data struc vol. 4, No. 8, May 1992, pp. 1-11, ALSIB, London. tures. U.S. Pat. No. 5,428,774, expressly incorporated herein 0363 A. Pizano et al., “Communicating with Pictorial by reference relates to a system of updating an index file of Databases”. Human-Machine Interactive Systems, pp. frame sequences so that it indexes non-overlapping motion 61-87, Computer Science Dept, UCLA, 1991. image frame sequences. U.S. Pat. No. 5,550.965, expressly incorporated herein by reference, relates to a method and 0364 A. Yamamoto et al., “Extraction of Object Features system for operating a data processor to indeX primary data from Image and its Application to Image Retrieval, IEEE in real time with iconic table of contents. U.S. Pat. No. 9th International Conference On Pattern Recognition, 1988, 5,083,860, expressly incorporated herein by reference, 988-991. relates to a method for detecting change points in motion 0365 A. Yamamoto et al., “Image Retrieval System picture images. U.S. Pat. No. 5,179,449, expressly incorpo Based on Object Features”, IEEE Publication No. CH2518 rated herein by reference, relates to a scene boundary 9/87/0000-0132, 1987, pp. 132-134. detecting apparatus. See also: 0366 A. Yamamoto et al., “Extraction of Object Features 0343 “A show and tell of the QBIC technology Query and Its Application to Image Retrieval'. Trans. of IEICE, By Image Content (QBIC), IBM QBIC Almaden web site, vol. E72, No. 6, 771-781 (June 1989). pp. 1-4. 0367 A. Yamamoto et al., “Extraction of Object Features 0344) “Chaos & Non-Linear Models in Economics'. from Image and Its Application to Image Retrieval. Proc. 0345 "Chaos Theory in the Financial Markets. Applying 9th nual Conference on Pattern Recognition, Vol. 11, pp. Fractals, Fuzzy Logic, Genetic Algorithms’. 988-991 (November 1988). 0368 A. Soffer and H. Samet. Retrieveal by content in 0346). “Evolutionary Economics & Chaos Theory”. symbolic-image databases. In Symposium on Electronic 0347 “Four Eyes', MIT Media Lab web site, pp. 1-2. Imaging: Science and Technology—Storage & Retrieval for Image and Video Databases IV, pages 144-155. IS&T/SPIE, 0348 “Frequently asked questions about visual informa 1996. tion retrieval'', Virage Inc. web site, pp. 1-3. 0369 Abadi, M., et al., “Authentication and Delegation 0349) “IBM Ultimedia Manager 1.1 and Clinet Search”, with Smart-cards', Oct. 22, 1990, revised Jul. 30, 1992 IBM software web site, pp. 1-4. Report 67, Systems Research Center, Digital Equipment 0350 “Image Detection and Registration”. Digital Image Corp., Palo Alto, Calif. Processing, Pratt, Wiley, N.Y., 1991. 0370 Advertisement for “TV Decision.” CableVision, 0351) “Jacob Methodology” WWCSAI.diepa.unipa.it/re Aug. 4, 1986. search/projects/acob/acob-method.html. 0371 ANSI, “Financial Institution Retail Message 0352 “Market Analysis. Applying Chaos Theory to Authentication, ANSI X9.19 1986. Invstment & Economics'. 0372 American National Standard, “Interchange Mes sage Specification for Debit and Credit Card Message 0353 “Photobook', MIT Media Lab web site, Aug. 7, Exchange Among Financial Institutions”. ANSI X9.2-1988. 1996: pp. 1-2. 0373) Antonofs, M., “Stay Tuned for Smart V.” Popular 0354) “Profiting from Chaos. Using Chaos Theory for Science, November 1990, pp. 62-65. Market Timing, Stock Selection & Option'. 0374. Arman et al., “Feature Management for Large 0355 "Shape Analysis, Digital Image Processing, Pratt, Vdeo Databases”, 1993. (Abstract Only). Wiley, N.Y., 1991. 0375 ASIAN TECHNOLOGY INFORMATION PRO 0356) “The QBIC Project”, IBM QBIC Almaden web GRAM (ATIP) REPORT: ATIP95.65: Human Computer site, home page (pp. 1-2). Interface International, July 1995 Yokohama. US 2007/0016476 A1 Jan. 18, 2007

0376 Barber et al. “Ultimedia Manager: Query by Image 0395 Chang et al., “Intelligent Database Retrieval by Content and it’s Applications' IEE, 1994, pp. 424-429, Janu Visual Reasoning, PROC Fourteenth Annual International ary 1994. Computer Software and Application Conference, 31 Oct.-1 0377 Barros, et al. “Indexing Multispectral Images for Nov. 1990, pp. 459-464. Content-Based Retrieval, Proc. 23rd APR WS on Image 0396 Chang, Yuh-Lin, Zeng, Wenjun, Kamel, Ibrahim, and Information Retrieval, Proc. 23rd Workshop, Washing Alonso, Rafael, “Integrated Image and Speech Analysis for ton, D.C., October 1994, pp. 25-36. Content-Based Video Indexing. 0378 Belkin, N.J., Croft, W. B., “Information Filtering 0397) Chaum et al., “Untraceable Electronic Cash', And Information Retrieval: Two Sides of the Same Coin'?”, Advances in Cryptology, 1988, pp. 319-327. Communications of the ACM, December 1992, vol. 35, No. 0398 Chaum et al., “Achieving Electronic Privacy”, Sci 12, pp. 29-38. entific American, pp. 319-327, 1988. 0379 Benoit Mandelbrot: “Fractal Geometry of Nature', 0399 Chaum, D. “Security without Identification: Card WH Freeman and Co., New York, 1983 (origed 1977). Computers to Make Big Brother Obsolete'. Communica 0380 Benoit Mandelbrot: “Fractals. Form, Chance and tions of the ACM, 28(10), October 1985, pp. 1030-1044. Dimensions', W H Freeman and Co., San Francisco, 1977. 04.00 Chaum, D. “Untraceable Electronic Mail, Retur 0381 Bimbo et al., “Sequence Retrieval by Contents nAddresses, and Digital Pseudonyms. Communications of through Spatio Temporal Indexing, IEEE on CD-ROM, pp. the ACM, vol. 24, No. 2, February 1981. 88-92, Aug. 24, 1993. 04.01 Chaum, D., “Achieving Electronic Privacy”. Sci 0382 Bolot, J.; Turletti, T. & Wakeman, I.: “Scalable entificamerican, August 1992, pp. 96-101. Feedback Control for Multicast Video Distribution. In the 0402 Chaum, D. L. et al.: “Implementing Capability Internet’. Computer Communication Review, vol. 24, No. 4 Based Protection Using Encryption’: Electronics Research October 1994, Proc. SIGCOMM 94, pp. 58-67. Laboratory, College of Eng., University of California, Ber 0383 Bos et al., “SmartCash: a Practical Electronic Pay keley, Calif.; Jul. 17, 1978. ment System, pp. 1-8; August 1990. 0403. Cliff Pickover, Spiral Symmetry (World Scientific). 0384 Branko Pecar: “Business Forecasting for Manage 04.04 Cliff Pickover, Chaos in Wonderland: Visual ment, McGraw-Hill Book Co., London, 1994. Adventures in a Fractal World (St. Martin’s Press). 0385 Brian H Kaye. “A Random Walk Through Fractal 04.05 Cliff Pickover, Computers and the Imagination (St. Dimensions. VCH Verlagsgesellschaft, Weinheim, 1989. Martin's Press) Cliff Pickover, Mazes for the Mind: Com 0386 Brugliera, Vito, “Digital On-Screen Display—A puters and the Unexpected (St. Martin’s Press). New Technology for the Consumer Interface'. Symposium 0406 Cliff Pickover, Computers, Pattern, Chaos, and Record Cable Sessions. Jun. 11, 1993, pp. 571-586. Beauty (St. Martin's Press). 0387 Burk et al., “Value Exchange Systems Enabling 04.07 Cliff Pickover, Frontiers of Scientific Visualization Security and Unobservability”. Computers & Security, 9 (Wiley). 1990, pp. 715-721. 0408 Cliff Pickover, Visions of the Future: Art, Technol 0388 C. Chang et al., “Retrieval of Similar Pictures on ogy, and Computing in the 21st Century (St. Martin’s Press). Pictorial Databases, Pattern Recognition, vol. 24, No. 7, 1991, pp. 675-680. 04.09 Cohen, Danny, “Computerized Commerce', ISI Reprint Series ISI/RS-89/243; October, 1989; Reprinted 0389 C. Chang, “Retrieving the Most Similar Symbolic from Information Processing 89, Proc. IFIP World Computer Pictures from Pictorial Databases, Information Processing Cong. Aug. 28-Sep. 1, 1989. & Management, vol. 28, No. 5, 1992. 0410 Cohen, Danny: “Electronic Commerce', Univer 0390 C. Faloutsos et al. “Efficient and Effective Query sity of Southern California, Information Sciences Institute, ing by Image Content, Journal of Intelligent Information Research Report ISI/RR-89-244; October, 1989. Systems:Integrating Artificial Intelligence and Database 0411 Common European Newsletter, Multimedia Con Technologies, vol. 3-4, No. 3, July 1994, pp. 231-262. tent manipulation and Management, ww.esat.kuleuven 0391) C. Goble etal, “The Manchester Multimedia Infor .ac.be/-konijin? . . . . mation System”. Proc. IEEE Conference, Eurographics 0412 CompuServe Information Service Users Guide, Workshop, April 1991, pp. 244-268. CompuServe International, 1986, pp. 109-114. 0392 C. C. Chang and S. Y. Lee. Retrieval of similar 0413 Computer Shopper, November 1994, “Internet for pictures on pictorial databases. Pattern Recog. 24(7), 1991. Profit', pp. 180-182, 187, 190-192, 522-528, 532, 534. 0393 Case Study: The CIRRUS Banking Network, Comm. ACM 8, 28 pp. 7970-8078, August 1985. 0414 Computer, Vol. 28(9), September 1995. 0394 Chalmers, M., Chitson, P., “Bead: Explorations In 0415 Compuvid Sales Manual (date unknown). Information Visualization, 15th Ann. Int’l SIGIR 92/Den 0416) Corporate Overview, Virage Incorporated web site: mark June 1992, pp. 330-337. pp. 1-4. US 2007/0016476 A1 Jan. 18, 2007

0417 Cox, Ingemar J., et al., “Pichunter: Bayesian Rel 0437 G F Page, J B Gomm & D Williams: “Application evance Feedback for Image Retrieval.” Proc. ICPR 96, of Neural Networks to Modelling and Control, Chapman & IEEE, pp. 361-369. Hall, London, 1993. 0418 Cutting, D. R.: Karger, D. R.; Pedersen, J. O. & 0438 G. Mannes, “Smart Screens”. Video Magazine, Tukey, J. W. "Scatter/Gather: A Cluster-based Approach to December 1993) (2 Pages). Browsing Large Document Collections”, 15 Ann. Int’l 0439 G. Tortora et al., “Pyramidal Algorithms”. Com SIGIR 92, ACM, 1992, pp. 318-329. puter Vision, Graphics and Images Processing, 1990, pp. 0419 DK Arrowsmith & C M Place: An Introduction to 26-56. Dynamical Systems’. Cambridge University Press, Cam 0440 Gautama, S., D'Haeyer, J. P. F., “Context Driven bridge, 1990. Matching in Structural Pattern Recognition'. 0420 Damashek, M., Gauging Similarity via N-Grams: 0441 Gautama, S., Haeyer, J. D., “Learning Relational Language-Independent Sorting, Categorization, and Models of Shape: A Study of the Hypergraph Formalism’. Retrieval of Text, pp. 1-11, Jan. 24, 1995. 0442 Gene F Franklin, J David Powell & Abbas Emami 0421) Data Partner 1.0 Simplifies DB Query Routines, Naeini: "Feedback Control of Dynamic Systems’. Addison PC Week, Sep. 14, 1992, pp. 55 & 58. Wesley Publishing Co. Reading, 1994. 0422 David E. Rumelhart & James L. McClelland: “Par 0443) George E P Box & Gwilym M Jenkins: “Time allel Distributed Processing, Vol 1. The MIT Press, Cam Series Analysis: Forecasting and Control, Holden Day, San bridge, Mass., 1986. Francisco, 1976. 0423 Deering, S.; Estrin, D.; Farinacci, D.; Jacobson, V.: 0444 Gifford, D., “Notes on Community Information Liu, C.; Wei, L., “An Architecture for Wide-Area Multicast Systems', MIT LCSTM-419, December 1989. Routing, Computer Communication Review, vol. 24, No. 4, October 1994, Proc. SIGCOMM 94, pp. 126-135. 0445 Gifford, David K.: “Cryptographic Sealing for Information Secrecy and Authentication', Stanford Univer 0424 Donal Daly: “Expert Systems Introduced', sity and Xerox Palo Alto Research Center; Communication Chartwell-Bratt, Lund, 1988. of the ACM; vol. 25, No. 4; April, 1982. 0425 Dukach, Seymon; Prototype Implementation of the 0446 Gifford, David K.: “Digital Active Advertising: SNPP Protocol; allspic.lcs.mit.edu: 1992. U.S. patent application Ser. No. 08/168,519; filed Dec. 16, 0426 E. Binaghi et al., “Indexing and Fuzzy Logic Based 1993. Retrieval of Color Images”. Visual Database Systems, II, 0447 Gligor, Virgil D. et al.: “Object Migration and 1992, pp. 79-92. Authentication', IEEE Transactions on Software Engineer 0427 E. Binaghi et al., “A Knowledge-Based Environ ing, vol. SE-5, No. 6; November, 1979. ment for Assessment of Color Similarity”. Proc. 2nd Annual 0448 Gong et al. “An Image Database System with Conference on Topics for A1, pp. 268-285 (1990). Content Capturing and Fast Image Indexing Abilities’ IEEE, 0428 E. Lee, “Similarity Retrieval Techniques”, Pictorial 1994, pp. 121-130, May 1994. Information Systems, Springer Verlag, 1980 pp. 128-176. 0449 Gregory L. Baker & Jerry P Golub: “Chaotic Dynamics: An Introduction’, Cambridge University Press, 0429 E. G. M. Petrakis and C. Faloutsos. Similarity Cambridge, 1990. searching in large image databases. Technical Report 3388, Department of Computer Science, University of Maryland, 0450 Gupta, Amarnath; Weymount, Terry & Jain, 1995. Ramesh, “Semantic Queries With Pictures: The VIMSYS Model. Proc. 17th International Conference on Very Large 0430. Edward Reitman: “Exploring the Geometry of Data Bases, pp. 69-79, Barcelona, September, 1991. Nature”, Windcrest Books, Blue Ridge Summit, 1989. 0451 H. Tamura et al., “Image Database Systems: A 0431) Even et al: “Electronic Wallet'; pp. 383-386; 1983. Survey’. Pattern Recognition, vol. 17, No. 1, 1984, pp. 0432) F. J. Varela and P. Bourgine (eds.): Proc. first 29-34. European Conference on Artificial Life. Cambridge, Mass.: 0452 H. Tamura, et al., “Textural Features Correspond MIT Press. (1991). ing to Visual Perception.” IEEE Transactions on System, 0433) Fassihi, Theresa & Bishop, Nancy, “Cable Guide Man, and Cyb., vol. SMC-8, No. 6, pp. 460-473 (1978). Courting National Advertisers.” Adweek, Aug. 8, 1988. 0453 H. Samet. The quadtree and related hierarchical 0434 Flickner, et al. “Ouery by Image and Vdeo Content, data structures. ACM Computing Surveys, 16(2):187-260, the QBIC System”, IEEE Computer 28(9), 23-32, 1995. 1984. 0435 Foltz, P. W., Dumais, S. T., “Personalized Infor 0454 Hans Lauwerier: "Fractals—Images of Chaos”, mation Delivery: An Analysis Of Information Filtering Penguin Books, London, 1991. Methods’. Communications of the ACM, December 1992, 0455 Harty et al., “Case Study: The VISA Transaction vol. 35, No. 12, pp. 51-60. Processing System.” 1988. 0436 Frank Pettit: “Fourier Transforms in Action”, 0456 Heinz-Otto Peitgen & Deitmar Saupe: “The Sci Chartwell-Bratt, Lund, 1985. ence of Fractal Images'. Springer-Verlag, N.Y., 1988. US 2007/0016476 A1 Jan. 18, 2007

0457 Heinz-Otto Peitgen, Hartmut Jurgens & Deitmar 0477 J. Hasegawa et al., “Intelligent Retrieval of Chest Saupe: "Fractals for the Classroom'. Springer-Verlag, 1992. X-Ray Image Database Using Sketches’, System And Com 0458 Hirata, et al. "Query by Visual Example, Content puters. In Japan, 1989, pp. 29-42. Based Image Retrieval Advance in Database Technology 0478 J. M. Chassery, et al., “An Interactive Segmenta EDBT '92, Springer-Verlag, Berlin 1992, pp. 56-71 tion Method Based on Contextual Color and Shape Crite rion', IEEE Transactions on Pattern Analysis and Machine 0459 Hirzalla et al., “A Multimedia Query User Inter Intelligence, vol. PAMI-6, No. 6, (November 1984). face", IEEE (CDROM) pp. 590-593, Sep. 5, 1995. 0479. J. Wachman, “AVideo Browser that Learns by 0460 Hooge, Charles, "Fuzzy logic Extends Pattern Rec Example, Masters Thesis, Massachusetts Institute of Tech ognition Beyond Neural Networks”. Vision Systems Design, nology: 1996; also appears as MIT Media Laboratory Tech January 1998, pp. 32-37. nical Report No. 383. 0461 Hou et al., “Medical Image Retrieval by Spatial 0480 J. Hafner, H. S. Sawhney, W. Equitz, M. Flickner, Features”, IEEE (CD-ROM) pp. 1364-1369, Oct. 18, 1992. and W. Niblack. Efficient color histogram indexing for 0462 Iino et al., “An Object-Oriented Model for Spatio quadratic form distance functions. IEEE Trans. Pattern Anal. Temporal Synchronization of Multimedia Information'. Machine Intell., July 1995. May, 1994. 0481 J. R. Bach, C. Fuller, A. Gupta, A. Hampapur, B. 0463 Information Network Institute, Carnegie Mellon Horowitz, R. Humphrey, R. C. Jain, and C. Shu. Virage University, Internet Billing Server, Prototype Scope Docu image search engine: an open framework for image man agement. In Symposium on Electronic Imaging: Science and ment, Oct. 14, 1993. Technology Storage & Retrieval for Image and Video 0464 Ingemar J. Cox et al., “Target Testing and the Pic Databases IV, pages 76-87. IS&T/SPIE, 1996. Hunter Bayesian Multimedia Retrieval System.” Proc. 3d Forum on Research and Technology Advances in Digital 0482 J. R. Smith and S.-F. Chang. Querying by color regions using the VisualSEEk content-based visual query Libraries, ADL '96, IEEE, pp. 66-75. system. In M. T. Maybury, editor, Intelligent Multimedia Inf. 0465 Intel Corporation, iPower Technology, Marketing Retrieval. IJCAI, 1996. Brochure, date unknown. 0483 J. R. Smith and S.-F. Chang. Tools and techniques 0466 Intuit Corp. Quicken User's Guide, “Paying Bills for color image retrieval. In Symposium on Electronic Electronically', pp. 171-192; undated. Imaging: Science and Technology—Storage & Retrieval for 0467) ISO/IEC JTC1/SC29/WG 11 N1735, MPEG97, Image and Video Databases IV. volume 2670, San Jose, July 1997 Stockholm, “MPEG-7 Applications Docu Calif., February 1996. IS&T/SPIE. ment. 0484 Jacobs, Charles E. Finkelstein, Adam, Salesin, 0468 ISO/IEC JTC1/SC29/WG 11 N2460, MPEG98, David H., “Fast Multiresolution Image Querying. October 1998 “MPEG-7 Context and Objectives (v. 10– 0485 James Gleick: “Chaos—Making a New Science'. Atlantic City), ISO/IEC JTC1/SC29/WG 11 N1920, Heinemann, London, 1988. MPEG97, October 1997 “MPEG-7 Context and Objectives 0486 Jane Hunter, “The Application of Metadata Stan (v.5 Fribourg), ISO/IEC JTC1/SC29/WG 11 N1733, dards to Video Indexing www.dtsc.edu.au/RDU/staff.jane MPEG97, July 1997, “MPEG-7 Context and Objectives (V.4 Stockholm)'. hunter/EuroDL/final. html (

0494 K. Wakimoto et al., “An Intelligent User Interface 0512 M. Ioka, “A Method of Defining the Similarity of to an Image Database using a Figure interpretation Method”. Images on the Basis of Color Information', Bulletin Of The IEEE Publication No. CH2898-5/90/0000/0516, 1990, pp. National Museum Of Ethnology Special Issue, pp. 229-244. 516-52O. No. 17, November 1992. 0495 K. Woolsey, “Multimedia Scouting, IEEE Comp. 0513. M. Kurokawa, “An Approach to Retrieving Images Graphics & Applications, July 1991 pp. 26-38. by Using their Pictorial Features’, IBM Research, Japan, 0496 Kelly et al. “Efficiency Issues Related to Probabil September 1989. ity Density Function Comparison', SPIE vol. 2670, pp. 0514 M. Swain et al., “Color Indexing, International 42-49 January 1996. Journal Of Computer Vision, 1991, pp. 12-32. 0497 Kelly, P.M., et al. “Candid Comparison Algorithm 0515 M. Stricker and A. Dimai. Color indexing with for Navigating Digital Image Databases”. Proc. 7th Int. weak spatial constraints. In Symposium on Electronic Imag Working Conference on Scientific and Statistical Database ing: Science and Technology—Storage & Retrieval for Management, pp. 252-258, 1994. Image and Video Databases IV, pages 29-41. IS&T/SPIE, 0498 Krajewski, M. et al., “Applicability of Smart Cards 1996. to Network User Authentication’, Computing Systems, vol. 0516 M. Stricker and M. Orengo. Similarity of color 7, No. 1, 1994. images. In Storage and Retrieval for Image and Video 0499 Krajewski, M., “Concept for a Smart Card Ker Databases III, volume SPIE Vol. 2420, February 1995. beros”, 15th Nat. Comp. Security Conf., October 1992. 0517 Mackay et al., Virtual Video Editing in Interactive 0500 Krajewski, M., “Smart Card Augmentation of Multimedia Applications”, 1989. rberos, Privacy and Security Research Group Workshop on 0518 Manners, George, “Smart Screens; Development Network and Distributed System Security”, February 1993. of Personal Navigation Systems for TV Viewers. Video 0501 Lampson, Butler; Abadi, Martin, Burrows, Magazine, December 1993. Michael; and Wobber, Edward; “Authentication in Distrib uted Systems: Theory and Practice'. ACM Transactions on 0519 Martin Casdagli & Stephen Eubank: “Nonlinear Computer Systems; Vol. 10, No. 4; November 1992: pp. Modelling and Forecasting”. Addison-Wesley Publishing 265-31 O. Co., Redwood City, 1992. 0502 Landis, Sean, “Content-Based Image Retrieval 0520 Martinez et al. “Imagenet: A Global Distribution Systems for Interior Design, www.tc.cornell.edu/Visualiza Database for Color Image Storage and Retrieval in Medical tion/Education/cs718/fall 1995/landis/index.html. Imaging Systems’” IEEE, 1992, 710-719, May 1992. 0503 Langton C G (ed): Artificial Life, Proc. first inter 0521 Marvin A. Sirbu: Internet Billing Service Design national conference on Artificial life, Redwood City: Addi And Prototype Implementation; pp. 1-19: An Internet Bill son-Wessley (1989). ing Server. 0504 Lee et al., “Video Indexing Approach based on 0522 Masahiro Morita & Yoichi Shinoda, Information Moving Object and Track’. Proc. Storage and Retrieval for Filtering Based on User Behavior Analysis and Best Match Image and Video Databases, pp. 25-36. February 1993. Text Retrieval, Proc. Seventeenth Annual International ACM-SIGIR Conference on Research and Development in 0505 Lee, Denis, et al., “Query by Image Content Using Information Retrieval, Dublin, Jul. 3-6, 1994, Pages Title Multiple Objects and Multiple Features: User Interface Page (272)-281. Issues.” 1994 Int’l Conf. on Image Processing, IEEE, pp. 76-80. 0523 Medvinsy et al., “NetCash: A Design for Practical 0506 Lennart Ljung & Torsten Soderstrom: “Theory and Electronic Currency on the Internet”. Proc. 1st ACM Conf. Practice of Recursive Identification'. The MIT Press, Cam on Comp. and Comm. Security, November 1993. bridge, Mass., 1983. 0524 Medvinsy et al., “Electronic Currency for the Inter 0507 Lennart Ljung: "System Identification, Theory for net”, Electronic Markets, pp. 30-31, September 1993. the User', Prentice-Hall Englewood Cliffs, N.J., 1987. 0525) Meyer, J. A., Roitblat, H. L., Wilson, W. (eds.): 0508 Loeb, S., “Architecting Personalized Delivery of From Animals to Animats. Proc. Second International Con Multimedia Information’, Communications of the ACM, ference on Simulation of Adaptive Behaviour. Cambridge, December 1992, vol. 35, No. 12, pp. 39-50. Mass: MIT Press. (1991). 0509 MV Berry, I C Persival & N O Weiss: “Dynamical 0526 Middleton, G. V. ed., 1991, Nonlinear Dynamics, Chaos. The Royal Society, London, 1987, Proc. a Royal Chaos and Fractals, with Applications to Geological Sys Society Discussion Meeting held on 4 & 5 Feb. 1987. tems. Geol. Assoc. Canada Short Course Notes Vol. 9 (available from the GAC at Memorial University of New 0510) M. Bender, “EFTS: Electronic Funds Transfer Sys foundland, St. John's NF A1B 3X5). tems’, Kennikat Press, Port Washington, N.Y., pp. 43-46 1975. 0527 Mills, “Media Composition for Casual Users”, 0511 M. H. O'Docherty et al., “Multimedia Information 1992. System. The Management and Semantic Retrieval of all 0528 Minneman et al., “Where Were We: making and Electronic Data Types”. The Computer Journal, Vol. 34, No. using near-synchronous, pre-narrative video'. Multimedia 3, 1991. 93, pp. 1-11. December 1993. US 2007/0016476 A1 Jan. 18, 2007 20

0529 N. Hutheesing, “Interactivity for the passive”, 0548 Phillips, “MediaView: a general multimedia digital Forbes magazine Dec. 6, 1993, 2pp. publication system”. Comm. of the ACM. V. 34, n. 7, pp. 0530 N. S. Chang et al., “Query-by-Pictorial Example, 75-83. July 1991. IEEE Transactions on Software Engineering, vol. SE-6, No. 0549. Predrag Cvitanovic: “Universality in Chaos”, 6, pp. 519-524 (November 1980). Adam Hilger, Bristol, 1989. 0531 N. S. Chang, et al., “Picture Query Languages for 0550 R. Mehrotra et al. “Shape Matching Utilizing Pictorial Data-Base Systems”. Computer vol. 14, No. 11, pp. Indexed Hypotheses Generation and Testing, IEEE Trans 23-33 (November 1981). actions On Robotics, vol. 5, No. 1, February 1989, pp. 0532 Nagasaka et al., Automatic Video Indexing and 70-77. Full-Video Search for Object Appearances”. Visual Data 0551 R. Price, et al., “Applying Relevance Feedback to base Systems, (Knuth et al., eds.), pp. 113-126. January a Photo Archival System. Journal of Information Science 1992. 18, pp. 203-215 (1992). 0533 National Westminster Bank Group Brochure: pp. 0552) R. W. Picard et al., “finding Similar Patterns in 1-29; undated. Large Image Databases, IEEE ICASSP. Minneapolis, 0534 Needham, Roger M. and Schroeder, Michael D.: Minn., vol. V. pp. 161-164, April 1993; (MIT Media Lab. “Using Encryption for Authentication in Large Networks of Technical Report No. 205). Computers’, Communications of the ACM; vol. 21, No. 12, 0553 Rangan et al., “A Window-based Editor for Digital December 1978; pp. 993-999. Video and Audio', January 1992. 0535 Needham, Roger M.: “Adding Capability Access to 0554 Richards et al., “The Interactive Island, IEE Rev Conventional File Servers', Xerox Palo Alto Research Cen ies, July/August 1991 pp. 259-263. ter, Palo Alto, Calif. 0555 Rivest, R.; “The MD5 Message-Digest Algo 0536 Newman, B. C., “Proxy-Based Authorization and rithm’: MIT Laboratory for Computer Science and RSA Accounting for Distributed Systems'. Proc. 13th Int. Conf. Data Security, Inc.; April, 1992. on Dist. Comp. Sys. May 1993. 0556 Rivest, R. L. et al., “A Method for Obtaining 0537 Niblack, W. et al., “The QBIC Project: Querying Digital Signatures and Public-Key Cryptosystems.” Labo Images by Content Using Color, Texture, and Shape'. IBM ratory for Computer Science, Massachusetts Institute of Computer Science Research Report, pp. 1-20 (Feb. 1, 1993). Technology, Cambridge, Mass. 0538 Nussbaumer et al., “Multimedia Delivery on 0557. Rivest, R. L.; Shamir.A. & Adleman, L.: “A Method Demand: Capacity Analysis and Implications”. Proc 19th for Obtaining Digital Signatures and Public-Key Cryptosys Conference on Local Computer Networks, 2-5 Oct. 1994, tems”. Communications of the ACM, February 1978, vol. pp. 380-386. 21, No. 2, pp. 120-126. 0539 O. Guenther and A. Buchmann. Research issues in 0558 Robert Brown: “Statistical Forecasting for Inven spatial databases. In ACM SIGMOD Record, volume 19, tory Control”, McGraw-Hill, N.Y., 1958. December 1990. 0559 Robinson, G., and Loveless, W., “Touch-Tone 0540 Okamoto et al; “Universal Electronic Cash', pp. Teletext. A Combined Teletext-Viewdata System.” IEEE 324-337; 1991. Transactions on Consumer Electronics, Vol. CE-25, No. 3, July 1979, pp. 298-303. 0541. Ono, Atsushi, et al., “A Flexible Content-Based Image Retrieval System with Combined Scene Description 0560 Roizen, Joseph, “Teletext in the USA,” SMPTE Keyword,” Proc. Multimedia '96, IEEE, pp. 201-208. Journal, July 1981, pp. 602–610. 0542. Otis Port, “Wonder Chips-How They’ll Make 0561 Rose, D. E.; Mander, R.; Oren, T., Ponceleon, D. Computing Power Ultrafast and Ultracheap', Business B.; Salomon, G. & Wong, Y.Y. "Content Awareness in a File Week, Jul. 4, 1994, pp. 86-92. System Interface Implementing the Pile Metaphor for Organizing Information, 16 Ann. Int’l SIGIR 93, ACM, 0543 P G Drazin: “Nonlinear System”, Cambridge Uni pp. 260-269. versity Press, Cambridge, 1992. 0562 Ross Anderson, “Why Cryptosystems Fail', Proc. 0544 P. Stanchev et al., “An Approach to Image Indexing 1st Conf. Computer and Comm. Security, pp. 215-227, of Documents”. Visual Database Systems, II, 1992, pp. November 1993. 63-77. 0563 Ross J. Anderson, “UEPS A Second Generation 0545) Peter J Diggle: “Time Series: A Biostatistical Intro Electronic Wallet'. Proc. Second European Symposium on duction’, Clarendon Press, Oxford, 1990. Research in Computer Security (ESORICS), Touluse, 0546 Peters: “Chaos and Order in the Capital Markets', France, pp. 411-418. Wiley, 1991 Gershenfeld & Weigend: “The Future of Time 0564 Rui, Yong, Huang, Thomas S., Chang, Shih-Fu, Series”. Addison-Wesley, 1993. “Image Retrieval: Past Present and Future'. 0547 Pfitzmann et al: “How to Break and Repair a 0565 Rui, Yong, Huang, Thomas S., Mehotra, Sharad, Provably Secure Untraceable Payment System'; pp. 338 “Browsing and retrieving Video Content in a Unified Frame 350; 1991. work. US 2007/0016476 A1 Jan. 18, 2007

0566 Rui, Yong, Huang, Thomas S., Ortega, Michael, 0584) Sheth et al., “Evolving Agents for Personalized Mehotra, Sharad, “Relevance Feedback: A Power Tool for Information Filtering, 1-5 Mar. 1993, (345-352). Interactive Content-Based Image Retrieval'. 0585 Sheth, B. & Maes, P. “Evolving Agents For Per 0567 S. Chang et al., “An Intelligent Image Database sonalized Information Filtering”. Proc. 9th IEEE Confer System”, IEEE Transactions On Software Engineering, vol. ence, 1993 pp. 345-352. 14, No. 5, May 1988, pp. 681-688. 0586) Sincoskie, W. D. & Cotton C. J. “Extended Bridge 0568 S. Chang et al., “Iconic Indexing by 2-D Strings', Algorithms for Large Networks', IEEE Network, January IEEE Transactions On Patternalysis And Machine Intelli 1988-vol. 2, No. 1, pp. 16-24. gence, vol. PAMI-9, No. 3, May 1987. 0587 Smith, J. et al., “Quad-Tree Segmentation for Tex 0569. S. Chang et al., “Iconic Indexing by 2-D Strings', ture-Based Image Query' Proceeding ACM Multimedia 94. IEEE Transactions On Patternalysis And Machine Intelli pp. 1-15, San Francisco, 1994. gence, vol. 9, No. 3, May 1987, pp. 413-427. 0570 S. Charles et al. “Using Depictive Queries to 0588 Smoliar, S. et al., “Content-Based Video Indexing Search Pictorial Databases”. Human Computer Interaction, and Retrieval, IEEE Multimedia, pp. 62-72 (Summer 1990, pp. 493-498. 1994). 0571 S. Lee et al., “2D C-string: A New Spatial Knowl 0589 Society for Worldwide Interbank Financial Tele edge Representation for Image Database Systems’. Pattern communications S.C., “A.S.W.I.F.T. Overview'. Recognition, vol. 23, 1990, pp. 1077-1087. 0590 S. Makridakis & S. Wheelwright: “The Handbook 0572 S. Lee et al., “Similarity Retrieval of Iconic Image of Forecasting”, John Wiley, N.Y. 1982. Database'. Pattern Recognition, vol. 22, No. 6 1989, pp. 0591 S. C. Chapra & R. P. Canale: Numerical Methods 675-682. for Engineers”, McGraw-Hill, N.Y. 1988. 0573 S. Lee et al., “Spatial Reasoning and Similarity 0592 T. Arndt, “A Survey of Recent Research in Image Retrieval of Images. Using 2D C-string Knowledge Repre Database Management', IEEE Publication No. THO330-1/ sentation’, Pattern Recognition, 1992, pp. 305-318. 90/0000/0092, pp. 92-97, 1990. 0574. S. Negahdaripour et al "Challenges in Computer Vision: Future Research Direction', IEEE Transactions On 0593. T. Gevers et al., “Enigma: An Image Retrieval Systems, Man And Cybernetics, pp. 189-199, 1992, at Con System”, IEEE 11th IAPR International Conference On ference on Computer Vision and Pattern Recognition. Pattern Recognition, 1992, pp. 697-700. 0575 S. Tanaka et al., “Retrieval Method for an Image 0594 T. Gevers et al., “Indexing of Images by Pictorial Database based on Topological Structure'. SPIE, Vol. 1153, Information'. Visual Database Systems, II, 1992 IFIP. pp. 1989, pp. 318-327. 93-101. 0576 S.-F. Chang. Compressed-domain techniques for 0595) T. Kato et al., “A Cognitive Approach Interaction'. image/video indexing and manipulation. In Proc. I.E.E.E. International Conference Of Multimedia Information Sys International Conference on Image Processing, Washington, tems, January, 1991, pp. 109-119. D.C., October 1995. invited paper to the special session on 0596) T. Kato et al., “Trademark: Multimedia Database Digital Library and Video on Demand. with Abstracted Representation on Knowledge Base, Proc. 0577 S-K Chang, Q. Y. Shi, and C. Y. Yan. Iconic 2" Int. Symp. On Interoperable Information Systems, pp. indexing by 2-D strings. IEEE Trans. Pattern Anal. Machine 245-252, November 1988. Intell, 9(3):413-428, May 1987. 0597 T. Kato et al., “Trademark: Multimedia Image Data 0578 S-K Chang. Principles of Pictorial Information base System with Intelligent Human Interface'. System And Systems Design. Prentice Hall, 1989. Computers. In Japan, 1990, pp. 33-46. 0579 Salton, G., “Developments in Automatic Text 0598. T. Kato, “A Sketch Retrieval Method for Full Color Retrieval, Science, 253:974-980, Aug. 30, 1991. Image Database-Query by Visual Example, IEEE, Publi 0580 Schamuller-Bichl, I., “IC-Cards in High-Security cation No. 0-8186-2910-X/92, 1992, pp. 530-533. Applications', in Selected Papers from the Smart Card 2000 0599 T. Kato, “Intelligent Visual Interaction with Image Conference, Springer Verlag, 1991, pp. 177-199. Database Systems Toward the Multimedia Personal Inter 0581) Semyon Dukach, “SNPP: A Simple Network Pay face'. Journal Of Information Processing, vol. 14, No. 2, ment Protocol', MIT Laboratory for Computer Science, 1991, pp. 134-143. Cambridge, Mass., 1993. 0600 T. Minka, “An Image Database Browser that 0582 Shann et al. “Detection of Circular Arcs for Con Learns from User Interaction', Masters Thesis, Massachu tent-Based Retrieval from an Image Database IEE Proc.- setts Institute of Technology: 1996 (MIT Media Lab. Tech Vis. Image Signal Process, vol. 141, No. 1 February 1994, nical Report 365.) pp. 49-55. 0601 T.-S. Chua, S.-K Lim, and H.-K Pung. Content 0583. Sheldon G. Lloyd & Gerald DAnderson: “Industrial based retrieval of segmented images. In Proc. ACM Intern. Process Control, Fisher Controls Co., Marshalltown, 1971. Conf. Multimedia, October 1994. US 2007/0016476 A1 Jan. 18, 2007 22

0602 Tak W. Yan & Hector Garcia-Molina, SIFT A 0620. William L. Thomas, “Electronic Program Guide Tool for Wide-Area Information Dissemination, 1995 Applications-The Basics of System Design”, 1994 NCTA USENIX Technical Conference, New Orleans, La., January Technical Papers, pp. 15-20. 16-20, pp. 177-186. 0621 X. Zhang, et al., “Design of a Relational Image 0603 Tanton, N. E., “UK Teletext Evolution and Database Management System: IMDAT, IEEE Publication Potential, IEEE Transactions on Consumer Electronics, No. THO166-9/87/0000-0310, 1987, pp. 310-314. vol. CE-25, No. 3, July 1979, pp. 246-250. 0622 Y. Okada, et al., “An Image Storage and Retrieval 0604 Tenenbaum, Jay M. and Schiffman, Allan M.: System for Textile Pattern Adaptable to Color Sensation of “Development of Network Infrastructure and Services for the Individual', Trans. Inst. Elec. Inf. Comm., Vol. J70D, Rapid Acquisition', adapted from a white paper Submitted No. 12, pp. 2563-2574, December 1987 (Japanese w/English to DARPA by MCC in collaboration with EIT and ISI. Abstract). 0605 Training Computers To Note Images, New York 0623) Y. Yoshida et al., “Description of Weather Maps and Times, Apr. 15, 1992. Its Application to Implementation of Weather Map Data 0606 Turcotte, Donald L., 1992, Fractals and Chaos in base', IEEE 7th International Conference On Pattern Rec Geology and Geophysics. Cambridge U.P. ognition, 1984, pp. 730-733. 0624 Yan et al., “Index Structures for Information Fil 0607 TVCommunications Advertisement for MSI Data tering Under the Vector Space Model”, PROC the 10th casting Systems, January 1973. International Conference on Data Engineering, pp. 14-18 of 0608 V. Gudivada et al., “A Spatial Similarity Measure DRD203RWUser's Manual relating to the DSS Digital for Image Database Applications”, Technical Report 91-1, System. Department of Computer Science, Jackson, Miss., 39217. 0625 Z. Chen et al, “Computer Vision for Robust 3D 1990-1991. Aircraft Recognition with Fast Library Search', Pattern 0609 V. N. Gudivada and V. V. Raghavan. Design and Recognition, vol. 24, No. 5, pp. 375-390, 1991, printed in evaluation of algorithms for image retrieval by spatial Great Britain. similarity. ACM Trans. on Information Systems, 13(2), April 0626. Zhuang, Yueting, Rui, Yong, Huang, Thomas S. 1995. Mehotra, Sharad, “Applying Semantic Association to Sup 0610 Vittal, J., “Active Message Processing: Message as port Content-Based Video Retrieval’. Messengers', pp. 175-195, 1981. 0627 Video on Demand 0611 Voydock, Victor et al.; “Security Mechanisms in 0628 Video on demand has long been sought as a means High-Level Network Protocols', Computing Surveys; vol. for delivering personalized media content. The practical 15, No. 2: June 1981. systems raise numerous issues, including data storage for 0612 W. Gellert, H Kustner, M Hellwich & H Kastner: mats, retrieval software, server hardware architecture, mul “The VNR Concise Encyclopedia of Mathematics”. Van titasking and buffering arrangements, physical communica Nostrand Reinhols Co., New York, 1975. tions channel, logical communications channel, receiver and 0613 W. Grosky et al., “A Pictorial Index Mechanism for decoder system, user interface, etc. In addition, typically a Model-based Matching. Data 7 Knowledge Engineering 8. pay-per-view concept may be employed, with concomitant 1992, pp. 309-327. Subscription, royalty collection and accounting issues. See, C.2. 0614 W. Grosky et al., “Index-based Object Recognition 0629. A. D. Gelman, et al.: A Store-And-Forward Archi in Pictorial Data Management'. Computer Vision, 1990, pp. tecture For Video-On-Demand Service, ICC 91 Conf; June 416-436. 1991; pp. 842-846. 0615 W. Niblack et al., “Find me the Pictures that Look 0630 Caitlin Bestler: Flexible Data Structures and Inter Like This: IBM'S Image Query Project’, Advanced Imag face Rituals For Rapid Development of OSD Applications: ing, April 1993, pp. 32-35. 93 NCTATech. Papers; Jun. 6, 1993: pp. 223-236. 0616 W. Niblack, R. Barber, W. Eitz, M. Flickner, E. Glasman, D. Petkovic, P. Yanker, and C. Faloutsos. The 0631 Consumer Digest advertisement: Xpand Your TV's QBIC project: Querying images by content using color, Capability: Fall/Winter 1992; p. 215. texture, and shape. In Storage and Retrieval for Image and 0632 Daniel M. Moloney: Digital Compression in Video Databases, volume SPIE Vol. 1908, February 1993. Todays Addressable Enviroment: 1993 NCTA Technical 0617 W. T. Freeman et al., “The Design and Use of Papers, Jun. 6, 1993: pp. 308-316. Steerable Filters’, IEEE Transactions on Pattern Analysis 0633 Great Presentations advertisement: Remote, and Machine Intelligence, vol. 15, No. 9, September 1991, Remote: 1987; p. 32H. pp. 891-906. 0634 Henrie van den Boom: A Interactive Videotex 0618. Weber et al., “Marquee: ATool for Real-Time Vdeo System for Two-Way CATV Networks: AEU, Band 40: Logging, CHI '94. April 1994. 1986; pp. 397-401. 0619 Willett, P., “Recent Trends in Hierarchic Document 0635 Hong Kong Enterprise advertisement: Two Inno Clustering: A Critical Review”, Information Processing & vative New Consumer Products From SVI; November 1988: Management, vol. 24, No. 5, pp. 557-597, 1988 p. 379. US 2007/0016476 A1 Jan. 18, 2007

0636 IEEE Communications Magazine, vol. 32, No. 5, 0.643. This is the basis for a “least common denominator May 1994 New York, N.Y., US, pp. 68-80, XP 000451097 theory of marketing, that mass media must merchandise to Chang et al “An Open Systems Approach to Video on the masses, while specialty media merchandises to selected Demand. Subpopulations. As a corollary, using Such types of media, it 0637 Proc. IEEE, 82(4): 585-589 (April 1994) New may be difficult to reach certain specialized populations who York, N.Y., XP 000451419 Miller “A Scenario for the do not consistently receive a common set of publications or Deployment of Interactive Multimedia Cable Television who receive primarily publications which are unspecialized Systems in the United States in the 1990’s”. or directed to a different specialty. 0644) Where a recipient has limited time for reviewing 0638) Reimer, “Memories in my Pocket, Byte, pp. 251 media, he or she must divide his or her available time 258, February 1991. between mass media and specialty media. Alternatively, 0639 Sharpless, “Subscription teletext for value added publication on demand services have arisen which select services’. August 1985. content based on a user's expressed interests. Presumably, these same content selection algorithms may be applied to Demographically Targeted Advertising through Electronic commercial messages. However, these services are prima Media rily limited distribution, and have content that is as variable 0640 Since the advent of commercially subsidized print as commercial messages. Likewise, mass media often has media, attempts have been made to optimize the placement regionally variable content, Such as local commercials on and compensation aspects relating to commercial messages television or cable systems, or differing editions of print or advertisements in media. In general, advertisers Subsidize media for different regions. Methods are known for demo a large percentage of the cost of mass publications and graphic targeting of commercial information to consumers, communications, in return for the inclusion and possibly however, both the delivery methods and demographic tar strategic placement of advertisements in the publication. geting methods tend to be suboptimal. Therefore, the cost of advertising in such media includes the 0.645 Sometimes, however, the system breaks down, cost of preparation of the advertisement, a share of the cost resulting in inefficiencies. These result where the audience of publication and a profit for the content provider and other or a Substantial proportion thereof is inappropriate for the services. Since the advertiser must bear some of the cost of material presented, and thus realize a low response rate for production and distribution of the content, in addition to the an advertiser or even a negative response for the media due cost of advertisement placement itself, the cost may be to the existence of particular commercial advertisers. The substantial. The advertiser justifies this cost because the recipients are bombarded with inappropriate information, wide public reception of the advertisement, typically low while the advertiser fails to realize optimal return on its cost per consumer “impression', with a related stimulation advertising expenditures. In order to minimize the occur of sales due to commercial awareness of the advertisers rence of these situations, services are available, including products and services. Therefore, the advertisement is A.C. Nielsen Co. and Arbitron, Inc., which seek to deter deemed particularly effective if either the audience is very mine the demographics of the audience of broadcast media. large, with ad response proportionate to the size of the 0646 U.S. Pat. No. 5,436,653, incorporated herein by audience, or if it targets a particularly receptive audience, reference, relates to a broadcast segment recognition system with a response rate higher than the general population. in which a signature representing a monitored broadcast 0641. On the other hand, the recipient of the commercial segment is compared with broadcast segment signatures in publication is generally receptive of the advertisement, even a data base representing known broadcast segments to though it incurs a potential inefficiency in terms of increased determine whether a match exists. Therefore, the broadcast data content and inefficiencies in receiving the content viewing habits of a user may be efficiently and automatically segment, for two reasons. First, the advertisements subsidize monitored, without pre-encoding broadcasts or the like. the publication, lowering the monetary cost to the recipient. 0647 U.S. Pat. No. 5,459,306, incorporated herein by Second, it is considered economically efficient for a recipi reference, relates to a method for delivering targeting infor ent to review commercial information relating to prospective mation to a prospective individual user. Personal user infor purchases or expenditures, rather than directly soliciting mation is gathered, as well as information on a users use of Such information from the commercial source, i.e., "push’ is a product, correlated and stored. Classes of information better than “pull'. For this reason specialty publications are potentially relevant to future purchases are then identified, produced, including commercial messages appropriate for and promotions and recommendations delivered based on the particular content of the media or the intended recipients. the information and the user information. In fact, in Some forms of publications, most, if not all the 0648 U.S. Pat. No. 5,483.278, incorporated herein by information content is paid advertisements, with few edito reference, relates to a system having a user interface which rial or independently produced pieces. can access downloaded electronic programs and associated 0642 Mass media, on the other hand, tends not to include information records, and which can automatically correlate specialty commercial messages, because the interested the program information with the preferences of the user, to population is too disperse and the resulting response rate create and display a personalized information database from an advertisement too low, and further because the based upon the results of the correlation. Likewise, U.S. Pat. majority of the audience will be disinterested or even No. 5,223,914, expressly incorporated herein by reference, respond negatively to certain messages. Thus, mass media relates to a system and method for automatically correlating generally includes a majority of retail advertisements, with user preferences with a T.V. program information database. specialty advertisements relegated, if at all, to a classified 0649) U.S. Pat. No. 5,231,494, expressly incorporated section which is not interspersed with other content. herein by reference, relates to a system which selectively US 2007/0016476 A1 Jan. 18, 2007 24 extracts one of a plurality of compressed television signals image is repeated in sequential frames, with possible move from a single channel based on viewer characteristics. ment, this redundant information is transmitted infrequently and filtered from the transmitted image stream, and 0650 U.S. Pat. No. 5,410,344 relates to a system for described with motion vector information. This motion selecting video programs based on viewers preferences, vector information encodes objects which are fixed or move based on content codes of the programs. Somewhat between frames. Such known techniques include 0651 U.S. Pat. No. 5,485,518, incorporated herein by H.261, with integer pixel motion estimation, and H.263, reference, relates to a system for electronic media program which provides 4 pixel motion estimation. Other techniques recognition and choice, allowing, for example, parental for video compression are known or have been proposed, control of the individual programs presented, without requir such as H.263+, and MPEG-4 encoding. Many standard ing a transmitted editorial code. Videoconferencing protocols require the initial transmission of a full frame image, in order to set both transmitting and 0652 Videoonferencing Technologies receiving stations to the same encoding state. The digital 0653 Videoconferencing systems are well known in the data describing this image is typically Huffman encoded for art. A number of international standards have been defined, transmission. Multiple frames may be combined and coded providing various telecommunication bandwidth and com as a unit, for example as so-called PB frames. Other tech munication link options. For example, H.320, H.323 and niques are also known for reducing image data transmission H.324 are known transport protocols over ISDN, packet bandwidth for various applications, including video confer switched networks and public switched telephone networks, encing. respectively. H.324 provides a multimedia information com 0656. Each remote videoconference terminal has an inter munication and videoconferencing standard for communi face system, which receives the digital data, and separates cation over the standard “plain old telephone system net the video information (H.261, H.263), audio information work (“POTS), in which the video signal is compressed (G.711, G.723, G.723.1), data protocol information (HDLC, using DCT transforms and motion compensation for trans V.14, LAPM, etc.) and control information (H.245, H.221/ mission over a V.80 synchronous V.34-type modem link. The H.223) into discrete streams, which are processed sepa video image is provided as a video window with relatively rately. Likewise, each terminal interface system also slow frame rate. This image, in turn, may be presented on a assembles the audio information, video information, data computer monitor or television system, with appropriate protocols and control data for transmission. signal conversion. See, Andrew W. Davis, "Hi Grandma: Is It Time for TV Set POTS Videoconferencing?', Advanced 0657 The control information consists of various types Imaging, pp. 45-49 (March 1997); Jeff Child, “H.324 Paves of information, the standard control protocol which Road For Mainstream Video Telephony”, Computer Design, addresses the data format, error correction, exception han January 1997, pp. 107-110. A newly proposed set of exten dling, and other types of control; and the multipoint control sions to H.324, called H.324/M, provides compatibility with information, such as which remote videoconference termi mobile or impaired telecommunications systems, and nal(s) to receive audio information from, selective audio accommodates errors and distortions in transmissions, muting, and such. Generally, the standard, low level control reduced or variable transmission rates and other anomalies information is processed locally, at the codec interface of known available mobile telecommunications systems, system, and filtered from the remainder of the multipoint such as Cellular, GSM, and PCS. control system, with only the extracted content information made available to the other stations. 0654 Four common standards are employed, which are necessary for videoconferencing stations to communicate 0.658. The ITU has developed a set of multipoint video with each other under common standards. The first is called conferencing standards or recommendations, T. 120-T133, h.320, and encompasses relatively high bandwidth systems, T.RES series, H.231, H.243, etc. These define control in increments of 64 kbits/sec digital communication with a schemes for multiple party video conferences. Typically, synchronous communication protocol. Generally, these sys these protocols are implemented in Systems which either tems communicate with 128 kbits/sec. 256 kbits/sec or 384 identically replicate the source image data stream from one kbits/sec, over a number of “bonded ISDN B-channels. The Source to a plurality of destinations, or completely decode second standard H.324, employs a standard POTS commu and reencode the image in a different format in a nication link with a V.80/v.34bis modem, communicating at “transcoder arrangement, to accommodate incompatible 33.6 kbits/sec synchronous. The third standard, is the newly conference stations. The ITU standards also allow optional established H.321 standard, which provides for videocon data fields which may be used to communicate digital ferencing over a packet Switched network, Such as Ethernet, information essentially outside the videoconference Scheme, using IPX or TCP/IP. Finally, there are so-called Internet and provide data conferencing capabilities, which allow videophone systems, such as Intel Proshare. See, Andrew W. Videoconferencing and data conferencing to proceed simul Davis, “The Video swering Machine: Intel ProShare's Next taneously. See, ITU T. 120-T127, T. 130-T133, TRES, Step”. Advanced Imaging, pp. 28-30 (March 1997). T.Share and T.TUD recommendations, expressly incorpo 0655. In known standards-based videoconferencing sys rated herein by reference. tems, the image is generally compressed using a discrete 0659 There are a number of known techniques for trans cosine transform, which operates in the spatial frequency mitting and displaying alphanumeric data on a television, domain. In this domain, visually unimportant information, the most common of which are teletext, used primarily in Such as low frequencies and high frequency noise are Europe, and closed caption, which is mandated in television eliminated, leaving visually important information. Further, sets larger than 13 inches by the Television Decoder Cir because much of the information in a videoconference cuitry Act of 1990, and Section 305 of the Telecommuni US 2007/0016476 A1 Jan. 18, 2007

cations Act of 1996, and Federal Communication Commis a continuous connection or an autodial function, to create a sion (FCC) regulations. The American closed caption communications link as necessary. Therefore, heating ven standard is EIA 608. The later is of particular interest tilation and air conditioning control (HVAC), lighting, appli because many current generation televisions, especially ances, machinery, valves, security sensors, locks, gates, larger sizes, include a closed caption decoder, and thus access points, etc., may all be controlled locally or remotely require no external hardware or connections, separate from through interfaces of the local system, which may include the hardware and cabling for Supplying the video signal. logic level signals, relays, serial ports, computer networks, See, TCC Tech Facts, Vols. 1-4 (www.wgbh.org, rev. Sep. fiber optic interfaces, infrared beams, radio frequency sig 1995) expressly incorporated herein by reference. The nals, transmissions through power lines, standard-type com closed caption signal is distributed on Line 21 of the vertical puter network communications (twisted pair, coaxial cable, blanking interval. The existing standard supports 480 bits/ fiber optic cable), acoustic transmissions and other known sec, with a potential increase to 9600 bits/sec in the forth techniques. Likewise, inputs from various devices and sen coming ATSC standard. sors, such as light or optical, temperature, humidity, mois ture, pressure, fluid level, security devices, radio frequency, 0660 Known systems provide a videoconferencing sys acoustic, may be received and processed locally or remotely. tem which resides in a “set top box’ i.e., a stand-alone A video and audio signal transmission may also be com hardware device suitable for situation on top of a television bined with the data signals, allowing enhanced remote set, providing all of the necessary functionality of a video monitoring and control possibilities. This information, when conferencing system employing the television as the display transmitted through the telecommunication link, may be and possibly audio speaker functions. These systems, how directed to another remote terminal, for example a monitor ever, do not integrate the television functions, nor provide ing service or person seeking to monitor his own home, or interaction between the video and videoconferencing sys intercepted and processed at a central control unit or another tems. C-Phone Inc., Wilmington N.C., provides a C-Phone device. Remote events may be monitored, for example, on Home product line which provides extensions to H.324 a closed caption display mode of a television attached to a and/or H.320 communications in a set-top box. Videophone. 0661. Other known videophone and videoconferencing devices are disclosed, e.g., in U.S. Pat. Nos. 5,600,646; 0665 While the preferred embodiments of the invention 5,565,910; 5,564,001: 5,555,443; 5,553,609; 5,548,322; adhere to established standards, the present invention also 5,542,102: 5,537,472: 5,526.405; 5,509,009; 5,500,671; encompasses communications which deviate from or extend 5,490,208; 5,438,357; 5,404,579; 5,374,952: 5,224,151: beyond Such standards, and thus may engage in proprietary 4,543,665; 4,491,694; 4,465,902: 4,456,925; 4,427,847: communications protocols, between compatible units. 4.414,432; 4,377,729; 4,356,509; 4,349,701; 4,338,492; OTHER REFERENCES 4,008,376 and 3,984,638 each of which is expressly incor 0.666. In addition, the following patents are considered porated herein by reference. relevant to the data compression and pattern recognition 0662. Known Web/TV devices (from Sony/Magnavox functions of the apparatus and interface of the present Philips) allow use of a television to display alphanumeric invention and are incorporated herein by reference: U.S. Pat. data, as well as audiovisual data, but formats this data for Nos. 3,609.684; 3,849,760; 3,950,733; 3,967,241: 4,025, display outside the television. In addition, embedded Web 851; 4,044,243; 4,100,370; 4,118,730; 4,148,061; 4,213, servers are also known. See, Richard A. Quinell, “Web 183, 4,225,850, 4,228,421, 4,230,990, 4,245,245, 4,254, Servers in embedded systems enhance user interaction', 474; 4,264,924; 4,264,925; 4,305,131; 4,326,259; 4.331, EDN, Apr. 10, 1997, pp. 61-68, incorporated herein by 974; 4,338,626; 4,390,904; 4,395,780; 4.420,769: 4,442, reference. Likewise, combined analog and digital data trans 544; 4,449,240; 4,450,531; 4,468,704; 4,491.962: 4,499, mission schemes are also known. See. U.S. Pat. No. 5,404, 601; 4,501,016: 4,511,918; 4,543,660; 4,546,382: 4,547, 579. 811; 4,547,899; 4,581,762: 4,593,367; 4,602,279; 4,630, 0663 A class of computing devices, representing a con 308; 4,646,250; 4,656,665; 4,658,429: 4,658,370; 4,660, vergence of personal computers and entertainment devices, 166; 4,677,466; 4,697,209; 4,672,683; 4,677,680; 4,682, and which provide network access to the Internet (a publicly 365; 4,685, 14.5; 4,695,975; 4,710,822; 4,710,964; 4,716, available network operating over TCP/IP). ITU standards 404; 4,719,591; 4,731,863; 4,734,786; 4,736,439; 4,739, for communications systems allow the selective addition of 398; 4,742,557: 4,747,148; 4,752,890; 4,653,109: 4,760, data, according to T. 120-T133, T.RES series of protocols, as 604; 4,764,971; 4,764,973: 4,771,467; 4,773,024; 4,773, well as HDLC, V.14, IAPM, to the videoconference stream, 099; 4,774,677; 4,775,935; 4,783,752: 4,783,754; 4,783, especially where excess bandwidth is available for upload or 829; 4,789,933; 4,790,025; 4,799,270; 4,802,103; 4,803, download. 103: 4,803,736; 4,805,224; 4,805,225; 4,805.255; 4,809, 331; 4,809,341: 4,817,171; 4,817, 176: 4,821,333; 4,823, 0664) A system may be provided with features enabling 194; 4,829,453; 4,831,659; 4,833,637; 4,837,842; 4,843, it to control a so-called Smart house and/or to be a part of a 562; 4,843,631; 4,845,610; 4,864,629; 4,872,024; 4,876, security and/or monitoring system, with imaging capability. 731; 4,881,270; 4,884,217: 4,887,304; 4,888,814; 4,891, These functions are provided as follows. As discussed 762; 4,893,346; 4,897,811; 4,905,162: 4,905,286; 4,905, above, various data streams may be integrated with a vid 296; 4,906,099: 4,906,940; 4,908,758: 4,914,708; 4,920, eoconference data stream over the same physical link. 499: 4,926,491; 4,930,160; 4,931,926; 4,932,065; 4,933, Therefore, external inputs and outputs may be provided to 872: 4,941, 193; 4,944,023; 4,949, 187: 4,956,870; 4,958, the videophone or videoconference terminal, which maybe 375; 4,958,375; 4,964,077; 4,965,725: 4,967,273; 4,972, processed locally and/or transmitted over the telecommuni 499: 4,979,222: 4,987,604; 4,989,256; 4,989,258: 4,992, cations link. The local device, in this case, is provided with 940; 4,995,078; 5,012,334; 5,014,219; 5,014,327; 5,018, US 2007/0016476 A1 Jan. 18, 2007 26

218: 5,018,219; 5,019,899; 5,020, 112: 5,020,113: 5,022, in the art and relate to the pattern recognition and image 062: 5,027,400; 5,031,224; 5,033,101; 5,034,991; 5,038, compression functions of the apparatus and interface of the 379; 5,038,390; 5,040,134; 5,046,121: 5,046,122; 5,046, present invention: 179; 5,047,867; 5,048, 112: 5,050,223: 5,051,840: 5,052, 0668 "Fractal Geometry-Understanding Chaos', Geor 043; 5,052,045; 5,052,046; 5,053,974; 5,054,093; 5,054, gia Tech Alum. Magazine, p. 16 (Spring 1986). 095; 5,054,101; 5,054,103; 5,055,658; 5,055,926; 5,056, 147; 5,058,179; 5,058,180. 5,058,183; 5,058, 186: 5,059, 0669) “Fractal Modelling of Biological Structures”, 126; 5,060,276: 5,060,277; 5,060,279; 5,060,282: 5,060, School of Mathematics, Georgia Institute of Technology 285: 5,061,063: 5,063,524; 5,063.525; 5,063,603; 5,063, (date unknown). 605; 5,063,608; 5,065,439; 5,065,440; 5,065,447; 5,067, 0670) “Fractal Modelling of Real World Images”. Lec 160; 5,067,161; 5,067,162; 5,067,163; 5,067,164: 5,068, ture Notes for Fractals: Introduction, Basics and Perspec 664: 5,068,723; 5,068,724; 5,068,744; 5,068,909; 5,068, tives, Siggraph (1987). 911; 5,076,662: 5,099,422: 5,103,498: 5,109,431; 5,111, 0671) “Fractals Yield High Compression’, Electronic 516; 5,119,507: 5,122,886; 5,130,792: 5,132,992: 5,133, Engineering Times, Sep. 30, 1991, p. 39. 021; 5,133,079; 5,134,719; 5,148,497; 5,148,522; 5,155, 591; 5,159,474; 5,161,204; 5,168,529; 5,173,949; 5,177, 0672) “Fractals—A Geometry of Nature', Georgia Inst. 796; 5,179,652; 5,202,828: 5,220,420; 5,220,648: 5,223, Tech. Research Horizons, p. 9 (Spring 1986). 924; 5,231,494; 5,239,617; 5,247,347; 5,247,651: 5,259, 0673 “Voice Recognition and Speech Processing, Ele 038: 5,274,714; 5,283,641; 5,303,313; 5,305,197; 5,307, ktor Electronics, September 1985, pp. 56-57. 421; 5,315,670; 5,317,647; 5,317,677; 5,343,251; 5,351, 078; 5,357,276; 5,381,158: 5,384,867; 5,388,198: 5,390, 0674) Aleksander, I., "Guide to Pattern Recognition 125; 5,390,281; 5,410,343; 5,410,643; 5,416,856; 5,418, Using Random-Access Memories’. Computers and Digital 951; 5,420,975; 5,421,008: 5,428,559; 5,428,727; 5,428, Techniques, 2(1):29-40 (February 1979). 730; 5,428,774; 5,430,812; 5,434,933; 5,434,966: 5,436, 0675 Anderson, F. W. Christiansen, B. Kortegaard, 653; 5,436,834; 5,440,400; 5,446,891; 5,446,919; 5,455, “Real Time, Video Image Centroid Tracker”, Apr. 16-20, 892; 5,459.517; 5,461,699; 5,465,308; 5,469,206; 5,477, 1990. 447: 5,479,264; 5,481,294; 5,481,712; 5,483.278: 5,485, 219; 5,485,518; 5,487,132: 5,488,425; 5,488,484; 5,495, 0676 Anson, L., M. Barnsley, “Graphics Compression 292; 5,496,177; 5,497,314: 5,502,774; 5,504,518; 5,506, Technology”, SunWorld, pp. 43-52 (October 1991). 768: 5,510,838: 5,511,134: 5,511,153: 5,515,098: 5,515, 0677 Appriou, A., “Interet des theories de l’incertain en 099; 5,515, 173; 5,515,453; 5,515,471; 5,517,598; 5,519, fusion de donnees’, Colloque International sur le Radar 452: 5,521,841; 5,521,984; 5,522,155; 5,523,796; 5,524, Paris, 24-28 avril 1989. 065; 5,526,427; 5,535,302: 5,541,638; 5,541,662: 5,541, 738; 5,543,929; 5,544,254; 5,546,475; 5,548,667; 5,550, 0678. Appriou, A., “Procedured aide a la decision multi 575: 5,550,928; 5,550,965; 5,552,833; 5,553,221; 5,553, informateurs. Applications a la classification multi-capteurs 277; 5,554,983; 5,555,495; 5,557,728; 5,559,548; 5,560, decibles”. Symposium de lAvionics Panel (AGARD) Tur 011; 5,561,649; 5,561,718; 5,561,796; 5,566,274; 5,572, quie, 25-29 avril 1988. 604; 5,574,845; 5,576,950, 5,579,471; 5,581,658; 5,586, 0679 Arrow, K. J., “Social choice and individual 218: 5,588,074; 5,592,560; 5,574,845; 5,579,471; 5,581, valves”, John Wiley and Sons Inc. (1963). 665; 5,581,800; 5,583,560; 5,586,025; 5,594,661; 5,594, 911; 5,596,705; 5,600,733; 5,600,775; 5,604,542: 5,604, 0680 Barnsley et al., “A Better Way to Compress 820; 5,604.823: 5,606,655; 5,611,020; 5,613,032: 5,614, Images”. Byte Magazine, January 1988. 940; 5,617,483; 5,617,565; 5,621,454; 5,621,484; 5,621, 0681 Barnsley et al., “Harnessing Chaos For Images 579; 5,621,903; 5,625,715; 5,625,783: 5,627,915; 5,634, Systhesis”. Computer Graphics, 22(4) (August 1988). 849; 5,635,986: 5,642,434; 5,644,686; 5,644,735; 5,654, 771; 5,655,117; 5,657,397; 5,659,653; 5,659,368; 5,659, 0682 Barnsley et al., “Hidden Variable Fractal Interpo 732; 5,664.046: 5,668,897; 5,671,343; 5,671,411; 5,682, lation Functions”. School of Mathematics, Georgia Institute 437; 5,696,964; 5,701,369; 5,710,601; 5,710,833; 5,710, of Technology, Atlanta, Ga. 30332, July 1986. 834; 5,715,400; 5,717,814; 5,724,424; 5,724,472: 5,729, 741; 5,734,893; 5,737,444; 5,740,274; 5,745,126; 5,745, 0683 Batchelor, B. G., “Pattern Recognition, Ideas in 640; 5,745,710; 5,751,286; 5,751,831; 5,754,938; 5,758, Practice'. Plenum Press, London & NY. (1978). 257; 5,761,655; 5,764,809; 5,767,893; 5,767,922; 5,768, 0684 Batchelor, B. G., “Practical Approach to Pattern 421; 5,768,426; 5,768,437; 5,778,181; 5,797,001; 5,798, Classification’. Plenum Press, London, N.Y., (1974). 785; 5,799,109; 5,801,750; 5,801,753; 5,805,763; 5,809, 471; 5,819,288: 5,828,809; 5,835,087; 5,850,352: 5,852, 0685 Bellman, R. E., L. A. Zadeh, "Decision making in 823: 5,857,181; 5,862,260; H 331; and Re. 33,316. The a fuzzy environment. Management Science, 17(4) (Decem aforementioned patents, some of which are mentioned else ber 1970). where in this disclosure, and which form a part of this 0686 Bhatnagar, R. K. L. N. Kamal, “Handling uncertain disclosure, may be applied in known manner by those skilled information: a review of numeric and non-numeric meth in the art in order to practice various embodiments of the ods’. Uncertainty in Artificial Intelligence, L. N. Kamal and present invention. J. F. Lemmer, Eds. (1986). 0667 The following scientific articles, some of which are 0687 Blair, D., R. Pollack, "La logique du choix collec discussed elsewhere herein, are understood by those skilled tif, Pour la Science (1983). US 2007/0016476 A1 Jan. 18, 2007 27

0688 Burr, D.J., “A Neural Network Digit Recognizer”, 0706 Dubois, D., N. Prade, “Combination of uncertainty Proc. 1986 IEEE International Conference of Systems, Man with belief functions: a reexamination', Proc. 9th Interna and Cybernetics, Atlanta, Ga., pp. 1621-1625. tional Joint Conference on Artificial Intelligence, Los Ange 0689 Caffery, B., "Fractal Compression Breakthrough les (1985). for Multimedia Applications'. Inside, Oct. 9, 1991. 0707 Dubois, D., N. Prade, “Fuzzy sets and systems 0690 Carpenter, G. A. S. Grossberg, “The Art of Adap Theory and applications”. Academic Press NY (1980). tive Pattern Recognition by a Self-Organizing Neural Net 0708 Dubois, D., N. Prade, “Theorie des possibilites: work”, IEEE Computer, March 1988, pp. 77-88. application a la representation des connaissances en infor 0691 Casasent, D., et al., “General I and Q Data Pro matique', Masson, Paris (1985). cessing on a Multichannel AO System”, Applied Optics, 0709) Duda, R. O., P. E. Hart, M. J. Nilsson, “Subjective 25(18):3217-24 (Sep. 15, 1986). Bayesian methods for rule-based inference systems”, Tech 0692 Caudill, M., “Neural Networks Primer-Part III Al nical Note 124, Artificial Intelligence Center, SRI Interna Expert, June 1988, pp. 53-59. tional. 0693 Chao, J. J. E. Drakopoulos, C. C. Lee, “An evi 0710 Dunning, B. B., “Self-Learning Data-Base For dential reasoning approach to distributed multiple hypoth Automated Fault Localization', IEEE, 1979, pp. 155-157. esis detection'. Proc. 20th Conf. on decision and control, 0711 Farrelle, Paul M. and Jain, il K, “Recursive Block Los Angeles, Calif., December 1987. Coding-A New Approach to Transform Coding, IEEE 0694 Chao, T.-H.; Hegblom, E.; Lau, B.: Stoner, W. W. Transactions on Communications, Com. 34(2) (February Miceli, W. J., “Optoelectronically implemented neural net 1986). work with a wavelet preprocessor. Proc. SPIE The Inter 0712 Fitzpatrick, J. M., J. J. Grefenstette, D. Van Gucht, national Society for Optical Engineering, 2026:472 “Image Registration by Genetic Search”. Conf. Proc. IEEE 82(1993). Southeastcon 1984, pp. 460-464. 0695 Chen et al., “Adaptive Coding of Monochrome and 0713 Fua, P. V., “Using probability density functions in Color Images”, November 1977, pp. 1285-1292. the framework of evidential reasoning Uncertainty in knowl 0696 Cheong, C. K. Aizawa, K: Saito, T.; Hatori, M., edge based systems”. B. Bouchon, R. R. Yager, Eds. “Adaptive edge detection with fractal dimension'. Transac Springer Verlag (1987). tions of the Institute of Electronics, Information and Com munication Engineers D-II, J76D-II(11):2459-63 (1993). 0714 Gogoussis et al., Proc. SPIE Intl. Soc. Opt. Eng., November 1984, Cambridge, Mass., pp. 121-127. 0697) Computer Visions, Graphics, and Image Process ing, 1987, 37:54-115. 07.15 Grossberg, S., G. Carpenter, “A Massively Parallel Architecture for a Self-Organizing Neural Pattern Recogni 0698 Computers and Biomedical Research 5, 388-410 tion Machine'. Computer Vision, Graphics, and Image Pro (1972). cessing, 1987, 37, 54-115, 252-315. 0699 Cooper, L. N., “A Possible Organization of imal 0716 Gullichsen, E., E. Chang, “Pattern Classification by Memory and Learning. Nobel 24, (1973), Collective Prop Neural Network: An Experiment System for Icon Recogni erties of Physical Systems, pp. 252-264. tion”, ICNN Proceeding on Neural Networks, March 1987, 0700 Crawford et al., “Adaptive Pattern Recognition pp. IV-725-32. Applied To Expert System For Fault Diagnosis. In Telecom 0717 Haruki, K. et al., “Pattern Recognition of Hand munications Equipment', pp. 10/1-8 (Inspec. Abstract No. written Phonetic Japanese Alphabet Characters’, Interna 86C010699, Inspec IEE (London) & IEE Coll. on “Adaptive tional Joint Conference on Neural Networks, Washington, Filters', Digest No. 76, Oct. 10, 1985). D.C., January 1990, pp. 11-515 to 11-518. 0701 Danielsson, Erik, et al., “Computer Architectures 0718 Hayashi, Y., et al., “Alphanumeric Character Rec for Pictorial Inf. Systems, IEEE Computer, November, ognition Using a Connectionist Model with the Pocket 1981, pp. 53-67. Algorithm”. Proc. International Joint Conference on Neural 0702 Dempster, A P., “A generalization of Bayesian Networks, Washington, D.C. Jun. 18-22, 1989, vol. 2, pp. inference'. Journal of the Royal Statistical Society, Vol. 30. 606-613. Series B (1968). 0719. Hayes, H. I.; Solka, J. L.; Priebe, C. E.; “Parallel 0703 Dempster, A. P. “Upper and lower probabilities computation of fractal dimension'. Proc. SPIE The Inter induced by a multivalued mapping. Annals of mathematical national Society for Optical Engineering, 1962:219-30 Statistics, no. 38 (1967). (1993). 0704. Denker, 1984 International Test Conf., October 0720 Hinton et al., “Boltzmann Machines: Constraint 1984, Philadelphia, Pa., pp. 558-563. Satisfaction Networks that Learn'. Tech. Report CMU-CS 85-119, Carnegie-Mellon Univ, May 1984. 0705 Dubois, D., “Modeles mathematiques de l'imprecis et de l’incertain en Vue d'applications aux techniques d'aide 0721 Hoare, F.; de Jager, G., “Neural networks for a la decision”. Doctoral Thesis, University of Grenoble extracting features of objects in images as a pre-processing (1983). stage to pattern classification'. Proc. 1992 South African US 2007/0016476 A1 Jan. 18, 2007 28

Symposium on Communications and Signal Processing. 0738 Ksienski et al., “Low Frequency Approach to Tar COMSIG 92 (Cat. No.92TH0482-0). Inggs, M. (Ed.), p. get Identification', Proc. IEEE. 63(12):1651-1660 (Decem 239-42 (1992). ber 1975). 0722 Hopfield et al., “Computing with Neural Circuits: 0739 Kyburg, H. E., “Bayesian and non Bayesian evi A Model”, Science, 233:625-633 (Aug. 8, 1986). dential updating, Artificial Intelligence 31:271-293 (1987). 0723) Hopfield, “Neural Networks and Physical Systems 0740) LeCun, Y. et al., “Handwritten Digit Recognition: with Emergent Collective Computational Abilities”. Proc. Applications of Neural.”. IEEE Comm. Magazine, Novem Natl. Acad. Sci. USA, 79:2554-2558 (April 1982). ber 1989, pp. 41-46. 0724 Hopfield, “Neurons with graded response have 0741. LeCun, Y., “Connectionism in Perspective', in R. collective computational properties like those of two-state Pfeifer, Z. Schreter, F. Fogelman, L. Steels (Eds.), 1989, neurons'. Proc. Natl. Acad. Sci. USA, 81:3088-3092 (May “Generalization and Network Design Strategies”, pp. 143 1984). 155. 0725 Hurtgen, B.: Buttgen, P., “Fractal approach to low 0742 Liepins, G. E., M. R. Hilliard, “Genetic Algo rate video coding. Proc. SPIE The International Society rithms: Foundations &Applications'. Annals of Operations for Optical Engineering, 2094(pt. 1):120-31 (1993). Research, 21:31-58 (1989). 0726) Information Processing 71, North-Holland Pub 0743) Lin, H. K, et al., “Real-Time Screen-Aided Mul lishing Company (1972) pp. 1530-1533. tiple-Image Optical Holographic Matched-Filter Correla 0727 Ishizuka, M., “Inference methods based on tor”, Applied Optics, 21 (18):3278-3286 (Sep. 15, 1982). extended Dempster and Shafer's theory for problems with 0744 Lippman, R. P., “An Introduction to Computing uncertainty/fuzziness”, New Generation Computing, Ohm with Neural Nets', IEEE ASSP Magazine, April 1987, pp. sha, Ltd, and Springer Verlag, 1:159-168 (1983). 4-22. 0728 Jackel, L. D., H. P. Graf, J. S. Denker, D. Hend 0745 Lippmann, R. P., “An Introduction to Computing erson and I. Guyon, “An Application of Neural Net Chips: with Neural Nets', IEEE ASSP Magazine, vol. 4(2):4-22 Handwritten Digit Recognition”, ICNN Proceeding, 1988, (April 1987). pp. 11-107-15. 0746) Liu, Y., “Extensions of fractal theory”, Proc. 0729 Jean, J. S. N., et al., “Input Representation and SPIE The International Society for Optical Engineering, Output Voting Considerations for Handwritten Numeral 1966:255-68(1993). Recognition with Backpropagation'. International Joint 0747 Liu, Y., “Pattern recognition using Hilbert space'. Conference on Neural Networks, Washington, D.C., January Proc. SPIE The International Society for Optical Engi 1990, pp. I-408 to I-411. neering, 1825:63-77 (1992). 0730 Jeffrey, R. J., “The logic of decision”. The U. of 0748 Mahalanobis, A., et al., “Minimum Average Cor Chicago Press, Ltd., London (1983)(2" Ed.). relation Energy Filters”, Applied Optics, 26(17):3633-40 0731 Kaufmann, A., “Introduction a la theorie des sous (Sep. 1, 1987). ensembles flous'', Vol. 1-3, Masson, Paris 1975. 0749 Martin, G. L. et al., “Recognizing Hand-Printed 0732 Keeney, R. L., B. Raiffa, “Decisions with multiple Letters and Digits. Using Backpropagation Learning. Tech objectives: Preferences and value tradeoffs', John Wiley and nical Report of the MCC, Human Interface Laboratory, Sons, New York (1976). Austin, Tex., January 1990, pp. 1-9. 0733 Kellman, P., “Time Integrating Optical Signal Pro 0750 McAulay, A. D., J. C. Oh, “Image Learning Clas cessing, Ph.D. Dissertation, Stanford University, 1979, pp. sifier System Using Genetic Algorithms, IEEE Proc. 51-55. National Aerospace & Electronics Conference, 2:705-710 0734 Kim, D. H.; Caulfield, H.J.; Jannson, T.: Kostrze (1989). wski, A.; Savant, G., “Optical fractal image processor for 0751 Miller, R. K., Neural Networks ((c) 1989: Fairmont noise-embedded targets detection’. Proc. SPIE The Inter Press, Lilburn, Ga.), pp. 2-12 and Chapter 4, “Implementa national Society for Optical Engineering, Vol. 2026 p. 144-9 tion of Neural Networks', pp. 4-1 to 4-26. (1993) (SPIE Conf: Photonics for Processors, Neural Net works, and Memories 12-15 Jul. 1993, San Diego, Calif., 0752 Molley, P., “Implementing the Difference-Squared USA). Error Algorithm Using Acousto-Optic Processor, SPIE, 0735. Kohonen, “Self-Organization & Memory”, Second 1098:232-239 (1989). Ed., 1988, Springer-Verlag, pp. 199-209. 0753 Molley, P., et al., “A High Dynamic Range Acousto-Optic Image Correlator for Real-Time Pattern Rec 0736 Kortegaard, B. L., “PAC-MAN, a Precision Align ment Control System for Multiple Laser Beams Self-Adap ognition”, SPIE, 938:55-65 (1988). tive Through the Use of Noise', Los Alamos National 0754 Mori, “Towards the construction of a large-scale Laboratory, date unknown. neural network, Electronics Information Communications 0737 Kortegaard, B. L., “Superfine Laser Position Con Association Bulletin PRU 88-59, pp. 87-94. trol Using Statistically Enhanced Resolution in RealTime'. 0755. Naik et al., “High Performance Speaker Verifica Los Alamos National Laboratory, SPIE-Los geles Technical tion.”, ICASSP 86, Tokyo, CII2243-4/86/0000-0881, IEEE Symp., Jan. 23-25, 1985. 1986, pp. 881-884. US 2007/0016476 A1 Jan. 18, 2007 29

0756 Ney, H., et al., “A Data Driven Organization of the 0774 Rumelhart, D. E., et al., “Learning Internal Repre Dynamic Programming Beam Search for Continuous sentations by Error Propagation’, Parallel Distr. Proc.: Speech Recognition’, Proc. ICASSP 87, pp. 833-836, 1987. Explorations in Microstructure of Cognition, 1:318-362 0757 Nilsson, N.J., The Mathematical Foundations of (1986). Learning Machines (c) 1990: Morgan Kaufmann Publish 0775 Rumelhart, D. E., et al., Parallel Distributed Pro ers, San Mateo, Calif.) and particularly section 2.6 “The cessing, (c) 1986: MIT Press, Cambridge, Mass.), and Threshold Logic Unit (TLU), pp. 21-23 and Chapter 6, specifically Chapter 8 thereof, “Learning Internal Represen “Layered Machines' pp. 95-114. tations by Error Propagation’, pp. 318-362. 0758 Ohsuga et al., “Entrainment of Two Coupled van 0776 Rutherford, H. G., F. Taub and B. Williams, der Pol Oscillators by an External Oscillation', Biological “Object Identification and Measurement from Images with Cybernetics, 51:225-239 (1985). Access to the Database to Select Specific Subpopulations of Special Interest', May 1986. 0759) Omata et al., “Holonic Model of Motion Percep tion', IEICE Tech. Reports, Mar. 26, 1988, 339-346. 0777 Rutter et al., “The Timed Lattice-A New Approach To Fast Converging Equalizer Design”, pp.VII/1-5 (Inspec. 0760 O'Neal et al., “Coding Isotropic Images', Novem Abstract No. 84C044315, Inspec lEE (London) & IEE ber 1977, pp. 697-707. Saraga Colloquium on Electronic Filters, May 21, 1984). 0761 Pawlicki, T. F. D. S. Lee, J. J. Hull and S. N. 0778 Sadadi, F., “Experiments in the use of fractal in Srihari, “Neural Network Models and their Application to computer pattern recognition’. Proc. SPIE The Interna Handwritten Digit Recognition”, ICNN Proceeding, 1988, tional Society for Optical Engineering, 1960:214-22(1993). pp. 11-63-70. 0779) Sakoe, H., “A Generalization of Dynamic Pro 0762) Perry et al., “Auto-Indexing Storage Device, IBM gramming Based Pattern Matching Algorithm Stack DP Tech. Disc. Bulletin, 12(8): 1219 (January 1970). Matching. Transactions of the Committee on Speech 0763 Peterson, Ivars, “Packing It In, Science News, Research, The Acoustic Society of Japan, p. S83-23, 1983. 131(18):283-285 (May 2, 1987). 0780 Sakoe, H., “A Generalized Two-Level DP-Match 0764) Priebe, C. E.; Solka, J. L.; Rogers, G. W., “Dis ing Algorithm for Continuous Speech Recognition'. Trans criminant analysis in aerial images using fractal based actions of the IECE of Japan, E65(11):649-656 (November features”. Proc. SPIE The International Society for Optical 1982). Engineering, 1962:196-208(1993). 0781 Scharlic, A., “Decider sur plusieurs criteres. Pan 0765) Proc., 6th International Conference on Pattern Rec orama de laide a la decision multicritere'. Presses Poly ognition 1982, pp. 152-136. techniques Romandes (1985). 0766) Psaltis, D., “Incoherent Electro-Optic Image Cor 0782 Schurmann, J., “Zur Zeichen und Worterkennung relator, Optical Engineering, 23(1):12-15 (January/Febru bein Automatischen schriftenlesen”, Wissenschaftlichl, ary 1984). Berichte, 52(1/2) (1979). 0767 Psaltis, D., “Two-Dimensional Optical Processing 0783 Scientific American, “Not Just a Pretty Face”, Using One-Dimensional Input Devices'. Proc. IEEE, March 1990, pp. 77-78. 72(7):962-974 (July 1984). 0784 Shafer, G., “A mathematical theory of evidence'. 0768 Rahmati, M.: Hassebrook, L. G., “Intensity- and Princeton University Press, Princeton, N.J. (1976). distortion-invariant pattern recognition with complex linear 0785 Shimizu et al., “Principle of Holonic Computer and morphology”. Pattern Recogntion, 27(4):549-68(1994). Holovision', Journal of the Institute of Electronics, Infor 0769 Reusens, E., “Sequence coding based on the fractal mation and Communication, 70(9):921-930 (1987). theory of iterated transformations systems”. Proc. SPIE 0786) Shinan et al., “The Effects of Voice Disguise.”, The International Society for Optical Engineering, ICASSP 86, Tokyo, CH2243-4/86/0000-0885, IEEE 1986, 2094(t.1): 132-40(1993). pp. 885-888. 0770 Rhodes, W., “Acousto-Optic Signal Processing: 0787 Silverston et al., “Spectral Feature Classification Convolution and Correlation, Proc. IEEE, 69(1):65-79 and Spatial Pattern Rec.”. SPIE 201:17-26, Optical Pattern (January 1981). Recognition (1979). 0771 Rosenfeld, Azriel and Avinash C. Kak, Digital 0788 Simpson, W. R., C. S. Dowling, “WAPLE: The Picture Processing, Second Edition, Volume 2, Academic Weighted Repair Assistance Program Learning Extension'. Press, 1982. IEEE Design & Test, 2:66-73 (April 1986). 0772 Roy, B., "Classements et choix en presence de 0789 Specht, IEEE Internatl. Conf. Neural Networks, points de Vue multiples, R.I.R.O.-2eme annee-no. 8, pp. 1:1525-1532 (July 1988), San Diego, Calif. 57-75 (1968). 0790 Sprageu, R. A., “A Review of Acousto-Optic Sig 0773 Roy, B., “Electre III: un algorithme de classements nal Correlators'. Optical Engineering, 16(5):467-74 (Sep fonde Sur une representation floue des preferences en pres tember/October 1977). ence de criteres multiples’. Cahiers du CERO, 2001): 3-24 0791 Sprinzak, J.; Werman, M., “Affine point matching, (1978). Pattern Recognition Letters, 15(4):337-9(1994). US 2007/0016476 A1 Jan. 18, 2007 30

0792 Stanley R. Sternberg, “Biomedical Image Process 0813 The above-mentioned references are exemplary, ing, IEEE Computer, 1983, pp. 22-34. and are not meant to be limiting in respect to the resources and/or technologies available to those skilled in the art. Of 0793 Stewart, R. M.. “Expert Systems For Mechanical course it should be realized that the hardware for imple Fault Diagnosis”, IEEE, 1985, pp. 295-300. menting a system may be integrally related to the choice of 0794 Sugeno, M.. “Theory of fuzzy integrals and its specific method or software algorithm for implementing the applications”, Tokyo Institute of Tech. (1974). system, and therefore these together form a system. It is noted that in view of the present disclosure, it is within the 0795 Svetkoffet al., Hybrid Circuits (GB), No. 13, May skill of the artisan to combine in various fashions the 1987, pp. 5-8. available methods and apparatus to achieve the advanced 0796 Udagawa, K, et al., “A Parallel Two-Stage Decision interface and control system of the present invention. Method for Statistical Character Recognition.”: Electronics and Communications in Japan (1965). SUMMARY AND OBJECTS OF THE 0797 Vander Lugt, A., “Practical Considerations for the INVENTION Use of Spatial Carrier-Frequency Filters'. Applied Optics, 0814 The present invention provides, according to one 5(11):1760-1765 (November 1966). embodiment, an adaptive user interface which changes in 0798 Vander Lugt, A., “Signal Detection. By Complex response to the context, past history and status of the system. Spatial Filtering, IEEE Transactions On Information The strategy employed preferably seeks to minimize, for an Theory, IT-10, 2:139-145 (April 1964). individual user at any given time, the search and acquisition time for the entry of data through the interface. 0799 Vander Lugt, A., et al., “The Use of Film Nonlin earites in Optical Spatial Filtering”. Applied Optics, 0815. The interface may therefore provide a model of the 9(1):215-222 (January 1970). user, which is employed in a predictive algorithm. The model parameters may be static (once created) or dynamic, 0800 Vannicola et al., “Applications of Knowledge and may be adaptive to the user or alterations in the use Based Systems to Surveillance', Proc. 1988 IEEE National pattern. Radar Conference, 20-21 Apr. 1988, pp. 157-164. 0816. The present invention also provides a model-based 0801 Vitols, “Hologram Memory for Storing Digital pattern recognition system, for determining the presence of Data”, IBM Tech. Disc. Bull 8(11):1581-1583 (April 1966). an object within an image. By providing models of the 0802 Wald, SequentialAnalysis, Dover Publications Inc., objects within an image, the recognition process is relatively 1947, pp. 34-43. unaffected by perspective, and the recognition may take place in a higher dimensionality space than the transmitted 0803 Wasserman, Philip D., “Neural Computing-Theory media. Thus, for example, a motion image may include four & Practice', 1989, pp. 128-129. degrees of freedom; X, y, chroma uma, and time. A model of 0804 Willshaw et al., “Non-Holographic Associative an object may include further dimensions, including Z, and Memory”, Nature, 222:960-962 (Jun. 7, 1969). axes of movement. Therefore, the model allows recognition 0805 Yager, R. R., “Entropy and specificity in a math of the object in its various configurations and perspectives. ematical theory of Evidence'. Int. J. General Systems, 0817. A major theme of the present invention is the use 9:249-260 (1983). of intelligent, adaptive pattern recognition in order to pro vide the operator with a small number of high probability 0806 Yamada et. al., “Character recognition system choices, which may be complex, without the need for using a neural network”. Electronics Information Commu explicit definition of each atomic instruction comprising the nications Association Bulletin PRU 88-58, pp. 79-86. desired action. The interface system predicts a desired action 0807 Yamane et al., “An Image Data Compression based on the user input, a past history of use, a context of Method Using Two-Dimensional Extrapolative Prediction use, and a set of predetermined or adaptive rules. Discrete Sine Transform”, Oct. 29-31, 1986, pp. 311-316. 0818 Because the present invention emphasizes adaptive 0808 Zadeh, L.A., “Fuzzy sets as a basis for a theory of pattern recognition of both the input of the user and data possibility', Fuzzy sets and Systems, 1:3-28 (1978). which may be available, the interface system proposes the 0809 Zadeh, L. A., “Fuzzy sets’, Information and Con extensive use of advanced signal processing and neural trol, 8:338-353 (1965). networks. These processing systems may be shared between the interface system and the functional system, and therefore 08.10 Zadeh, L. A., “Probability measures of fuzzy a controller for a complex system may make use of the events”, Journal of Mathematical Analysis and Applications, intrinsic processing power available rather than requiring 23:421-427 (1968). additional computing power, although this unification is not 0811 Zhi-Yan Xie: Brady, M., “Fractal dimension image required. In the case where the user interface employs for texture segmentation”, ICARCV 92. Second Int. Con common hardware elements, it is further preferred that the ference on Automation, Robotics and Computer Vision, p. interface Subsystem employ common models of the under CV-4.3/1-5 vol.1, (1992). lying data structures on which the device functionally oper 0812 Zhu, X., et al., “Feature Detector and Application ates. to Handwritten Character Recognition'. International Joint 0819. In fact, while hardware efficiency dictates common Conference on Neural Networks, Washington, D.C., January hardware for the interface system and the operational rou 1990, pp. II-457 to II-460. tine, other designs may separate the interface system from US 2007/0016476 A1 Jan. 18, 2007 the operational system, allowing portability and efficient 0822. The present interface system analyzes data from application of a single interface system for a number of the user, which may be both the selections made by the user operational systems. Thus, the present invention also pro in context, as well as the efficiency by which the user poses a portable human interface system which may be used achieves the selection. Thus, information concerning both to control a number of different devices. In this case, a web the endpoints and time-dependent path of the process are browser metaphor is preferred, as it has become a standard considered and analyzed by the interface system. for electronic communications. 0823. The interface of the present invention may be 0820) A portable interface may, for example, take the advantageously applied to an operational system that has a form of a personal digital assistant or downloaded JAVA plurality of functions, certain of which are unnecessary or applet, with the data originating in a web server. The data are rarely used in various contexts, while others are used from a web server or embedded web server may include a with greater frequency. In Such systems, the functionality binary file, a generic HTML/XML file, or other data type. use is usually predictable. Therefore, the present invention The interface receives the data and formats it based, at least provides an optimized interface system which, upon recog in part, on parameters specific to the client or user. Thus, the nizing a context, dynamically reconfigures the availability or presentation of data is responsive to the user, based on user ease of availability of functions and allow various subsets to preferences, as opposed to hardware limitations or compat be used through “shortcuts”. The interface presentation will ibility issues. In a preferred embodiment, the data is trans therefore vary over time, use and the particular user. mitted separately from the presentation definition. The pre 0824. The advantages to be gained by using an intelligent sentation definition, on the other hand, provides a set of data analysis interface for facilitating user control and parameters that propose or constrain the data presentation. operation of the system are more than merely reducing the The user system also provides a set of parameters that set average number of selections or time to access a given preferences on presentation. Further, the data itself is ana function. Rather, advantages also arise from providing a lyzed for appropriate presentation parameters. These three means for access and availability of functions not necessar sets of considerations are all inputs into a “negotiation' for ily previously existing or known to the user, therefore an ultimate presentation scheme. Thus, the presentation is improving the perceived quality and usefulness of the prod adaptive to server parameters, user parameters, and the data uct. Further advantages over prior interfaces accrue due to itself. For example, in a typical web-context, the color, size, the availability of pattern recognition functionality as a part typestyle, and layout of text may be modified based on these of the interface system. considerations. Other factors that may be altered include frame size and layout, size of hotspots, requirement for 0825. In those cases where the pattern recognition func single or double clicks for action, and the like. tions are applied to large amounts of data or complex data sets, in order to provide a Sufficient advantage and accept 0821. The adaptive nature of the present invention able response time, powerful computational resources, such derives from an understanding that people learn most effi as advanced DSPs or neural network processors are made ciently through the interactive experiences of doing, think available to the interface system. On the other hand, where ing, and knowing. For ease-of-use, efficiency, and lack of the data is simple or of limited scope, aspects of the frustration of the user, the interface of the device should be technology may be easily implemented as added software intuitive and self explanatory, providing perceptual feedback functionality as improvements of existing products having to assist the operator in communicating with the interface, limited computational resources. which in turn allows the operational system to receive a description of a desired operation. Another important aspect 0826. The application of these technologies to multime of man-machine interaction is that there is a learning curve, dia systems provides a new model for performing image which dictates that devices which are especially easy to pattern recognition on multimedia data and for the program master become frustratingly elemental after continued use, ming of applications including Such data. The ability of the while devices which have complex functionality with many interface of the present invention to perform abstractions options are difficult to master and may be initially rejected, and make decisions regarding a closeness of presented data or the user stops exploring. One Such system which to selection criteria makes the interface suit able for use in addresses this problem is U.S. Pat. No. 5,005.084, expressly a programmable control, i.e., determining the existence of incorporated herein by reference. The present invention certain conditions and taking certain actions on the occur addresses these issues by determining the most likely rence of detected events. Such advanced technologies might instructions of the operator, and presenting these as easily be especially valuable for disabled users. available choices, by analyzing the past history data and by 0827. In a multimedia environment, a user often wishes detecting the 'sophistication of the user in performing a to perform an operation on a multimedia data event. Past function, based on all information available to it. The systems have required explicit indexing of images and context of use may also be a significant factor. The interface events. The present technologies, however, allow an image, seeks to optimize the relevant portion of the interface diagrammatic, abstract or linguistic description of the adaptively and immediately in order to balance and optimize desired event to be acquired by the interface system from the the interface for both quantitative and qualitative factors. user and applied to identify or predict the multimedia This functionality may greatly enhance the quality of inter event(s) desired without requiring a separate manual index action between man and machine, allowing a higher degree ing or classification effort. These technologies may also be of overall system Sophistication to be tolerated and a greater applied to single media data. value added than other interface designs. See, Commaford, C. “User-Responsive Software Must Anticipate Our 0828 The interface system according to the present Needs”, PC Week, May 24, 1993. invention is not limited to a single data source, and may US 2007/0016476 A1 Jan. 18, 2007 32 analyze data from many different sources for its operation. compressed and used directly by the pattern recognition This data may be stored data or present in a data stream. system, great efficiencies in storage and data transmission Thus, in a multimedia system, there may be a real-time data are achieved. The image preprocessing may include Fourier, stream, a stored event database, as well as an exemplar or DCT wavelet, Gabor, fractal, or model-based approaches, or model database. Further, since the device is adaptive, infor a combination thereof. mation relating to past experience of the interface, both with respect to exposure to data streams and user interaction, is 0832. The potential significant hardware requirement for also stored. This data analysis aspect of the operation of the image processing and pattern recognition is counterbalanced present interface system may be substantially processor by the enhanced functionality available by virtue of the intensive, especially where the data includes abstract or technologies. When applied to multimedia devices, the linguistic concepts or images to be analyzed. Interfaces interface system allows the operator to define complex which do not relate to the processing of Such data may be criteria with respect to image, abstract or linguistic concepts, implemented on simpler hardware. On the other hand, which would otherwise be difficult or impossible to formu systems which handle complex data types may necessarily late. Thus, the interface system becomes part of a compu include Sophisticated processors, adaptable for use with the tational system that would otherwise be too cumbersome for interface system, thus minimizing the additional computing use. It is noted that, in many types of media streams, a power necessary in order to implement the interface accord number of "clues” are available defining the content, includ ing to the present invention. A portion of the data analysis ing close caption text, electronic program guides, simulcast may also overlap the functional analysis of the data for data, related Internet web sites, audio tracks, image infor operation. mation, and the like. The latter two data types require difficult processing in order to extract a semantic content, 0829. A fractal-based image processing system exempli while the former types are inherently semantic data. fies one application of the technologies. A fractal-based system includes a database of image objects, which may be 0833) A pattern recognition subsystem allows a “descrip preprocessed in a manner which makes them Suitable for tion of an “event without explicit definition of the data comparison to a fractal-transformed image representation of representing the "event. Thus, instead of requiring explicit an image to be analyzed. Thus, corresponding “fractal programming, an operator may merely define parameters of transforms are performed on the unidentified image or a the desired “event'. This type of system is useful, for portion thereof and on an exemplar of a database. A degree example, where a user seeks a generic type of data repre of relatedness is determined in this “fractal transform senting a variety of events. This eliminates the need for domain', and the results used to identify objects within the preindexing or standardized characterization of the data. The image. The system then makes decisions based on the interface system therefore facilitates the formulation of a information content of the image, i.e. the objects contained request, and then searches the database for data which therein. corresponds to the request. Such preindexing or standard ized characterization is extremely limiting with image and 0830) The fractal-based image processing system pre multimedia data, because “a picture is worth a thousand sents many advantages. First, fractal-processed images may words, and without a priori knowing the ultimate search have dramatically reduced storage size requirements as criteria, all possible criteria must be accounted for. Pattern compared to traditional methods while substantially retain recognition systems do not require initial translation of ing information important for image recognition. The pro visual aspects into linguistic concepts, thus allowing broader cess may be parallelized, and the exemplars may be multi searching capability. Of course, a pattern recognition system dimensional, further facilitating the process of identifying a may be used in conjunction with other searching schemes, to two-dimensional projection of an object. The efficient stor mutual advantage. age of information allows the use of inexpensive storage media, i.e., CD-ROM, or the use of an on-line database 0834 The pattern recognition functionality of the inter through a serial data link, while allowing acceptable face system is not limited to multimedia data, and may be throughput. See, Zenith Starsight Telecast brochure, (1994); applied to data of almost any type, e.g., real-time sensor U.S. Pat. No. 5,353,121, expressly incorporated herein by data, distributed control, linguistic data, etc. reference. 0835. It is noted that, in consumer electronics and par 0831. As applied to a multimedia database storage and ticularly entertainment applications, the reliability of the retrieval system, the user programs, through an adaptive user system need not be perfect, and errors may be tolerable. On interface according to the present invention, the processing the other hand, in industrial control applications, reliability of data, by defining a criteria and the actions to be taken must be much higher, with fail-safe backup systems in place, based on the determination of the criteria. The criteria, it is as well as advanced error checking. One way to address this noted, need not be of a predefined type, and in fact this is a issue is to allow the advanced user interface to propose an particular feature of the present invention. A pattern recog action to the user, without actually implementing the action. nition Subsystem is employed to determine the existence of However, in this case, the action and its proposed basis are selected criteria. To facilitate this process, a database of preferably presented to the user in a Sophisticated manner, to image objects may be stored as two counterparts: first, the allow the basis for the action to be independently assessed data is stored in a compressed format optimized for normal by the user. Therefore, in a complex, multistep process, the use, such as human viewing on a video monitor, using, e.g., user interface may be simplified by permitting a three step MPEG-2 or Joint Photographic Experts Group (JPEG) com process: the user triggers a proposed response, analyzes the pression, second, it is stored in a preprocessed and highly proposal and rationale, and confirms the proposal. There compressed format adapted to be used with the pattern fore, single step processes are inferior candidates for intel recognition system. Because the preprocessed data is highly ligent assistance. US 2007/0016476 A1 Jan. 18, 2007

0836 Another notable aspect of the technologies is the 0839 For a given system, status, history of use and contextual analysis. Often, multimedia data often includes a context may be interrelated factors. For example, the status data component that closely corresponds to a format of a of the machine is determined by the prior use, while the search criteria. Thus, while a search may seek a particular status also intersects context. The intended meaning of status image, other portions of the datastream correlate well with is information relating to a path independent state of the the aspect of the image being searched, and may be analyzed machine at a given point in time. History of use is intended by proxy, avoiding the need for full image analysis. The to implicate more than the mere minimum instructions or resulting preselected reduced number of images may then be actions necessary to achieve a given state, and therefore fully analyzed, if necessary. Thus, especially as with respect includes information unnecessary to achieve a given state, to consumer electronics applications, where absolute accu i.e., path dependent information. Context is also related to racy may not be required, the processing power available for status, but rather is differentiated in that context refers to pattern recognition need not be sufficient for compete real information relating to the environment of use, e.g., the time signal analysis of all data. The present invention variable inputs or data upon which the apparatus acts or therefore proposes use of a variety of available data in order responds. Status, on the other hand, is a narrower concept to achieve the desired level functionality at minimum cost. relating more to the internal and constant functionality of the apparatus, rather than the particularities of its use during 0837. One aspect of the present invention therefore specific circumstances. relates to a mechanism for facilitating a user interaction with a programmable device. The interface and method of use of 0840 U.S. Pat. No. 5,187,797 relates to a machine inter the present invention serves to minimize the learning and face system having hierarchical menus, with a simple (three searching times, better reflect users’ expectations, provide button) input scheme. The choice(s) presented relate only to better matching to human memory limits, be usable by both the system status, and not the particular history of use novices and experienced users, reduce intimidation of nov employed to obtain the system status nor the context of the ice users by the device, reduce errors and simplify the choice. This system has a predetermined hierarchical menu entering of programming data. The present invention opti structure, which is invariant with usage. The goal of this mizes the input format scheme for programming an event interface system is not to provide a learning interface, but driven device, and can also be applied to many types of rather to teach the user about or conform the user to the programmable devices. Thus, certain human factors design dictates of the predetermined and invariant interface of the concepts, heretofore unexploited in the design of consumer device. While many types of programmable devices are electronics devices and industrial controls, have been incor known to exist, normally, as provided in U.S. Pat. No. porated, and new precepts developed. Background and 5,187,797, instructions are entered and executed in a pre theory of various aspects of the present invention is dis determined sequence, with set branch points based on input closed in AN IMPROVED HUMAN FACTORED INTER conditions or the environment. See also U.S. Pat. Nos. FACE FOR PROGRAMMABLE DEVICES: A CASE 4,878,179, 5,124,908, and 5,247,433. STUDY OF THE VCR, Master's Thesis, Tufts University 0841. An aspect of the present invention provides a (Master of Sciences in Engineering Design, November, device having a predetermined or a generic style interface 1990, publicly available January, 1991), by Linda I. Hoff upon initial presentation to the user, with an adaptive pro berg. This thesis, and cited references, are incorporated gression in which specialized features become more easily herein by reference, and attached hereto as an appendix. available to a user who will likely be able to make use of Also referenced are: Hoffberg, Linda I., “Designing User them, while unused features are or remain “buried within Interface Guidelines For Time-Shift Programming of a the interface. The interface also extracts behavioral infor Video Cassette Recorder (VCR), Proc. Human Factors Soc. mation from the user and to alter the interface elements to 35th Ann. Mtg. pp. 501-504 (1991); and Hoffberg, Linda I., optimize the efficiency of the user. “Designing a Programmable Interface for a Video Cassette Recorder (VCR) to Meet a User's Needs”, Interface 91 pp. 0842) A videocassette recorder is a ubiquitous example of 346-351 (1991). See also, U.S. patent application Ser. No. a programmable device, and therefore forms the basis of 07/812,805, filed Dec. 23, 1991, incorporated herein by much of the discussion herein. It should, of course, be reference in its entirety, including appendices and incorpo realized that many of the aspects of the present invention rated references. could be applied by one of ordinary skill in the art to a variety of controls having human interfaces, and that these 0838. The present invention extends beyond simple pre other applications are included within the scope of the dictive Schemes which present exclusively a most recently present invention. executed command or most recently opened files. Thus, the possible choices are weighted in a multifactorial method, 0843. The VCR apparatus typically involves a remote e.g., history of use, context and system status, rather than a control entry device, and the interface of the present inven single simple criterion alone. Known simple predictive tion contains a graphical interface displayed for program criteria often exclude choices not previously selected, rather ming programmable devices. This aspect of the present than weighing these choices in context with those which invention seeks more accurate programming through the use have been previously selected. While the system according of program verification to ensure that the input program is to the present invention may include initial weightings, both valid and executable. Thus, it has a mechanism to store logical preferences or default settings, through use, the and check to verify that there are no conflicting programs. derived weightings are obtained adaptively based on an An apparatus according to the present invention can be analysis of the status, history of use and context. It is noted connected, for example, to any infrared programmable that not all of the possible choices need be weighted, but device in order to simplify the programming process. By rather merely a subset thereof. way of example only, an improved VCR interface forms the US 2007/0016476 A1 Jan. 18, 2007 34 basis of a disclosed example. It is, of course, realized that the with an audio data stream or within a video frame, which present method and apparatus may be applied to any pro may be in analog format or digitally encoded. grammable controller, i.e., any device which monitors an 0847 Cable systems may transmit information either in event or sensor and causes an event when certain conditions the broadcast band or in a separate band. HDTV schemes or parameters are met, and may also be used in other also generally provide for the transmission of digital data of programming environments, which are not event driven. various sorts. Thus, known audio and video transmission While the present interface is preferably learning and adap systems may be used, with little or no modifications to tive, it may also detect events and make decisions based on provide enhanced functionality, according to the present known or predetermined characteristics. Where a number of invention. It is therefore possible to use known and available criteria are evaluated for making a decision, conflicts among facilities for transmitting additional information relating to the various criteria are resolved based on a strength of an the broadcast information, in particular, the characteristics evaluated criteria, a weighting of the criteria, an interactivity of the video broadcast, and doing so could provide signifi function relating the various criteria, a user preference, cant advantages, used in conjunction with the interface and either explicitly or implicitly determined, and a contextual intelligent pattern recognition controller of the present analysis. Thus, a user override or preference input may be invention. If this information were directly available, there provided to assist in resolving conflicts. would be a significantly reduced need for advanced image 0844. The present invention may incorporate an intelli recognition functions, such advanced image recognition gent program recognition and characterization system, mak functions requiring costly hardware devices, while still ing use of any of the available cues, which allows an maintaining the advantages of the present invention. intelligent determination of the true nature of the broadcast 0848. It is noted, however, that the implementation of a and therefore is able to make a determination of whether system in which characterization data of the broadcast is parameters should be deemed met even with an inexact transmitted along therewith might require a new set of match to the specified parameters. Therefore, in contradis standards and the cooperation of broadcasters, as well as tinction with VPV, the present invention provides, for possibly the government regulatory and approval agencies. example, intelligence. The VPV is much more like the The present invention does not require, in all of its aspects, “VCR Plus' device, known to those skilled in the art, which Such standardization, and therefore may advantageously requires that a broadcast be associated with a predetermined implement Substantial data processing locally to the code, with the predetermined code used as a criteria for receiver. It is nevertheless within the scope of the invention initiating recording. Some problems with VCR Plus include to implement such a broadcast system with broadcast of identification of the codes which identify channel and time, characterization data in accordance with the present inven post scheduling changes, incorrect VCR clock setting, and tion. Such broadcast characterization data may include char irregular schedules. VCR Plus also is limiting with respect acterizations as well as preprocessed data useful for char to new technologies and cable boxes. acterizing according to flexible criteria in the local receiving 0845 The videotext signal of the prior art includes a device. digitally encoded text message that may be displayed in 0849. According to the present invention, if such char conjunction with the displayed image, similar to the closed acterizations are broadcast, they may, as stated above, be in caption system. The aforementioned West German system band or out of band, e.g., making use of unused available demonstrates one way in which the transmitted signal may spectrum bandwidth within the NTSC channel space, or be received by a device and interpreted to provide useful other broadcast system channel space, or may be "simulcast' information other than the transmitted program itself. How on a separate channel. Such as an FM sideband or separate ever, the prior art does not disclose how this signal may be transmission channel. Use of a separate channel would allow used to index and catalog the contents of a tape, nor does it a separate organization, other than the network broadcasters, disclose how this signal may be used to classify or interpret to provide the characterization data for distribution to users the character of the broadcast. In other words, in one of devices that make use of the present intelligent system for embodiment of the present invention, the videotext or closed controlling a VCR or other broadcast information processing caption signal is not only interpreted as a literal label, as in device. Thus, the characterization generating means need the prior art, but is also further processed and analyzed to not be directly linked to the local user machine in order to yield data about the content of the broadcast, other than fall within the scope of the present invention. The present merely an explicit identification of the simultaneously invention also provides a mechanism for copyright holders broadcast information. or other proprietary interests to be protected, by limiting 0846 Beyond or outside the visible region of an U.S. access to information be encryption or selective encryption, National Television Standards Committee (NTSC) broad and providing an accounting system for determining and cast video frame are a number of scan lines which are tracking license or broadcast fees. dedicated to presenting digital information, rather than ana 0850 Research has been performed relating to VCR log picture information. Various known coding schemes are usability, technology, implementation, programming steps, available for transmitting and receiving information in this current technology, input devices, and human mental capac non-viewing portion of the video transmission, and indeed ity. This research has resulted in a new paradigm for the standard exist defining the content of these information entry of programming data into a sequential program execu fields. Of course, various other transmission Schemes pro vide a format for transmitting data. For example, standard tion device, such as a VCR, by casual users. frequency modulation (FM) transmissions may be associ 0851) Four major problems in the interfaces of VCRs ated with digital data transmissions in a Subcarrier. Like were found to exist. The first is that users spend far too much wise, satellite transmissions may include digital data along time searching for necessary information, which is neces US 2007/0016476 A1 Jan. 18, 2007

sary in order to complete the programming process. Second, data separated from the commercial users of such data. Thus, many people do not program the VCR to record at a later for example, the particular users of a system may register time (time-shift) frequently, and thus forget the program their biometric characteristics, e.g., face. Thereafter, the ming steps in the interim, i.e., the inter-session decay of the imager captures facial images and correlates these with its learning curve is significant. Third, the number of buttons on internal database. The image itself therefore need not be many remote control devices has become overwhelming. stored or transmitted. Viewer preferences and habits, on the Fourth, people have become reluctant to operate or program other hand, likely must be transmitted to a central processing VCRs because of their difficult operation. It was found that, system for analysis. by minimizing the learning and searching times, the user's 0856. Because the system is intelligent, copy protection programming time and frustration level can be greatly and royalty accounting schemes may readily be imple reduced. If VCRs are easier to program, users might pro mented. Thus, broadcasters and content providers may gram them more frequently. This would allow more effi encode broadcasts in Such a way as to control the operation ciency and flexibility in broadcast scheduling, especially late of the consumer device. For example, an IEEE-1394-type night for time shift viewing. The present invention therefore encryption key support/copy protection or DIVX Scheme provides an enhanced VCR programming interface having a may be implemented. Further, certain commercial sponsors simplified information structure, an intuitive operational may be able to avoid deletion of their advertisement, while structure, simplified control layout and enhanced automated others may allow truncation. The acceptability of this to the functionality. consumer may depend on Subsidies. In other words, a 0852. A new class of consumer device has been pro company is willing to pay for advertising. Instead of paying posed, which replaces the videotape of a traditional video for placements directly to the media, a portion is paid to a tape recorder with a random-access storage device, such as service provider, based on consumer viewing. The media, on a magnetic hard disk drive. Multimedia data is converted the other hand, may seek to adopt a pay-per-view policy, at through a codec (if necessary), and stored in digital form. least with respect to the service provider, in lieu of direct Such systems are proposed by Tivo, Inc., Philips Electronics advertising revenues. The service provider will account to (Personal TV), Replay Networks, Inc. and Metabyte, Inc. both advertisers and content providers for use. With suffi Some of these systems employ a user preference based cient viewing of commercials, the entire service charge for programming/recording method similar to that of the present a system might be covered for a user. On the other hand, a invention. viewer might prefer to avoid all commercials, and not get the benefit of a subsidy. The service provider performs the 0853. In these systems, typically a content descriptive economically efficient function of delivering optimized, data stream formulated by human editors accompanies the Substituted commercials for the almost random commercials broadcast or is available for processing and analysis. Based which flood the commercial broadcast networks, and thus on a relation of the user preferences, which may be implied can accrue greater profits, even after paying content provid by actual viewing habits or input through simple accept/veto ers a reasonable fee. An advertiser, by selecting a particular user feedback, selected media events may be recorded. audience, may pay less than it would otherwise pay to a However, Such systems rely on a correspondence between broadcaster. The content providers may also charge more for the factors of interest to users and those encoded in the data the privilege of use of their works. stream, e.g., a program guide'. This is not always the case. 0857. As stated above, the content may be copy protected However, where the available data describing the program by the use of encryption and/or lockout mechanisms. Thus, maps reasonably well into the user preference space. Such a by providing an alternative to an analog VCR, a full end system may achieve acceptable levels of performance, or to-end encrypted signal may be provided, such as that stated otherwise, the program material selected by the sys proposed for the IEEE-1394 copy protection scheme. tem will be considered acceptable. Because enhanced recording capabilities are provided to the 0854. One particular aspect of these time-shifting con consumer, the acceptance will be high. Because of the Sumer media recording devices is how they deal with encryption, lack of port ability and continued royalty advertising materials which accompany program material. accounting, content provider acceptance will also likely be In many instances, the user seeks to avoid "commercials'. high. and the device may be programmed to oblige. However, as 0858. The user interface concepts according to the Such devices gain wider acceptance, advertisers will be present invention are easily applied to other special purpose reluctant to Subsidize broadcasts. Therefore, an advertising programmable devices, and also to general purpose pro system may be integrated into the playback device which grammable devices wherein the programming paradigm is seeks to optimize the commercial messages presented to a event-driven, as well as other programming systems. It viewer. By optimizing the messages or advertisements, the should also be noted that it is within the scope of the present viewer is more receptive to the message, and economic invention to provide an improved interface and program implications ensue. For example, a viewer may be compen ming environment for all types of programmable devices, sated, directly or indirectly, for viewing the commercials, and in this regard, the present invention incorporates adap which may be closely monitored and audited. Such as by tive features which optimize the programming environment taking pictures of the audience in front of a “set-top box”. for both the level of the user and the task to be programmed. The acquired data, including viewer preferences, may be transmitted back to commercial sponsors, allowing detailed 0859. In optimizing the interface, four elements are par demographic analysis. ticularly important: the input device, the display format, the sequence of the programming operation, and the ability of 0855. In order to ensure privacy, the preference informa the device to properly interpret the input as the desired tion and/or images may be analyzed by a proxy, with the raw program Sequence. US 2007/0016476 A1 Jan. 18, 2007 36

0860. The present invention proceeds from an under 0865. In formulating a group preference, individual dis standing that an absence of user frustration with respect to likes may be weighted more heavily than likes, so that the a programmable consumer or industrial device or interface, resulting selection is tolerable by all and preferable to most may be particularly important with respect to achieving the group members. Thus, instead of a best match to a single maximum potential functionality thereof. The interface must preference profile for a single user, a group system provides be designed to minimize the users frustration level. This can a most acceptable match for the group. It is noted that this be accomplished by clearly furnishing the possible choices, method is preferably used in groups of limited size, where presenting the data in a logical sequence, and leading the individual preference profiles may be obtained, in circum user through the steps necessary to program the device. stances where the group will interact with the device a 0861. When applied to other than audiovisual and/or number of times, and where the Subject source program multimedia application, the pattern recognition function material is the Subject of preferences. Where large groups may be used to control the execution of a program or are present, demographic profiles may be employed, rather selectively control execution of portions of the software. For than individual preferences. Where the device is used a example, in a programmable temperature controller appli small number of times by the group or members thereof, the cation, a sensor or sensor array could be arranged to detect training time may be very significant and weigh against a "door opening. On the occurrence of the door opening, automation of selection. Where the source material has little the system would recognize this pattern, i.e. a mass of air at variety, or is not the Subject of strong preferences, the a different temperature entering the environment from a predictive power of the device as to a desired selection is single location, or a loss of climate controlled air through a limited. single location. In either event, the system would take 0866 The present invention provides a system and appropriate action, including: halt of normal climate control method for making use of the available broadcast media and impose a delay until the door is closed; after closure, set forms for improving an efficiency of matching commercial a time constant for maintenance of a steady state of the information to the desires and interests of a recipient, replaced air with the climate controlled air, based on the improving a cost effectiveness for advertisers, improving a actual climatic condition after assimilation, or a predicted perceived quality of commercial information received by climatic condition after assimilation, begin a climate com recipients and increasing profits and reducing required infor pensation control, optionally, during the door opening, con mation transmittal by publishers and media distribution trol a pressure or flow of air to counterbalance the normal entities. flow through the door, by using a fan or other device. The climate may differ in temperature, humidity, pollutants, or 0867. This improved advertising efficiency is accom the like, and appropriate sensors may be employed. plished by providing a system for collating a constant or 0862 The present invention also allows a dynamic user underlying published content work with a varying, demo preference profile determination based on explicit or implicit graphically or otherwise optimized commercial information desires, e.g., moods, which assist in processing data to make content. This commercial information content therefore decisions which conform to the user preference at a given need not be predetermined or even known to the publisher point in time. For example, Voice patterns, skin temperature, of the underlying works, and in fact may be determined on heat pulse rate, external context, skin resistance (galvanic an individual receiver basis. It is also possible to integrate skin response), blood pressure, stress, as determined by the demographically optimized information within the con EMG, EEG or other known methods, spontaneous motor tent. For example, overlays in traditional media, and elec activity or twitching, may be detected in order to determine tronic Substitutions or edits in new media, may allow seam or infer a user mood, which may be used as a dynamic less integration. The content alteration need not be only influence on the user preference. These dynamic influences based on commercial information, and therefore the content are preferably stored separately from static influences of the may vary based on the user or recipient. preferences, so that a resultant determined preference 0868 U.S. Pat. No. 5,469,206, expressly incorporated includes a dynamic influence based on a determined mood herein by reference, relates to a system that automatically or other temporally varying factor and a static influence correlates user preferences with electronic shopping infor associated with the user. mation to create a customized database for the user. 0863. When a group of people are using the system 0869. Therefore, the granularity of demographic market simultaneously, the system must make a determination of a ing may be very fine, on a receiver-by-receiver basis. composite preference of the group. In this case, the prefer Further, the accounting for advertisers will be more accurate, ences of the individuals of the group, if known, may be with a large sample and high quality information. In fact, in correlated to produce an acceptable compromise. Where a further embodiment, an interactive medium may be used individual preferences are not a priori known, individual or allowing immediate or real time communication between group “interviews” may be initially conducted to assist in recipient and advertiser. This communication may involve determining the best composite group preference. the Internet, private networks or dial-up connections. 0864. It is therefore an object according to the present Because the commercial messages are particularly directed invention to provide a radio receiver or video receiver to recipients, communication with each selected recipient is device, having a plurality of different available program more valuable to an advertiser and that advertiser is willing Sources, determining a program preference for one or more to pay more for communication with each selected recipient. individuals subject to a presented program, comparing the Recipients may therefore be selected to receive the highest determined program preference and a plurality of different valued appropriate commercial message(s). Thus, advertis program sources, and selects at least one program based on ers will tend to pay less and media producers will gain more the comparison. revenues. Recipients will gain the benefit of selected and US 2007/0016476 A1 Jan. 18, 2007 37 appropriate media, and further, may provide feedback for history of use of the device by the recipient(s), the context determining their preferences, which will likely correspond of use, the arrangements made by the commercial informa with their purchasing habits. Thus, the recipient will benefit tion provider(s) for presentation of information, and the by receiving optimized information. availability of information for presentation. Other factors may include the above-mentioned accounting system fac 0870. Likewise, a recipient may place a value on receiv tors. Typically, the local device will include a store of ing certain information, which forms the basis for "pay-per commercial information, downloaded or otherwise transmit view systems. In this case, the recipient’s values may also ted to the recipient (e.g., a CD-ROM or DVD with MPEG-2 be considered in defining the programming. compressed images). A telecommunication link may also be 0871. This optimization is achieved by providing a provided to control the process, provide parameters for the device local to the recipient which selectively presents presentation or the information itself. This telecommunica commercial information to the recipient based on charac tion link may be provided through the public telephone teristics individual to the recipient, which may be input by network, Internet, private network (real or virtual) cable the recipient, the publisher, the advertiser, and/or learned by network, or a wireless network, for example. Generally, the the system based on explicit or implicit feedback. The local underlying work will have a gap of fixed length, so that the device either has a local memory for advertising materials, commercial information must be selected to fit in this gap. or a telereception link for receiving commercial information Where the gap is of variable length, such as might occur in for presentation, either on a real time basis or stored for later live coverage, the commercial information is interrupted or presentation. In a further embodiment, a user may control the underlying work buffered and delayed to prevent loss. the content and/or commercial information received. In this Thus, the presentation to the user is constructed from pieces, case, the accounting system involves the user's account, typically at the time of presentation, and may include and, for example, the recipient may be denied the Subsidy invariable content, variable content, invariable messages, from the commercial advertiser, and pay for the privilege of variable messages, targeted content and/or messages, and commercial free content. hypervariable content. Hypervariable content includes, for 0872. It is also possible to employ the methods and example, transition material selected based on the stream of systems according to the present invention to create a information present, and other presentations which my customized publication, which may be delivered physically optionally include useful information which are individual to the recipient, for example as print media, facsimile ized for the particular recipient or situation. transmission, e-mail, R-CD-ROM, floppy disk, or the like, 0877 According to another embodiment, a recording, without having a device local to the consumer. Such as on a videotape, is retained by a recipient which 0873. It is noted that this system and method is usable for includes proprietary content. This may include a commercial both real time media, Such as television, radio and on-line broadcast, a private broadcast, or distributed media. In the telecommunication, as well as manually distributed periodi case of a commercial broadcast, some or all of the commer cals, such as newspapers, magazines, CD-ROMs, diskettes, cial advertising or other time-sensitive information is old etc. Therefore, the system and method according to the and/or stale. Therefore, in operation, this old or time sensi present invention includes a set of related Systems with tive information is eliminated and substituted with new varying details of implementation, with the underlying char and/or different information. Thus, the presentation system acteristic of optimization of variable material presentation at freshens the presentation, editing and Substituting where the recipient level rather than the publisher level. necessary. 0874) The system and method according to the present 0878. By such a method, content distributed even through invention preferably includes an accounting system which private channels may include advertisements, and thus be communicates information relating to receipt of commercial subsidized by advertisers. The advertisements and other advertising information by a recipient to a central system for added content are generally more acceptable to the audience determination of actual receipt of information. This feed because they are appropriately targeted. back system allows verification of receipt and reduces the 0879 For example, where the broadcaster has a high possibility of fraud or demographic inaccuracies. degree of control over the initial broadcast, e.g., pay per 0875. The accounting system, for example, may place view under license, or where the broadcaster may claim value on the timeslot, associated content, the demographics Substantial continuing rights in the work after recording, the of the user, users associated valuation, competition for enforcement of a proprietary replay system may be accepted. placement, past history (number of impressions made to For example, a work is broadcast as an encrypted digital data same recipient) and exclusivity. stream, with selective decryption at the recipient's receiver, under license from the broadcaster. In this case, a recording 0876. A preferred embodiment includes a subscription system is provided which retains the encryption character television system having a plurality of received channels. At istics, ensuring the integrity of the accounting process. least one of these channels is associated with codes to allow During presentation of the recorded work, commercial infor determination of content from variable segments. It is also mation is appropriately presented to the recipient during possible to identify these variable segments without these existing or created gaps, or in an associated output separate codes, although the preferred system includes use of Such from the content presentation. The recipient, as a result, codes. These codes also allow simple identification of the receives the benefit of the original subsidy, or may receive content for accounting purposes. Upon detection of a vari a new Subsidy. able segment, a commercial advertisement is selected for presentation to the recipient. This variable segment is 0880. Therefore, similar to the known DIVX system, an selected based on the characteristics of the recipient(s), the encrypted media may be mass distributed, which requires US 2007/0016476 A1 Jan. 18, 2007

authorization for display. Instead, however, of requiring the device selects program material that assists in a desired recipient to pay for the initial and Subsequent displays of the mood transition. The operation of the device may addition content, the player integrates advertising content into the ally acquire data relating to an individual and the respective output, which may vary based on the audience, time and past moods, desires and characteristics, altering the path pro history, as well as other factors discussed herein. Given the vided to alter the mood based on the data relating to the interactive and variable nature of the presentation, the user individual. As stated above, in a group setting, a most or audience may even veto (“fast forward through’) a acceptable path is presented rather than a most desirable particular commercial. In this case, the use may have to path as presented for an individual. account for a fee, or other advertisers may tack up the slack. 0886. In determining mood, a number of physiologic The veto provides information regarding the desires of the parameters may be detected. In a training circumstance, viewer, and may be used to help select future messages to these set of parameters are correlated with a temporally the displayed or presented. associated preference. Thus, when a user inputs a preference 0881. According to another embodiment, a radio trans into the system as feedback, mood data is also obtained. mission/reception system is provided which broadcasts con Invariant preferences may be separated, and analyzed glo tent, an overlay track and variable commercial information. bally, without regard for temporal variations, while varying The invariant works are preferably prerecorded music. The preferences are linked with information regarding the Sur overlay track is preferably a "DJ, who provides information rounding circumstances and stored. For example, the pref regarding the invariant works, commercial information or erence data may be used to train a neural network, e.g., using news. The commercial information in this instance therefore backpropagation of errors or other known methods. The refers to prerecorded segments. In this instance, the goal is inputs to the neural network include available data about to allow the invariant works to be received by the recipient Surrounding context, such as time, environmental bright and presented with improved optimization of the commer ness, and persons present, source program choices, which cial information content and other messages presented at the may be raw data, preprocessed data, and abstracted data; time of output. Further, this system allows optimization of explicit user input, and, in this embodiment, mood param the presentation of the invariant portions as well, i.e., the eters, which may be physiological or biometric data, Voice commercial information and the program content may be pattern, or implicit inputs. An example of an implicit input independently selected at the receiver, with appropriate is an observation of a man-machine interaction, Such as a accounting for commercial Subsidy. In a mobile receiver, it Video game. The manner in which a person plays a video is preferable to include as a factor in the selection of game or otherwise interacts with a machine may provide commercial information a location of the receiver, as might valuable data for determining a mood or preference. be obtained from a GPS system, cellular location system, 0887. According to one embodiment of the invention, the intelligent highway system or the like. This would allow image is preprocessed to decompose the image into object geographically appropriate selection of commercial infor elements, with various object-elements undergoing separate mation, and possibly overlay information as well, e.g., traffic further processing. For example, certain backgrounds may reports. be aesthetically modeled using simple fractal equations. 0882 Another embodiment according to the present While, in Such circumstances the results may be inaccurate invention provides a hypertext linked media or multimedia in an absolute sense, they may be adequate in a performance environment, such as HTML/World Wide Web, wherein sense. Faces, on the other hand, have common and variable information transmitted and/or displayed is adaptively elements. Therefore, a facial model may be based on param selected based on the particular user or the user's receiving eters having distinguishing power, Such as width between system. Thus, various elements may be dynamically substi eyes, mouth, shape of ears, and other proportions and tuted during use. dimensions. Thus, along with color and other data, a facial 0883. Therefore, it is an object according to the present image may be stored as a reference to a facial model with the invention to provide adaptive man-machine interfaces, espe distinguishing parameters for reconstruction. Such a data cially computer graphic user interfaces, which are ergo processing scheme may produce a Superior reconstructed nomically improved to provide an optimized environment. image and allow for later recognition of the face, based on Productivity of computer operators is limited by the time the stored parameters in reference to the model. Likewise, necessary to communicate a desired action through the user many different elements of an image may be extracted and interface to the device. To reduce this limitation, most likely processed in accordance with specific models to produce user actions are predicted and presented as easily available differentiating parameters, wherein the data is stored as a options. The technologies also extend beyond this core reference to the particular model along with the particular theme in many differing ways, depending on the particular data set derived from the image. Such a processing scheme application. allows efficient image storage along with ease of object recognition, i.e., distinction between objects of the same 0884 The system also provides an intelligent, adaptive class. This preprocessing provides a highly asymmetric pattern recognition function in order to provide the operator scheme, with a far greater processing complexity to initially with a small number of high probability choices, which may process the image than to Subsequently reconstruct or oth be complex, without the need for explicit definition of each erwise later employ the data. atomic instruction comprising the desired action. The inter 0888. By employing a model-based object extraction face system predicts a desired action based on the user input, system, the available bandwidth may be efficiently used, so a past history of use, and a context of use. that objects which fall within the scope of an available 0885. In yet another embodiment, a present mood of a model may be identified with a model identification and a user is determined, either explicitly or implicitly, and the series of parameters, and objects not within the scope of a US 2007/0016476 A1 Jan. 18, 2007 39 model may be allocated a comparatively greater bandwidth relating to the activity of the user and derived weighing of for general image description, e.g., JPEG, MPEG-1/MPEG at least a Subset of possible choices, the derivation being 2, wavelet, standard fractal image compression (FIC), or based on a history of use, a context of a respective choice other image processing schemes. In a worst case, therefore, and the status of the control; and a user feedback data the bandwidth required will be only slightly greater than that presenting system comprising an output device for presen required for a corresponding standard method, due only to tation of a variable sequence of programming options to the the additional overhead to define data types, as necessary. user, including the most probable intended action of the user, However, by employing a model based-object decomposi in a plurality of output messages, the output messages tion processing system, recognized elements may be differing in available programming options. described using only a small amount of data and a greater 0893. The programmable control may be employed for proportion of data used to describe unrecognized elements. performing an action based on user input and an information Further, the models available may be dynamically updated, content of a signal received from a signal Source, wherein so that, as between a communicating transmitted and the output device includes a display device, further com receiver, retransmission of unrecognized elements will be prising a user controlled direct manipulation-type input eliminated as a model is constructed. device, associated with the display device, having a device 0889 Where image processing systems may produce output, the device output being the user input; a plant artifacts and errors, an error minimization function may also capable of performing the action, being responsive to an be provided which compares an original image with a actuator signal; and the controller, being for receiving data decomposed-recomposed image and produces an error func from the device output of the input device and the signal, and tion which allows correction for these errors. This error displaying user feedback data on the display device, the function may be transmitted with the processed data to allow logical sequence of the user feedback data including at least more faithful reproduction. In a pattern recognition context, one sequence of options Sufficient to define an operable the error function may provide useful data relating to the control program, and a presentation of additional program reliability of a pattern correlation, or may provide useful ming options if the control program is not operable. data outside of the model and associated parameters for 0894 The programmable control may further comprise a pattern recognition. user input processing system for adaptively determining a 0890 Thus, in the case of an object-extraction model viewer preference based on the user input received by the based processing System, the resulting data stream may be controller, a program material processing system for char appropriate for both viewing and recognition. Of course, acterizing the program material based on its content, a acoustic data may be likewise processed using acoustic correlator for correlating the characterized content of the models with variable parameters. However, in Such a sys program material with the determined viewer preference to tem, information for pattern recognition may be filtered, produce a correlation index; and a processor, selectively Such as eliminating the error function or noise data. Further, processing the program material based on the correlation certain types of objects may be ignored, for example, under index, the data processing system receiving an input from normal circumstances, clouds in the sky provide little infor the processor. mation for pattern recognition and may be removed. In Such 0895. The programmable control may also comprise a a system, data intended for viewing or listening will likely plurality of stored profiles, a processor for characterizing the contain all objects in the original data stream, with as much user input to produce a characterized user input; and means original detail as possible given data storage and bandwidth for comparing the characterized user input with at least one constraints. of the plurality of stored profiles to produce a comparison 0891. An object extraction model based processing sys index, wherein the variable sequence of programming tem also allows for increased noise rejection, Such as over options is determined on the basis of the comparison index. terrestrial broadcast channels. By transmitting a model, the The processor for characterizing may perform an algorithm receiving system may interpolate or extrapolate data to fill in on the signal comprising a transform selected from the group for missing data. By extrapolate, it is meant that past data is consisting of an Affine transformation, a Fourier transfor processed to predict a Subsequent condition. By interpolate, mation, a discrete cosine transformation and a wavelet it is meant that data presentation is delayed, and missing data transformation. may therefore be predicted from both past and subsequent 0896. It is a further object according to the present data transmission. Missing portions of images may also be invention to provide a programmable controller for control reconstructed from existing portions. This reconstruction ling a recording device for recording an analog signal process is similar to that described in U.S. Pat. No. 5,247. sequentially on a recording medium having a plurality of 363, to reconstruct MPEG images, except that where model uniquely identifiable storage locations, further comprising a data is corrupted, the corruption must be identified and the sequential recording device for recording the analog signal, corrupt data eliminated and replaced with predicted data. and a memory for storing, in a directory location on the 0892. It is therefore an object according to the present recording medium which is separate from the storage loca invention to provide a programmable control, having a tion of the analog signal, information relating to the signal, status, responsive to an user input and a signal received from processed to selectively retain characterizing information, a signal source, comprising a controller, for receiving the and an identifier of a storage location on the recording user input and the signal and producing a control output, a medium in which the analog signal is recorded. memory for storing data relating to an activity of the user; 0897. It is another object according to the present inven a data processing system for adaptively predicting a most tion to provide a control, wherein program material is probable intended action of the user based on the stored data encrypted, further comprising a decryption system for US 2007/0016476 A1 Jan. 18, 2007 40 decrypting the program material if it is selected to produce two dimensions of display, based on the visual feedback unencrypted program material and optionally an associated received from the display device, the indicator segment decryption event, a memory for storing data relating to the being moved to a translated location of the display device occurrence of the decryption event, and a central database corresponding to a user command; and a controller for for storing data relating to the occurrence of the decryption controlling the graphical user interface and for producing the event in association with data relating to the viewer. input of the selector, receiving as a control the output of the multidimensional input device, the controller receiving the 0898. It is still another object according to the present output of the program database and presenting information invention to provide a control wherein the user input pro relating to at least one of the plurality of programs on the cessing system monitors a pattern of user activity and display device associated with a command, the command predicts a viewer preference; the program material process being interpreted by the control means as the user command ing system comprising a processor for preprocessing the to produce the input of the selector to select the at least one program material to produce a reduced data flow informa tion signal Substantially retaining information relating to the of the plurality of programs associated with the command. abstract information content of the program material and 0902 Another object of the present invention is to pro selectively eliminating data not relating to the abstract vide an apparatus, receiving as an input from a human user information content of the program material and for char having a user characteristic, comprising an input device, acterizing the information signal based on the abstract producing an input signal from the human user input; a information content, and a comparing system for determin display for displaying information relating to the input from ing if the correlation index is indicative of a probable high the user and feedback on a current state of the apparatus, correlation between the characterization of the information having an alterable image type, an input processor for signal and the viewer preference and causing the stored extracting an input instruction relating to a desired change in program material to be processed by the processing means a state of the apparatus from the input signal; a detector for based on the determination. The system according to this detecting one or more temporal-spatial user characteristics aspect of the present invention preferably comprises an of the input signal, independent of the input instruction, selected from the group consisting of a Velocity component, image program material storage and retrieval system. an efficiency of input, an accuracy of input, an interruption 0899) The present invention further provides a control of input and a high frequency component of input, a memory further comprising a memory for storing a characterization for storing data related to the user characteristics, and a of the program material; an input for receiving a feedback controller for altering the image type based on the user signal from the viewer indicating a degree of agreement with characteristics. The controller may alter the image type the correlation index determination, wherein the feedback based on an output of the detector and the stored data so that signal and the stored characterization are used by the viewer the display displays an image type which corresponds to the preference predicting means to predict a new viewer pref detected user characteristics. The controller may further be CCC. for controlling the causation of an action on the occurrence of an event, further comprising a control for receiving the 0900. According to another aspect of the invention, it is input instruction and storing a program instruction associ an object to provide an image information retrieval appara ated with the input instruction, the control having a memory tus, comprising a memory for storing compressed data Sufficient for storing program instructions to perform an representing a plurality of images, a data storage system for action on the occurrence of an event; and a monitor for retrieving compressed data representing at least one of the monitoring an environment of the apparatus to determine the plurality of images and having an output, a memory for occurrence of the event, and causing the performance of the storing characterization data representing a plurality of action on the occurrence of the event. The controller may image types, having an output, and an image processor, also alter the image type based on an output of the detector receiving as inputs the outputs from the data storage system and the stored data so that the display means displays an and the characterization data memory, and producing a image type which corresponds to the detected user charac signal corresponding to a relation between at least one of the teristics. plurality of images of the compressed data and at least one 0903. It is another object of the present invention to of the image types of the characterization data. provide an adaptive programmable apparatus having a plu 0901. It is a still further aspect of the present invention to rality of states, being programmable by a programmer and provide a video interface device for a user comprising a data operating in an environment in which a plurality of possible transmission system for simultaneously transmitting data events occur, each of the events being associated with representing a plurality of programs; a selector for selecting different data, comprising an data input for receiving data; at least one of the plurality of programs, being responsive to an programmer input, producing an input signal from the an input, a program database containing information relating programmer; a memory for storing data relating to the data to the plurality of programs, having an output, a graphical input or the input signal; a feedback device for adaptively user interface for defining commands, comprising (a) an providing information relating to the input signal and a image display device having at least two dimensions of current status of the apparatus to the programmer, based on display, being for providing visual image feedback; and (b) the data input or the programmer input, the stored data, and a multidimensional input device having at least two dimen derived weighing of at least a Subset of possible choices, the sions of operability, adapted to correspond to the two derived weighing being based on a history of use, a context dimensions of the display device, and having an output, so of a respective choice and the current status of the apparatus; that the user may cause the input device to produce a a memory for storing programming data associated with the corresponding change in an image of the display device by input signal; and a processor, having a control output, for translating an indicator segment of the display in the at least controlling the response of the apparatus relating to the US 2007/0016476 A1 Jan. 18, 2007

detection of the input signal or the data in accordance with 0907. The processor of the programmable apparatus may the stored programming data, the processor: (a) processing receive information from the input signal and/or from the the at least one of the input signal or the data to reduce an data input; and may further comprise an input signal amount of information while Substantially retaining an memory for storing at least a portion of the input signal or abstract portion of the information, (b) storing a quantity of the data, a profile generator for selectively generating a the abstracted information, (c) processing the abstract por profile of the input signal or the data, and an input signal tion of the information in conjunction with the stored profile memory for storing the profile of the input signal or quantity of abstracted information; and (d) providing the the data separately from the input signal or the data in the control output based on the processed abstract portion of the input signal memory. The programmable apparatus may information and the stored programming data. The apparatus further comprise a processor for comparing the input signal may further comprise an input for receiving a programming or the data with the stored profile of the input signal or the preference from the programmer indicating a plurality of data to determine the occurrence of an event, and the data possible desired events, the processor further including a optionally comprises image data and the processor for correlator for correlating the programming preference with comparing performs image analysis. The image data may the databased on an adaptive algorithm and for determining comprise data having three associated dimensions obtained a likelihood of occurrence of at least one of the desired by a method selected from the group consisting of synthe events, producing the control output. The apparatus may sizing a three dimensional representation based on a further comprise an input for receiving feedback from the machine based model derived from two dimensional image programmer indicating a concurrence with the control out data, Synthesizing a three dimensional representation put of the processor, and modifying the response control derived from a time series of pixel images, and synthesizing based on the received feedback to increase a likelihood of a three dimensional representation based on a image data concurrence. The apparatus may still further verify the representing a plurality of parallax views each having at programming data to ensure that the programming data least two dimensions. comprise a complete and consistent set of instructions; and 0908. A user feedback data presenting device according include a feedback system for interactively modifying the to the present invention may comprise a display having a programming data. The apparatus may also comprise a plurality of display images, the display images differing in chronological database and an accessing system for access available programming options. ing the chronological database on the basis of the program 0909. According to another aspect of the present inven ming data stored in the memory. tion, a program material processing System is provided comprising means for storing template data, means for 0904. It is also an object according to the present inven storing the image data; means for generating a plurality of tion to provide an apparatus comprising an input for receiv domains from the stored image data, each of the domains ing a programming preference from the programmer indi representing different portions of the image information, cating a plurality of possible desired events; and a correlator means for creating, from the stored image data, a plurality of for correlating the programming preference with the data addressable mapped ranges corresponding to different Sub based on an adaptive algorithm and for determining a sets of the stored image data, the creating means including likelihood of occurrence of at least one of the desired events, means for executing, for each of the mapped ranges, a producing the output, the output being associated with the procedure upon the one of the Subsets of the stored image initiation of the the response. data which corresponds to the mapped range; means for 0905. The present invention also provides as an object an assigning identifiers to corresponding ones of the mapped apparatus comprising an input for receiving feedback from ranges, each of the identifiers specifying for the correspond the programmer indicating a concurrence with the output of ing mapped range an address of the corresponding Subset of the correlator, and modifying the algorithm based on the stored image data; means for selecting, for each of the received feedback, the feedback device comprising a display domains, the one of the mapped ranges which most closely and the input device is remote from the display, and pro corresponds according to predetermined criteria, means for viding a direct manipulation of display information of the representing at least a portion of the image information as a display. set of the identifiers of the selected mapped ranges; and means for selecting, from the stored templates, a template 0906. According to an aspect of the present invention, a which most closely corresponds to the set of identifiers processor of the programmable apparatus verifies the pro representing the image information. The means for selecting gram instructions to ensure that the program instructions are may comprise means for selecting, for each domain, the valid and executable by the processor; an output for pro mapped range which is the most similar, by a method viding an option, selectable by the programmer input for selected from at least one of the group consisting of selecting changing an instruction stored by the processor, such that the a minimum Hausdorff distance from the domain, selecting apparatus enters a state wherein a new instruction may be the highest cross-correlation with the domain and selecting input to substitute for the instruction, wherein the processor the lowest mean square error of the difference between the verifies the instructions such that the instructions are valid, mapped range and the domain. The means for selecting may and wherein the feedback device further presents informa also comprise, for each domain, the mapped range with the tion requesting confirmation from the programmer of the minimum modified Hausdorff distance calculated as Ddb. instructions associated with the input signal. The apparatus mrb--D1-db.1-mrb), where D is a distance calculated may further comprise a chronological database and an between a pair of sets of data each representative of an accessing system for accessing the chronological database image, db is a domain, mrb is a mapped range, 1-db is the on the basis of the program instructions stored in the inverse of a domain, and 1-mrb is an inverse of a mapped memory. range. The means for representing may further comprise US 2007/0016476 A1 Jan. 18, 2007 42 means for determining a feature of interest of the image data, detecting a need by the programmer for more detailed selecting a mapped range corresponding to the feature of information displayed on the display means, by detecting interest, storing the identifiers of the selected mapped range, one or more characteristics of the input instruction signal selecting a further mapped range corresponding to a portion independent of the program instruction selected from the of image data having a predetermined relationship to the group consisting of a Velocity component, an efficiency of feature of interest and storing the identifiers of the further input, an accuracy of input, an interruption of input, a high mapped range. frequency component of input and a past history of input by 0910 According to an embodiment of the present inven the programmer, whereby when the control means detects a tion, the image data comprises data having three associated characteristic indicating that the display means is insuffi dimensions obtained by a method selected from the group ciently detailed information, the control means controls the consisting of synthesizing a three dimensional representa display means to display more detailed information. tion based on a machine based prediction derived from two 0913. It is a further object of the present invention to dimensional image data, synthesizing a three dimensional provide a programmable apparatus having a data input, the representation derived from a time series of pixel images, apparatus receiving instructions from a programmer and and synthesizing a three dimensional representation based causing an action to occur on the receipt of data indicating on a image data representing a plurality of parallax views an event, comprising an input device, producing an input having at least two dimensions. instruction signal; a control means for receiving the input instruction signal, and storing a program instruction associ 0911. It is therefore an object of the present invention to ated with the input instruction signal, the control means provide a programmable apparatus for receiving instructions storing Sufficient program instructions to perform an action from a programmer and causing an action to occur on the on the receipt of data indicating an event, the control means happening of an event, comprising an input device, produc monitoring the data input; a display means for interactively ing an input instruction signal; a control means for receiving displaying information related to the instructions to be the input instruction signal, and storing a program instruc received, and responsive thereto, controlled by the control tion associated with the input instruction signal, the control means, so that the programmer is presented with feedback means storing Sufficient program instructions to perform an on a current state of the apparatus and the program instruc action on the occurrence of an event, the control means tion, wherein the control means receives a programming monitoring a status of the apparatus to determine the occur preference indicating a desired event from the input device rence of various events, comparing the determined events which does not unambiguously define the event, and the with the program instructions, and performing the action on control means monitors the data and causes the occurrence the occurrence of the event; a display means for interactively of the action when a correlation between the programming displaying information related to the instructions to be preference and the monitored data is above a predetermined received, and responsive thereto, controlled by the control threshold, indicating a likely occurrence of the desired means, so that the programmer is presented with feedback event. It is also object of the present invention to provide the on a current state of the apparatus and the program instruc programmable aforementioned apparatus, wherein the input tion, wherein the control means further comprises means for device is remote from the display means, and provides a detecting one or more characteristics of the input instruction direct manipulation of display information of the display signal independent of the program instruction selected from means, further comprising means for verifying the program the group consisting of a Velocity component, an efficiency instructions so that the program instructions are executable of input, an accuracy of input, an interruption of input, a high by the control means. The control means may further frequency component of input and a past history of input by comprise a calendar or other chronological database. the programmer, whereby when the control means detects a 0914. Another object of the present invention provides a characteristic indicating that the display means is displaying programmable information storage apparatus having a data information in a Suboptimal fashion, the control means input, for receiving data to be stored, the apparatus receiving controls the display means to display information in a more instructions from a programmer and causing an action to optimal fashion. occur on the receipt of data indicating an event, comprising 0912. It is also an object of the present invention to means for storing data from the data input, an input device, provide a programmable apparatus for receiving instructions producing an input instruction signal, a control means for from a programmer and causing an action to occur on the receiving the input instruction signal, and storing a program happening of an event, comprising an input device, produc instruction associated with the input instruction signal, the ing an input instruction signal; a control means for receiving control means storing Sufficient program instructions to the input instruction signal, and storing a program instruc perform an action on the receipt of data from the data input tion associated with the input instruction signal, the control indicating an event, the control means monitoring the data means storing Sufficient program instructions to perform an input to determine the occurrence of various events, com action on the occurrence of an event, the control means paring the determined events with the program instructions, monitoring a status of the apparatus to determine the occur and performing for storing the data the action on the rence of various events, comparing the determined events occurrence of the event, wherein the control means receives with the program instructions, and performing the action on identifying data from at least one of the input device and the the occurrence of the event; a display means for interactively data input, the identifying data being stored separately from displaying information related to the instructions to be the input data on a storage medium. The programmable received, and responsive thereto, controlled by the control information storage apparatus may also include means for means, so that the programmer is presented with feedback reading the identifying data stored separately on the storage on a current state of the apparatus and the program instruc medium, and may also receive as an input the identifying tion, wherein the control means further comprises means for data. US 2007/0016476 A1 Jan. 18, 2007

0915. It is also an object of the present invention to without entering data into the controller through the input provide a programmable apparatus, wherein the control device relating to both the action and the event; the display means provides an option, selectable by the input means in also comprising means for indicating completion of entry of conjunction with the display means, for changing an input a programming step, which means indicates to the user an program instruction prior to execution by the control means, indication that the programming step is not completed if so that the apparatus enters a state wherein a new program information necessary for execution of the step is not instruction may be input to Substitute for the changed input available to the controller; and the controller being capable step, wherein the control means verifies the program instruc of controlling the display device to present information to tions so that the program instructions are executable by the the user relating to the use of the apparatus if necessary for control means. use of the device by the user. 0916. It is still another object of the present invention to 0921. Another object of the present invention provides a provide a programmable apparatus, wherein the control system for presenting a program to a viewer, comprising a means further causes the display means to display a confir Source of program material, means for determining a viewer mation screen after the program instructions are input, so preference, the viewer preference optionally being context that the programmer may confirm the program instructions. sensitive, means for receiving the program material from the 0917. Another object of the present invention is to pro Source, means for characterizing the program material based vide a programmable information storage apparatus, on its content, means for correlating the characterized con wherein the control means further comprises means for tent of the program material with the determined viewer recognizing character data present in a data stream of the preference to produce a correlation index; and means for input data, the identifying data comprising the recognized presenting the program material to the viewer, if the corre character data. lation index indicates a probable high correlation between the characterization of the program material and the viewer 0918. It is a still further object of the present invention to preference. provide a video tape recording apparatus, comprising a Video signal receiving device, a recording device for record 0922. Another object of the present invention is to pro ing the video signal, wherein the control analyzes the video vide a system for presenting a program to a viewer, com signal for the presence of a symbol, and recognizes the prising a source of program material, means for determining symbol as one of a group of recognized symbols, and the a viewer preference; means for receiving the program mate control stores the recognized symbol separately from the rial from the source; means for storing the program material, Video signal. means for preprocessing the program material to produce a reduced data flow information signal retaining information 0919) Another object of the present invention is to pro relating to a character of the program material and elimi vide a recording device for recording an analog signal nating data not necessary to characterize the program mate sequentially on a recording medium, comprising means for rial, means for characterizing the information signal based characterizing the analog signal, wherein data representing on its content, means for correlating the characterized con the characterization and a location of the analog signal on tent of the information signal with the determined viewer the recording medium are stored in a directory location on preference to produce a correlation index; and means for the recording medium separately from the analog signal. presenting the stored program material to the viewer, if the 0920. It is a further object of the present invention to correlation index indicates a probable high correlation provide an interface for a programmable control for input of between the characterization of the information signal and a program for a controller to execute, which performs an the viewer preference. The system may also include a means action based on an external signal, comprising an input for storing the information signal, wherein the characteriz device, a controller for receiving data from the input device ing means characterizes the stored information signal, and and from an external stimulus, a plant being controlled by also a memory for storing the program material while the the controller based on an input from the input device and characterizing means produces characterized content and the the external stimulus, and a display device being controlled correlating means produces the correlation index. by the controller, for providing visual feedback to a user 0923 Still another object of the present invention is to operating the input device, wherein a predetermined logical provide a system, wherein the program material is sequence of programming options is presented to the user on encrypted, further comprising means for decrypting the the display device, in a plurality of display Screens, each of program material to produce a decryption event; and means the display Screens differing in available programming for charging an account of the viewer based on the occur choices; the logical sequence including a correct sequence of rence of a decryption event. Thus, a decryption processor choices to set an operable control program, so that no and an accounting database are provided for these purposes. necessary steps are omitted, the external stimulus comprises a timing device, and the display comprises a display option 0924. Another object of the present invention is to allow for programming the plant to perform an action at a time the means for characterizing the program material to operate which is input through the input device as a relative position without causing a decryption event. Thus, the data stream on the display device, the relative position including a means may include characterization data specifically Suitable for for displaying an absolute time entry and means for display processing by a characterizing system, or the decryption ing a relative time entry, the display also comprising a processor may be provided with multiple levels of function display option means for performing an action at a time, the ality, or both. Further, the system may comprise a memory control comprises means for presenting the user, on the for storing the program material while the characterizing display device, with a most probable action, which may be means produces characterized content and the correlating selected by the user through activation of the input device means produces the correlation index. The characterizing US 2007/0016476 A1 Jan. 18, 2007 44 means may also characterize the program material stored in 0928. Another object of the present invention provides a memory, and the program material stored in memory may be method wherein the step of selecting the most closely compressed. corresponding one of mapped ranges includes the step of 0925) Another object of the present invention is to pro selecting, for each domain, the mapped range with the vide a controller for controlling a plant, having a sensor for minimum modified Hausdorff distance calculated as Ddb. sensing an external event and producing a sensor signal, an mrb--D1-db.1-mrb), where D is a distance calculated actuator, responsive to an actuator signal, for influencing the between a pair of sets of data each representative of an external event, and a control means for receiving the sensor image, db is a domain, mrb is a mapped range, 1-db is the signal and producing an actuator signal, comprising means inverse of a domain, and 1-mrb is an inverse of a mapped for inputting a program, means for storing the program; range. means for characterizing the sensor signal to produce a 0929. Another object of the present invention provides a characterized signal; and means for comparing the charac method wherein the digital image data consists of a plurality terized signal with a pattern stored in a memory to produce of pixels each having one of a plurality of associated color a comparison index, wherein the actuator signal is produced map values, further comprising the steps of optionally on the basis of the comparison index and the program, transforming the color map values of the pixels of each wherein the characterization comprises an Afline transfor domain by a function including at least one Scaling function mation of the sensor signal. The characterization may com for each axis of the color map, each of which may be the prise one or more transformation selected from the group same or different, and selected to maximize the correspon consisting of an Affine transformation, a Fourier transfor dence between the domains and ranges to which they are to mation, a Gabor transformation, and a wavelet transforma be matched; selecting, for each of the domains, the one of tion. the mapped ranges having color map pixel values which most closely correspond to the color map pixel values of the 0926. It is another object of the present invention to domain according to a predetermined criteria, wherein the provide a method for automatically recognizing digital step of representing the image color map information image data consisting of image information, the method includes the Substep of representing the image color map comprising the steps performed by a data processor of information as a set of values each including an identifier of storing a plurality of templates, storing the image data in the the selected mapped range and the scaling functions; and data processor; generating a plurality of addressable selecting a most closely corresponding stored template, domains from the stored image data, each of the domains based on the identifier of the color map mapped range, the representing a portion of the image information, creating, Scaling functions and the set of identifiers representing the from the stored image data, a plurality of addressable image information. The first criteria may comprise minimiz mapped ranges corresponding to different Subsets of the ing the Hausdorff distance between each domain and the stored image data, the creating step including the Sub step of selected range. (a) executing, for each of the mapped ranges, a correspond ing procedure upon the one of the Subsets of the stored 0930. Another object of the present invention is to pro image data which corresponds to the mapped ranges; (b) vide a method further comprising the steps of storing assigning identifiers to corresponding ones of the mapped delayed image data, which represents an image of a moving ranges, each of the identifiers specifying for the correspond object differing in time from the image data in the data ing mapped range a procedure and a address of the corre processor, generating a plurality of addressable further sponding Subset of the stored image data; (c) optionally domains from the stored delayed image data, each of the Subjecting a domain to a transform selected from the group further domains representing a portion of the delayed image consisting of a predetermined rotation, an inversion, a information, and corresponding to a domain, creating, from predetermined scaling, and a predetermined preprocessing the stored delayed image data, a plurality of addressable in the time, frequency, and/or wavelet domain, (d) selecting, mapped ranges corresponding to different Subsets of the for each of the domains or transformed domains, the one of stored delayed image data, matching the further domain and the mapped ranges which most closely corresponds accord the domain by subjecting a further domain to one or both of ing to predetermined criteria, (e) representing the image a corresponding transform selected from the group consist information as a set of the identifiers of the selected mapped ing of a null transform, a rotation, an inversion, a scaling, a ranges; and (f) selecting, from the stored templates, a translation and a frequency domain preprocessing, which template which most closely corresponds to the set of corresponds to a transform applied to a corresponding identifiers representing the image information. The step of domain, and a noncorresponding transform selected from selecting the mapped ranges may also include the Substep of the group consisting of a rotation, an inversion, a scaling, a selecting, for each domain, a most closely corresponding translation and a frequency domain preprocessing, which one of the mapped ranges. does not correspond to a transform applied to a correspond ing domain, computing a motion vector between one of the 0927. It is another object of the present invention to domain and the further domain, or the set of identifiers provide a method wherein the step of selecting the most representing the image information and the set of identifiers closely corresponding one of the mapped ranges includes the representing the delayed image information, and storing the step of selecting, for each domain, the mapped range which motion vector, compensating the further domain with the is the most similar, by a method selected from one or more motion vector and computing a difference between the of the group consisting of selecting minimum Hausdorff compensated further domain and the domain, selecting, for distance from the domain, selecting the highest cross-cor each of the delayed domains, the one of the mapped ranges relation with the domain, selecting the highest fuZZy corre which most closely corresponds according to predetermined lation with the domain and selecting the minimum mean criteria, representing the difference between the compen square error with the domain. sated further domain and the domain as a set of difference US 2007/0016476 A1 Jan. 18, 2007 identifiers of a set of selected mapping ranges and an prises program selection means, operably connected to the associated motion vector and representing the further storing means and the receiving means, for selecting one or domain as a set of identifiers of the selected mapping ranges; more programs using a user's programming preferences and determining a complexity of the difference based on a the program control information. In this case, the program density of representation, and when the difference has a selection means may comprise a processor, wherein the user complexity below a predetermined threshold, selecting, programming preferences are generated from the user spe from the stored templates, a template which most closely cific data; and means, operably connected to the program corresponds to the set of identifiers of the image data and the selection means, for Suggesting the selected programs to the set of identifiers of the delayed image data. user. The apparatus processing means selectively may 0931. Another object of the present invention provides an records the content databased on the output of the process apparatus for automatically recognizing digital image data ing means. Further, the presenting means presents informa consisting of image information, comprising means for tion to the user in a menu format. The presenting means may storing template data, means for storing the image data; comprises means for matching the user attribute to content means for generating a plurality of addressable domains data. from the stored image data, each of the domains representing 0934 The data describing elements of an associated data a different portion of the image information, means for stream may, for example, comprise a program guide gener creating, from the stored image data, a plurality of addres ated remotely from the apparatus and transmitted in elec sable mapped ranges corresponding to different Subsets of tronically accessible form, data defined by a human input, the stored image data, the creating means including means and/or data defined by an automated analysis of the content for executing, for each of the mapped ranges, a procedure data. upon the one of the subsets of the stored image data which corresponds to the mapped range; means for assigning 0935 According to another embodiment, the present identifiers to corresponding ones of the mapped ranges, each invention comprises a method, comprising the steps of of the identifiers specifying for the corresponding mapped receiving data describing an user attribute, receiving a range an address of the corresponding Subset of stored image content data stream, and extracting from the content data data; means for selecting, for each of the domains, the one stream information describing a plurality of program of the mapped ranges which most closely corresponds options, and processing the data describing a user attribute according to predetermined criteria, means for representing and the information describing a plurality of program the image information as a set of the identifiers of the options to determine a likely user preference; selectively selected mapped ranges; and means for selecting, from the processing a program option based on the likely user pref erence. The method may be embodied in a terminal for a stored templates, a template which most closely corresponds television program delivery system for Suggesting programs to the set of identifiers representing the image information. to users for display on a television using program control 0932. It is also an object of the present invention to information and user specific data. In that case, the step of provide a method and system for processing broadcast receiving data describing an user attribute may comprise material having a first portion and a second portion, wherein gathering user specific data to be used in selecting programs, the first portion comprises a content segment and the second and storing the gathered user specific data; the step of portion comprises a commercial segment, in order to allow receiving a content data stream, may comprise receiving alteration in the presentation of commercial segments, based both programs and program control information for select on the recipient, commercial sponsor, and content provider, ing programs as the information describing a plurality of while providing means for accounting for the entire broad program options, the selectively processing step may com CaSt. prise selecting one or more programs using a user's pro 0933. Another object of an embodiment of the present gramming preferences and the received program control invention provides an apparatus comprising a user interface, information, wherein the user programming preferences are receiving a control input and a user attribute from the user; generated from the user specific data; and the method further a memory system, storing the control input and user including the step of presenting the program or information attribute, an input for receiving content data, means for describing a program option for the selected programs to the storing data describing elements of the content data, means USC. for presenting information to the user relating to the content 0936. The user attribute may comprise a semantic data, the information being for assisting the user in defining description of a preference, or some other type of descrip a control input, the information being based on the stored tion, for example a personal profile, a mood, a genre, an user attribute and the data describing elements of the content image representing or relating to a scene, a demographic data, and means for processing elements of the content data profile, a past history of use by the user, a preference against in dependence on the control input, having an output. This certain types of media, or the like. In the case of a semantic apparatus according to this embodiment may be further preference, the data processing step may comprise deter defined as a terminal used by users of a television program mining a semantic relationship of the user preference to the delivery system for Suggesting programs to users, wherein information describing a plurality of program options. The the user interface comprises means for gathering the user program options may, for example, be transmitted as an specific data to be used in selecting programs; the memory electronic program guide, the information being in-band system comprises means, connected to the gathering means, with the content (being transmitted on the same channel), on for storing the user specific data; the input for receiving data a separate channel or otherwise out of band, through a describing elements of the content data comprises means for separate communications network, e.g., the Internet, dial-up receiving the program control information containing the network, or other streaming or packet based communica program description data; and the processing means com tions system, or by physical transfer of a computer-readable US 2007/0016476 A1 Jan. 18, 2007 46 storage medium, such as a CD-ROM or floppy disk. The connected to the program selection means, for displaying for electronic program guide may include not only semantic or Suggestion the selected programs to the user. human-readable information, but also other types of meta 0939. It is a further aspect of the invention to provide data relating to or describing the program content. device a device comprising: a data selector, for selecting a 0937. In a further embodiment of the present invention, it program from a data stream; an encoder, for encoding is an object to provide a device for identifying a program in programs in a digitally compressed format; a mass storage response to user preference data and program control infor system, for storing and retrieving encoded programs; a mation concerning available programs, comprising means decoder, for decompressing the retrieved encoded programs; for gathering the user preference data, means, connected to and an output, for outputting the decompressed programs. the gathering means, for storing the gathered user preference 0940. Therefore, the present invention provides a system data; means for accessing the program control information, and method for making use of the available broadcast media and means, connected to the storing means and accessing forms for improving an efficiency of matching commercial means, for identifying one or more programs based on a information to the desires and interests of a recipient, correspondence between a user's programming preferences improving a cost effectiveness for advertisers, improving a and the program control information. For example, the perceived quality of commercial information received by identifying means identifies a plurality of programs, a recipients and increasing profits and reducing required infor sequence of identifications transmitted to the user being based on a degree of correspondence between a user's mation transmittal by publishers and media distribution programming preferences and the respective program con entities. trol information of the identified program. The device my 0941 This improved advertising efficiency is accom selectively record or display the program, or identify the plished by providing a system for collating a constant or program for the user, who may then define the appropriate underlying published content work with a varying, demo action by the device. Therefore, a user may, instead of graphically or otherwise optimized commercial information defining “like' preferences, may define “dislike' preference, content. This commercial information content therefore which are then used to avoid or filter certain content. Thus, need not be predetermined or even known to the publisher this feature may be used for censoring or parental screening, of the underlying works, and in fact may be determined on or merely to avoid unwanted content. Thus, the device an individual receiver basis. It is also possible to integrate comprises a user interface adapted to allow interaction the demographically optimized information within the con between the user and the device for response to one or more tent. For example, overlays in traditional media, and elec of the identified programs. The device also preferably com tronic Substitutions or edits in new media, may allow seam prises means for gathering the user specific data comprises less integration. The content alteration need not be only means for monitoring a response of the user to identified based on commercial information, and therefore the content programs. may vary based on the user or recipient. The technologies emphasize adaptive pattern recognition of both the user 0938. It is a further object of the invention to provide a input and data, with possible use of advanced signal pro device which serves as a set top terminal used by users of a cessing and neural networks. These systems may be shared television program delivery system for Suggesting programs between the interface and operational systems, and therefore to users using program control information containing a controller for a complex system may make use of the scheduled program description data, wherein the means for intrinsic processing power available, rather than requiring gathering the user preference data comprising means for additional computing resources, although this unification is gathering program watched data; the means, connected to not required. In fact, while hardware efficiency dictates that the gathering means, for storing the gathered user preference near term commercial embodiments employ common hard data comprising means, connected to the gathering means, ware for the interface system and the operational system, for storing the program watched data; the means for access future designs may successfully separate the interface sys ing the program control information comprising means for tem from the operational system, allowing portability and receiving the program control information comprising the efficient application of a single interface system for a num scheduled program description data; the means, connected ber of operational systems. to the storing means and accessing means, for identifying one or more programs based on a correspondence between 0942. The adaptive nature of the technologies derive a user's programming preferences and the program control from an understanding that people learn most efficiently information, being for selecting at least one program for through the interactive experiences of doing, thinking, and Suggestion to the viewer, comprising: means for transform knowing. Users change in both efficiency and strategy over ing the program watched data into preferred program indi time. To promote ease-of-use, efficiency, and lack of frus cators, wherein a program indicator comprises a program tration of the user, the interface of the device is intuitive and category with each program category having a weighted self explanatory, providing perceptual feedback to assist the value; means for comparing the preferred program indica operator in communicating with the interface, which in turn tors with the scheduled program description data, wherein allows the operational system to identify of a desired opera each scheduled program is assigned a weighted value based tion. Another important aspect of man-machine interaction on at least one associated program category; means for is that there is a learning curve, which dictates that devices prioritizing the scheduled programs from highest weighted which are especially easy to master become frustratingly value programs to lowest weighted value programs; means elemental after continued use, while devices which have for indicating one or more programs meeting a predeter complex functionality with many options are difficult to mined weight threshold, wherein all other programs are master and may be initially rejected, or used only at the excluded from program Suggestion, and means, operably simplest levels. The present technologies address these US 2007/0016476 A1 Jan. 18, 2007 47 issues by determining the most likely instructions of the event. Past systems have required explicit indexing or iden operator, and presenting these as easily available choices, by tification of images and events. The present technologies, analyzing the past history data and by detecting the 'sophis however, allow an image, diagrammatic, abstract or linguis tication of the user in performing a function, based on all tic description of the desired event to be acquired by the information available to it. The context of use is also a factor interface system from the user and applied to identify or in many systems. The interface seeks to optimize the inter predict the multimedia event(s) desired, without requiring a face adaptively and immediately in order to balance and separate manual indexing or classification effort. These optimize both quantitative and qualitative factors. This func technologies may also be applied to single media data. tionality may greatly enhance the quality of interaction 0949. The interface system analyzes data from many between man and machine, allowing a higher degree of different sources for its operation. Data may be stored or overall system sophistication to be tolerated. present in a dynamic data stream. Thus, in a multimedia 0943. The interface system analyzes data from the user, system, there may be a real-time video feed, a stored event which may be both the selections made by the user in database, as well as an exemplar or model database. Further, context, as well as the efficiency by which the user achieves since the device is adaptive, information relating to past the selection. Thus, information concerning both the end experience of the interface, both with respect to exposure to points and path are considered and analyzed by the human data streams and user interaction, is also stored. user interface system. 0950 This data analysis aspect of the interface system 0944. The interface may be advantageously applied to an may be substantially processor intensive, especially where operational system which has a plurality of functions, cer the data includes abstract or linguistic concepts or images to tain of which are unnecessary or are rarely used in various be analyzed. Interfaces which do not relate to the processing contexts, while others are used with greater frequency. In of such data may be implemented with simpler hardware. On Such systems, the application of functionality may be pre the other hand, systems which handle complex data types dictable. Therefore the present technologies provide an may necessarily include Sophisticated processors, adapt ale optimized interface system which, upon recognizing a con for use by the interface system. A portion of the data analysis text dynamically reconfigures the availability or ease of may also overlap the functional analysis of the data for the availability of functions and allows various functional sub operational system. sets to be used through “shortcuts”. The interface presenta 0951. Other objects and features of the present invention tion will therefore vary over time, use and the particular user. will become apparent from the following detailed descrip 0945. The advantages to be gained by using an intelligent tion considered in conjunction with the accompanying draw data analysis interface for facilitating user control and ings. It is to be understood, however, that the drawings are operation of the system are more than merely reducing the designed solely for the purposes of illustration and not as a average number of selections or time to access a given definition of the limits of the invention, for which reference function. Rather, advantages also accrue from providing a should be made to the appended claims. means for access and availability of functions not necessar ily previously existing or known to the user, improving the BRIEF DESCRIPTION OF THE DRAWINGS capabilities and perceived quality of the product. 0952 Embodiments of the present invention are shown in 0946. Further improvements over prior interfaces are also the figures in the drawings, in which: possible due to the availability of pattern recognition func 0953 FIG. 1 is a flow chart of the steps required to set a tionality as a part of the interface system. In those cases where the pattern recognition functions are applied to large VCR: amounts of data or complex data sets, in order to provide a 0954 FIG. 2 shows a graphical comparison of required Sufficient advantage and acceptable response time, powerful and extra keypresses for the prior art and the interface of the computational resources. Such as powerful RISC processors, present invention, advanced DSPs or neural network processors are made 0955 FIG.3 graphically shows the differences in seconds available to the interface system. On the other hand, where between total time for the prior art for each user, the data is simple or of limited Scope, aspects of the technology may be easily implemented as added software 0956 FIG. 4 graphically shows the differences in seconds based functionality in existing products having limited com between total time for the interface of the present invention putational resources. for each user, 0947 The application of these technologies to multime 0957 FIG. 5 graphically shows the programming steps dia data processing systems provides a new model for for the comparison of the prior art and the interface of the performing image pattern recognition and for the program present invention, ming of applications including Such data. The ability of the 0958 FIG. 6 graphically shows comparative statistics by interface to perform abstractions and make decisions regard user comparing the prior art and the interface of the present ing a closeness of presented data to selection criteria makes the interface Suitable for use in a programmable control, i.e., invention, determining the existence of certain conditions and taking 0959 FIGS. 7 and 8 graphically show the critical steps in certain actions on the occurrence of detected events. Such programming the prior art and the interface of the present advanced technologies might be especially valuable for invention, disabled users. 0960 FIG.9 graphically shows the number of keypresses 0948. In a multimedia environment, it may be desirable made by test participants comparing the prior art and the for a user to perform an operation on a multimedia data interface of the present invention, US 2007/0016476 A1 Jan. 18, 2007 48

0961 FIG. 10 graphically shows the comparison of the 0981 FIG. 33 shows a flow chart in accordance with an actual and theoretical number of keypresses necessary for embodiment of the present invention. programming the prior art and the interface of the present invention, DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 0962 FIG. 11 graphically compares the actual and theo retical time necessary for programming the prior art and the 0982) The preferred embodiments of the present inven interface of the present invention, tion will now be described with reference to the Figures. Identical elements in the various figures are designated with 0963 FIGS. 12a and 12b graphically compares the actual the same reference numerals. and theoretical time necessary for setting the programs in the prior art and the interface of the present invention, EXAMPLE 1. 0964 FIGS. 13 and 14 graphically show the percentage 0983 VCR Interface time for the critical steps in programming the prior art and 0984) A preferred embodiment of the interface of the the interface of the present invention, present invention, described in the present example, pro 0965 FIG. 15 is a flow diagram of a predictive user vides automatic sequencing of steps, leading the user interface of the present invention, through the correct sequence of actions to set a program on the Screen, so that no necessary steps are omitted, and no 0966 FIG. 16 is a flow diagram of the program input optional steps are accidentally or unintentionally omitted. verification system of the present invention, These steps are shown diagrammatically in FIG. 15 of the present invention. In addition, such a system does not 0967 FIG. 17 is a flow diagram of a predictive user burden the user with the necessity of inputting Superfluous preference aware interface of the present invention, information, nor overwhelm the user with the display of 0968 FIG. 18 is a block diagram of a non-program unnecessary data. See, Hoffberg, Linda I., “AN IMPROVED information feature extraction circuit of the present inven HUMAN FACTORED INTERFACE FOR PROGRAM tion, MABLE DEVICES: A CASE STUDY OF THE VCR, Master's Thesis, Tufts University, Hoffberg, Linda I., 0969 FIG. 19 is a diagram of a block of information for “Designing User Interface Guidelines For Time-Shift Pro a catalog entry of the present invention, gramming of a Video Cassette Recorder (VCR). Proc. 0970 FIG. 20 is a block diagram of a digital information Human Factors Soc. 35th n. Mtg. pp. 501-504 (1991); and and analog signal reading/recording apparatus, Hoffberg, Linda I., “Designing a Programmable Interface for a Video Cassette Recorder (VCR) to Meet a User's Needs', 0971 FIG. 21 is a block diagram of a user level deter Interface 91 pp. 346-351 (1991). See also, U.S. patent mining system of the present invention, application Ser. No. 07/812,805, incorporated herein by 0972 FIG. 22 is a block diagram of a template-based reference in its entirety, including appendices and incorpo pattern recognition system of the present invention, rated references. 0973 FIG. 23 is a block diagram of a control system of 0985. Many design considerations were found to be the present invention incorporating a pattern recognition important in the improved interface of the present invention: element and an interface, 0986 The interface should preferably employ only mini mal amounts of abbreviations and the use of complete words 0974 FIG. 24 is a block diagram of a control system for is especially preferred, except where a standard abbreviation characterizing and correlating a signal pattern with a stored is available or where an iconic' or symbolic figure or textual user preference of the present invention, cue is appropriate. Thus, standard abbreviations and sym 0975 FIG. 25 is a block diagram of a multiple video bols are acceptable, and displayed character strings may be signal input apparatus, with pattern recognition, data com shortened or truncated in order to reduce the amount of pression, data encryption, and a user interface of the present information that is to be displayed, where necessary or invention, desirable. An option may be provided to the user to allow full words, which may decrease the information which may 0976 FIG. 26 is a block diagram of a control system for be conveyed on each screen and increase the number of matching a template with a sensor input, of the present screens that must be displayed, or abbreviations and sym invention, bols, which may minimize the number of displayed screens of information, thus allowing the user to make the compro 0977 FIGS. 27, 28 and 29 are flow diagrams of an mise. This aspect of the system may also be linked to the iterated function system method for recognizing a pattern adaptive user level function of the present invention, according to the present invention, wherein abstract symbols and abbreviations are presented to 0978 FIG. 30 is a semi-cartoon flow diagram of the advanced users, while novices are presented with full words, object decomposition and recognition method of the present based on an implicit indication of user level. These abstract invention, symbols and abbreviations may be standard elements of the system, or user designated icons. Of course, the user could 0979 FIG.31 is a block diagram of an adaptive interface explicitly indicate his preference for the display type, thus system according to the present invention, deactivating the automatic adaptive user level function. 0980 FIG. 32 shows a clock diagram of a system in 0987) If multiple users use the device, then the device accordance with the present invention, and identifies the relevant users. This may be by explicit iden US 2007/0016476 A1 Jan. 18, 2007 49 tification by keyboard, bar code, magnetic code, Smart card should be easily understandable. This is shown in FIG. 15 of (which may advantageously include a user profile for use the present invention. All of the necessary information is with a number of devices), an RF-ID or IR-ID transponder, displayed on this screen, in addition to the change and cancel Voice recognition, image recognition, or fingerprint identi options, if possible. fication. It is noted that Smart cards or other intelligent or 0994 The entering of information on each screen is data-containing identifications systems may be used with preferably consistent throughout the various interface different types of devices, for example video, audio, home options and levels. All of the screens preferably have similar appliances, HVAC and automobile systems. layouts. “Buttons' or screen locations which are keyed to a 0988 Where a new user is identified to the system, an particular function, which appear on multiple screens, initial query may be made to determine an optimum initial should appear in approximately the same location on all user level. This allows further identification of the user and screens. However, in certain cases, relatively more impor preference determination to occur more efficiently. tant information on a given screen may be displayed more prominently, and possibly in a different Screen location, in 0989. In applications in which a user must program an order to reduce the search time. Further, when other factors event on a certain date, at a certain time, a built-in calendar dictate, each screen may be independently optimized for the menu screen is preferably employed so that the user cannot prescribed function. For example, a representation of an set the device with a program step that relies on a non analog clock dial may be used to set time information. existent date. Technology that will help eliminate the human However, even if the format does change, a standard Scheme problem of setting the wrong (yet existing) date may also be should be maintained. Such as the use of a particular color employed. Such technology might include accessing an to indicate that a particular program aspect has been on-line or other type of database containing media program changed. ming information, and prompting the user regarding the selected choice. In situations where it is applicable, the 0995 The interface should display data consistent with interface should indicate to the user the number of characters standards and conventions familiar to users. For, e.g., when the interface is expecting, such as when entering the year. entering dates, users are most familiar with calendars. How ever, this type of presentation of choices does not eliminate 0990 The interface system provides an easily accessible the human problem of entering incorrect information, e.g., CHANGE, CANCEL or UNDO (single or multiple level) setting a wrong, but existing, date. The problem of ensuring feature, which facilitates backtracking or reprogramming the the accuracy of user input may be addressed by an intelligent immediately previously entered information rather than interface which stores data concerning programming, user forcing the user to repeat all or a substantial portion of the preferences, and by means of Some logical method. Such as programming steps. A method of the type described is shown Boolean logic, fuzzy logic, neural network theory, or any in FIG. 16 of the present invention. User input is also other system which may be used to generate a prediction, to facilitated by the provision of frequently used settings as determine if an entry is likely in error, by comparing the explicit choices, such as, referring to the VCR example, prediction with the entry. Of course, these predictive sys “Record today,”“Record tomorrow,”“Noon,” and “Mid tems would also provide an initial default entry, so that an night.” So that the user does not have to specify a date in a priori most probably action or actions may be initially these cases. This will eliminate extra keypresses, and reduce presented to the user. the programming time. In addition, this could eliminate user errors. Frequently used choices for program selections are 0996. In addition to following conventions of informa also provided to the user to reduce the number of program tion presentation to the user, the interface of the present ming steps necessary and provide the user with all the invention may also provide emulations of other user inter frequently used selections. The especially preferred choices faces of which a particular user may be familiar, even if are “Once On.”, “Once a Week on.”, “Monday-Friday at.”, these are not optimized according to the presently preferred “Everyday at.”. These redundant, complex instructions embodiments of the present invention, or not otherwise well reduce the number of keystrokes required for data entry, and known. These emulations need not even be of the same type reduce the amount of programming time required. of device, so that a broad based standard for entry of information into programmable controls, regardless of their 0991. The presently described interface system also pro type, may be implemented. By allowing emulation, the vides, in the event that a color screen is available, conser interface could provide compatibility with a standard or vatively used color coding, which allows the user to effec proprietary interface, with enhanced functionality provided tively and quickly acknowledge the function of each aspect by the features of the present interface. of the screen. When programming, the preferred colors are royal blue for “help.' red for mistakes, light blue for 0997 These enhanced functional intelligent aspects of information previously entered, and yellow for current infor the controller may be implemented by means of software mation being entered. Of course, other colors could be used, programming of a simple microcomputer, or by use of more according to the user's or designer's preference, cultural specialized processors, such as a Fuzzy Set Processor (FSP) differences, and display parameters. or Neural Network Processor to provide real-time respon siveness, eliminating delays associated with the implemen 0992. When viewing, it is preferable that screen colors tation of complex calculations on general purpose comput change to indicate status changes, such as viewed/unviewed, ing devices. or to categorize the shows. 0998. In the various embodiments according to the 0993. The interface includes a confirmation screen which present invention, various control strategies are employed. displays to the user all of the categories and selections Depending on the application, fuzzy set processors (FSPs) previously explicitly entered or otherwise inferred, and may be preferred because they have the advantage of being US 2007/0016476 A1 Jan. 18, 2007 50 easier to program through the use of presumptions or rules cursor, in these cases, should be locked to a choice which for making the fuzzy inferences, which may be derived by returns the user to where they left off in the programming trial and error or the knowledge of experts, while Neural sequence, and this choice should be highlighted. Networks are less easily explicitly programmed and their network weighing values are not easily understood in the 1003. The “help' function may also comprise “balloon abstract, but these systems may be applied to learn appro help' similar to the system adopted by Apple Computer, Inc. priate responses from test data. Thus, neural networks tend in Macintosh Operating System, e.g., 7.0. 7.1, 7.5, etc. to require extensive “training, while Fuzzy Set Processors 1004 The interface preferably initiates the programming may be explicitly programmed without the need of dupli sequence where the user wants to be, so that the interface has cating or simulating actual operating conditions, but may so-called “smart screens”. For example, when a VCR is first require “fine tuning. powered up or after an extended power failure, and the time and date are not stored in the machine, the “set date' and “set 0999 The most frequently used choices preferably time screens should appear. The sequence of screens may should be displayed as the default setting. The screen cursor also vary depending on the system predicted requirements of preferably appears at the “accept screen button, when the the user and various aspects of the improved interface of the screen is displayed. This default can either be set in advance, present invention. This is shown schematically in FIG. 17 of or acquired by the system. In the case of acquired defaults, these may be explicitly set by the user or adaptively acquired the present invention. by the system through use. The interface of the present 1005 The preferable input device for the interface of the invention may be taught, in a “teach mode, the preferences present invention provides as few buttons as possible to of the user, or may also acquire this information by analyZ achieve the required functionality, thus reducing potential ing the actual choices made by the user during operation of user intimidation, focusing the user S attention on the the interface and associated controller. This type of opera interactive display screen, where the available choices are tion is shown schematically in FIG. 15 of the present minimized to that number necessary to efficiently allow the invention. The options of "Midnight” (12:00 AM) and user to program the discrete task presented. Such a minimi “Noon” (12:00 PM) should preferably be present, as some Zation of discrete inputs facilitates a voice recognition input, people often become confused when distinguishing between which may be used as an alternative to mechanical input them. Icons, such as those indicative of the 'sun' and the devices. The preferred embodiment includes a direct-ma “moon', may also be used to facilitate data entry for AM and nipulation type interface, in which a physical act of the user PM. The interface should preferably utilize an internal clock causes a proportionate change in the associated interface and calendar so that the user cannot set the time or program characteristic, such as cursor position. A computer mouse, to record on a nonexistent date. Such a system could also e.g. a two dimensional input device, with 1 to 3 buttons is compensate for daylight-savings time seasonal adjustments. the preferred input device, for use with a general purpose computer as a controller, while a trackball on a remote 1000 The cursor is preferably distinctive and readily control device is especially preferred for limited purpose distinguished from other parts of the screen. This may be by controllers because they do not require a flat surface for color, attribute (i.e. blinking), size, font change of underly operation. Other stationary or movement sensitive input ing text, or by other means. devices may, of course be used, such as joysticks, gyro Scopes, Sonic echo-location, magnetic or electrostatic loca 1001 The user can preferably exit the programming tion devices, RF phase location devices, Hallpots (joystick sequence at any time by selecting a “Main Menu button like device with magnets that move with respect to Hall which may exist on the lower left-hand corner of every effect transducers), etc. The present interface minimizes the screen. The user is preferably provided with an adequate number of necessary keys present on an input device, while amount of feedback, and error messages should be directive maintaining the functionality of the interface. It is noted that in nature. Some form of an acknowledgement is preferably a strict minimization without consideration of functionality, displayed after each entry. The user should preferably not be might lead to inefficiency. For example, in a VCR device, if able to go to the next programming step until the current step the user wants to record a program which airs Monday has been completed. A message to convey why the user can through Friday, he would have to set five separate programs, not continue should appear when an attempt to prematurely rather than one program if a “weeknights' choice is made continue is recognized. available. 1002 The “help' function is available for when the user 1006. The interface preferably should be easy to learn does not know what to do. The “help' screen(s) preferably and should not require that a user have prior knowledge of explains the functions of each of the available buttons or the interface in order to use it. An attempt has been made to functions, but may also be limited to those that are ambigu minimize the learning curve, i.e., to minimize the time it ous. The “help' screen may also be used to indicate a current takes to learn how to use the device. status of the interface and the controller. Further, the “help” function may also provide access to various other functions, 1007 Menu options are preferably displayed in logical Such as advanced options and configurations, and thus need order or in their expected frequencies. Research has shown not be limited to merely providing information on the that a menu-driven interface is best for applications involv display. The help system may incorporate a hypertext-type ing new users and does not substantially hinder experienced system, wherein text or information relating to concepts that users. Menu selection is preferably used for tasks which are conceptually linked may be easily accessed by indicating involve limited choices. They are most helpful for users with to the interface system that the related information is little or no training. Each menu should preferably allow only desired. To eliminate the possibility of the user trying to one selection at a time. Most of the information is preferably make selections on merely informative help screens, the entered using a numeric keypad (entry method), rather than US 2007/0016476 A1 Jan. 18, 2007 using up and down arrow keys (selection method). In 1011. Other characteristics of this interface include color addition, no leading Zeros are required for entry. If there is coding to help prompt the user as to which data must be more than one keystroke required, the user must then select entered. Red text signifies instructions or errors, yellow text an “OK” button to continue in the programming sequence. represents data which must be entered or has not been However, if the selection method is used, all of the choices changed, and blue text shows newly entered program data or are displayed on the Screen at once. The number of steps status information. Blue buttons represent buttons which required to complete the task through a sequence of menus should normally be pressed during the programming should be minimized. The choice of words used to convey sequence. Redbuttons signify an erratic pattern in the data information should not be device specific, i.e., computer entry, such as the "cancel and “return to main menu terms, but rather normal, everyday terms which are easy to buttons. Of course, these colors can be replaced by other understand. In addition, very few abbreviations should be display attributes, such as intensity, underline, reverse video, used. All necessary information which the user needs should blinking and pixel dithering pattern, in addition to the use of preferably be displayed at once. A user preferably should not various fonts. Such a situation would include a monochrome have to rely on his memory or his previous experience, in monitor or display. order to find the correct choice, at least at the lower user 1012 The date may be entered in the form of a calendar levels. If all selections cannot be displayed at once, a rather than as numbers (i.e., “Sep. 6, 1991). This calendar hierarchical sequence is preferably used. A main menu method is advantageous because users may wish to input should preferably provide a top level to which the user can date data in one of three ways: day of the week, day relative always return and start over. to the present, and day of the month. The present method 1008) Searching and learning times should be kept to a allows the current date to be highlighted, so that the calendar minimum in order to obtain a subjectively better interface. may be used to easily enter the absolute day, absolute date, The system's logic should reflect the users’ expectations, and relative day. Further, the choices “today' and “tomor offer visual clues and feedback, and stay within human row', the most frequently used relative recording times, are memory limits. For example, the VCR should turn on not included in addition to a month-by-month calendar. This only with the “Power' button, but also when inserting a tape information is provided to avoid an unnecessary waste of into the device. In addition, the sequence of steps for setting time and user frustration. Thus, another aspect of the present the machine to record, if the user does not indicate implicitly invention is to provide a partially redundant interactive or explicitly that he knows how to use the device, should display input system which allows, according to the highest assume that the user is a novice, and fully prompt the user probability, the choices to be prominently displayed and for elemental items of information. Nothing should be taken easily available, in addition to allowing random access to all for granted. By developing an improved interface, an choices. attempt is made to: reduce the searching time; reduce the 1013 The present device allows common user mistakes learning time; simplify the entering of data; and, reduce the to be recognized and possibly addressed. Such as the con intimidation experienced by certain persons when using fusion between 12:00 PM and 12:00 AM with midnight and electronic devices. noon, respectively. Therefore, the options of “noon and 1009 Tests by an inventor hereof show that people do not “midnight” are provided in addition to a direct numeric program their VCRs often, and they often forget the clock input. When entering time information, leading Zeros sequence of steps between recording sessions. Thus, the need not be entered, and Such information may be entered in present invention preferably incorporates an adaptive user either fashion. level interface, wherein a novice user is presented with a 1014 The criteria for system acceptance of input depends simpler interface with fewer advanced features initially on how many keystrokes are required on the screen. If only available, so that there is reduced searching for the basic one keystroke is required to complete input of the informa functions. A more advanced user is presented with more tion, upon depressing the key, the programming sequence advanced choices and functions available initially, as com will continue. If more than one keypress is required, the user pared to a novice user. must depress the “OK” button to continue programming. 1010 Thus, as shown in FIG. 17, the user identifies This context sensitive information entry serves to avoid himself to the controller in block 1701. The controller 1806 unnecessary input. of FIG. 18 thereafter uses a stored profile of the identified 1015) An on-line “help system and on-line feedback is user in controlling the interaction with the user, as shown in preferably provided to the user throughout various aspects of block 1702 of FIG. 17, from information stored in the the interface. Other features include minimizing the number database 1807 of FIG. 18 of the present invention. It has of keypresses required to program the device. These fea been found that in the case of novice users, a greater number tures, together with other aspects of the present invention of simple instructions may be more quickly and easily input allow the user to achieve a greater efficiency with the input rather than a potentially fewer number of a larger set of more device than with prior art devices. complex instructions. It has further been found that, even if presented with a set of instructions which will allow a 1016. The interface of the present invention applied to a program to be entered with a fewer number of inputs, a VCR control preferably comprises a virtual keypad entry novice user may choose to input the program using the device (i.e. a representation of an array of choices), a simple instructions exclusively, thus employing an increased directional input control for a cursor on a display screen, and number of instructions and being delayed by an increased selection buttons. The input device has an input correspond search time for those instructions that are used, from the ing to a direction of movement relative to the cursor posi larger set. tion. Thus, since the present input device seeks to minimize US 2007/0016476 A1 Jan. 18, 2007 52 the physical control elements of the human interface device, minimize searching time. Keys should be labeled with text the display elements for a preferred embodiment of the rather than with ambiguous graphics. However, a combina present interface include: tion of both may be preferable in some cases. 1017) 1. number keys 0-9. 1038. Fourth, steps required to complete tasks should be simple, require a short amount of time and not create user 1018 2. enter key. frustration. The system should guide the user along a deci 1019. 3. cancel key. sion path, providing automatic sequencing of steps. The most frequently used choices should be provided as defaults, 1020 4. status indicator. and Smart Screens may be employed. The learning curve 1021 5. return to menu option button. should be minimized through the use of easily understand 1022 6. program type indicator: program once, pro able choices. As a user becomes more Sophisticated, the gram once a week, program Monday-Friday, program interface may present more advanced choices. everyday. 1039 Fifth, there should be a reminder to set the timer and to insert the tape once the programming information is 1023 7. Day indicators: 7 weekdays, today, tomorrow. entered. This reminder may also be automated, to eliminate 1024 8. Noon and midnight choices. the commonly forgotten step of setting the timer, so that the VCR automatically sets the timer as soon as the necessary 1025 9. Help button. information is entered and a tape is inserted. Once the 1026 10. Main menu options: Review, Enter new program is set in memory, a message should appear if a tape recording time, Set time, Set date. is not inserted. If the VCR is part of a jukebox” (automatic changer), the tape may be automatically loaded. The VCR 1027 11. Timer button. should preferably turn on when a tape is inserted. In addi 1028 12. Power button. tion, users should also be able to control the VCR with a 1029) 13. AM/PM choices. Power button. 1040 Sixth, the VCR should be programmable from both 1030) 14.31 day calendar. the remote device and the control panel. 1031) 15. 12 month Choices. 1041. Seventh, each operation should require only one 1032) 16.3 tape speed choices. keypress, if possible, or otherwise reduce the number of keypresses required. There should be a 12 hour clock, not a 1033 User dissatisfaction is generally proportionate to 24 hour clock. There should be an on-screen keypad with the length of “search time, the time necessary in order to entry keys, not “up' and “down” selector keys, allowing for locate and execute the next desired function or instruction. the choice of specific day or time entry. There should be a Search time may be minimized by the inclusion of up to a “start” and a “stop’ recording time, rather than “start time maximum of 4-8 choices per screen and by use of consistent and "length of program' or duration exclusively. The num wording and placement of items on the display. ber of buttons on the remote control should be minimized so 1034) The present invention proceeds from the under that as few buttons as are required are provided. The input standing that there are a number of aspects of a program device should provide for the direct manipulation of screen mable interface that are desirable: elements. A menu driven interface should be provided. 1035 First, users should be able to operate the system 1042. The interface of the present invention provides an successfully, without wide disparities in time. It should take, automatic sequencing of steps which does not normally let e.g., a normal person interacting with a VCR interface, less the user think the previous step is complete. This is shown than seven minutes to set the time and two programs. schematically in FIG. 16. In this manner, important steps Searching time spent in setting the clock, programming, will not be inadvertently omitted. Upon entering the pro getting into the correct mode, and checking whether or not gramming sequence, if the current date or time is not set, the the VCR is set correctly should be kept to a minimum interface will prompt the user to enter this information. through the appropriate choices of menu layout and the Thereafter, the interface will normally default to the main presentation of available choices. menu, the most frequently used first screen. Thus, the interface of the present invention is adaptive, in that its 1036) Second, programming should be a stand-alone pro actions depend on the current state of the device, including cess, and not require an instruction manual. A help system prior programming or use of the device by the user. It can be should be incorporated in the interface. Word choices should appreciated that this adaptive behavior can be extended to be understandable, with a reduction in the use of confusing include extended “intelligence'. For example, if the device word terminology. Error messages should be understand is similarly programmed on a number of occasions, then the able. The system should provide the ability to cancel, change default setup may be adapted to a new "normal program or exit from any step. mode. Further, the apparatus could provide multiple levels 1037. Third, the system should provide on-screen under of user interface, e.g. beginner, intermediate, and advanced, standable information, with adequate visual feedback. The which may differ for various functions, based on the behav displays should be consistent. Color coding should be ior of the user. This user interface level determining feature employed, where applicable, using, e.g. blue-new input; extraction system is shown diagrammatically in FIG. 18. In red—error condition, yellow—static, unchanged value. Lay contrast, prior art interfaces that have different user interface outs should be logical, and follow a predictable pattern. levels, allow the user to explicitly choose the interface level, There should be a maximum of 4-8 choices per screen to which will then be used throughout the system until reset. US 2007/0016476 A1 Jan. 18, 2007

1043. The present system allows discrete tasks to be EXAMPLE 3 conducted more quickly, more efficiently, with reduced search time and with fewer errors than prior art systems. 1050 Serial Data Medium Index 1051 Another aspect of the present invention relates to EXAMPLE 2 the cataloging and indexing of the contents of a storage medium. While random access media normally incorporate 1044 Serial Recording Medium Index a directory of entries on a disk, and devices such as optical jukeboxes normally are used in conjunction with software 1045. In a preferred embodiment of the present invention, that indexes the contents of the available disks, serial access in a VCR, in order to track the content of the tape, a directory mass storage devices, such as magnetic tape, do not usually or a catalog is recorded, preferably digitally, containing the employ an index; therefore, the entire tape must be searched programming information, as well as additional information in order to locate a specific selection. about the recorded programs, in a header, i.e., at the begin ning of the tape, or at other locations on the tape. The device 1052. In the present invention, an area of the tape, may also catalog the tape contents separately, and based on preferable at the beginning of the tape or at multiple loca an identification of the tape, use a separately stored catalog. tions therein, is encoded to hold information relating to the A preferred format for storing information is shown in FIG. contents of the tape. This encoding is shown in FIG. 19. 19. which shows a data format for the information. This format has an identifying header 1901, a unique tape identifier 1046. Thus, if there are a number of selections on the 1902, an entry identifier 1903, a start time 1904, an end time tape, the entire contents of the tape could be accessible 1905 and/or a duration 1906, a date code 1907, a channel quickly, without the need for searching the entire tape. In a code 1908, descriptive information 1909 of the described sequential access medium, the tape transport apparatus must entry, which may include recording parameters and actual still shuttle to the location of the desired material, but it may recorded locations on the tape, as well as a title or episode do so at increased speeds, because there is no need to read identifying information, which may be a fixed or variable the tape once the location is determined; after the tape length entry, optionally representative scenes 1910, which transport nears the desired spot, the tape may be slowed or may be analog, digital, compressed form, or in a form precisely controlled to reach the exact location. related to the abstract characterizations of the scenes formed in the operation of the device. Finally, there are error 1047 The tape read and drive system is shown schemati correcting codes 1911 for the catalog entry, which may also cally in FIG. 20. The algorithm used in the final stage of include advanced block encoding schemes to reduce the approach to the desired portion of the tape or other recording affect of non-Gaussian correlated errors which may occur on medium may incorporate a control employing Fuzzy logic, Video tape, transmission media and the like. This informa Neural Networks, mathematical formulae modeling the sys tion is preferably a modulated digital signal, recorded on, in tem (differential equations) in a Model-based system, a the case of Hi-Fi VHS, one or more of the preexisting tracks Proportional-Differential-Integral (PID) system, or a con on the tape, including the video, overscan area, Audio, Hi-Fi troller employing an algorithm of higher order, or other stereo audio, SAP or control tracks. It should be noted that known control methods. an additional track could be added, in similar fashion to the 1048. If a selection is to be recorded over, the start and overlay of Hi-Fi audio on the video tracks of Hi-Fi VHS. It stop locations would be automatically determined from the is also noted that similar techniques could be used with Beta locations already indicated on the tape. Further, this infor format, 8 mm, or other recording systems, to provide the mation could be stored in memory device (which reads a necessary indexing functions. catalog or index of the tape when a new tape is loaded) or 1053 Digital data may also be superimposed as pseud non-volatile memory device (which stores information relat onoise in the image information, or as other information ing to known tapes within the device) or both types of intermixed or merged with the video information. memory in the VCR, so that an index function may be implemented in the VCR itself, without the need to read an 1054 The recording method is preferable a block encod entire tape. Optionally, a printer, Such as a thermal label ing method with error correction within each block, block printer (available from, e.g. Seiko Instruments, Inc.), redundancy, and interleaving. Methods are known for reduc attached to the device, could be available to produce labels ing the error rate for digital signals recorded on unverified for the tapes, showing the index, so that the contents of a media, Such as videotape, which are subject to burst errors tape may be easily indicated. A label on the tape may also and long term non-random errors. Such techniques reduce include a bar code or two-dimensional coding system to the effective error rate to acceptable levels. These are known store content or characterization information. The stored to those skilled in the art and need not be discussed herein identification and index information is thus stored in a in detail. A standard reference related to this topic is Digital human or machine readable form. Communications by John G. Proakis, McGraw-Hill (1983). The digital data recording scheme is best determined accord 1049. These contents, or a list of contents, need not ing to the characteristics of the recording apparatus. There necessarily be manually entered by the user or created by the fore, if an, e.g. Sony Corporation helical scan recording/ apparatus, rather, these may be derived from published data reproducing apparatus was employed, one of ordinary skill or a database, data transmitted to the control, and/or data in the art would initially reference methods of the Sony determined or synthesized by the control itself. For example, Corporation initially for an optimal error correcting record broadcast Schedules are available in electronic or machine ing scheme, which are available in the patent literature, in readable form, and this information may be used by the the U.S., Japan, and internationally, and the skilled artisan apparatus. would also review the known methods used by other manu US 2007/0016476 A1 Jan. 18, 2007 54 facturers of digital data recording equipment. Therefore, the input and record these characterizations in the index, the these methods need not be explained herein in detail. user may make an explicit request different from the record 1055) The catalog of entries is also preferably stored in ing criteria, after a selection has been recorded. The con non-volatile memory, such as hard disk, associated with the troller would then search the index for matching entries, VCR controller. This allows the random selection of a tape which could then be retrieved based on the index, and from a library, without need for manually scanning the without a manual search of the entire tape. Other advantages contents of each tape. This also facilitates the random of the present system are obvious to those of ordinary skill storage of recordings on tape, without the requirement of in the art. storing related entries in physical proximity with one 1058) A library system is available from Open Eyes another so that they may be easily located. This, in turn, Video, called “Scene Locator, which implements a non allows more efficient use of tape, because of reduced empty intelligent system for indexing the contents of a Videotape. space at the end of a tape. The apparatus is shown sche See NewMedia, November/December 1991, p. 69. matically in FIG. 20, in which a tape drive motor 2001, controlled by a transport control 2002, which in turn is 1059. It is noted that, if the standard audio tracks are used controlled by the control 2003, moves a tape 2005 past a to record the indexing information, then standard audio reading head 2004. The output of the reading head 2004 is frequency modems and recording/receiving methods are processed by the amplifier/demodulator 2006, which pro available, adapted to record or receive data in half-duplex duces a split output signal. One part of the output signal mode. These standard modems range in speed from 300 comprises the analog signal path 2007, which is described baud to about 64 kilobits per second, e.g. v.29, v.17. V.32. v.32bis, v.34, v.90, v.91, etc. While these systems are elsewhere. A digital reading circuit 2008 transmits the designed for dial-up telecommunications, and are therefore digital information to a digital information detecting circuit are designed for the limited data rates available from POTS. 2009, which in turn decodes the information and provides it These are limited to a slower speed than necessary and to the control 2003. incorporate features unnecessary for closed systems, they 1056. In order to retrieve an entry, the user interacts with require a minimum of design effort and the same circuitry the same interface that is used for programming the recorder may be multiplexed and also be used for telecommunication functions; however, the user selects different menu selec with an on-line database. Such as a database of broadcast tions, which guide him to the available selections. This listings, discussed above. It should be noted that a full function, instead of focusing mainly on the particular users duplex modem should be operated in half duplex mode history in order to predict a selection, would analyze the when reading or recording on a media, thus avoiding the entire library, regardless of which user instituted the record generation of unnecessary handshaking signals. Alterna ing. Further, there would likely be a bias against performing tively, a full duplex receiver may be provided with the identically the most recently executed function, and rather resulting audio recorded. A specially programmed receiver the predicted function would be an analogous function, may extract the data from the recording. DTMF codes may based on a programmed or inferred user preference. This is also be employed to stored information. because it is unlikely that a user will perform an identical 1060) The Videotext standard may also be used to record action repeatedly, but a pattern may still be derived. the catalog or indexing information on the tape. This 1057) It is noted that the present library functions differ method, however, if used while desired material is on the from the prior art VISA tape index function, because the screen, makes it difficult (but not impossible) to change the present index is intelligent, and does not require the user to information after it has been recorded, without re-recording mark an index location and explicitly program the VCR to entire frames, because the videotext uses the video channel, shuttle to that location. Rather, the index is content based. during non-visible scan periods thereof. The video recording Another advantage of the present library function is that it system according to the present invention preferably faith can automatically switch media and recording format, pro fully records all transmitted information, including SAP. viding an adaptive and/or multimode recording system. VAR, close caption and videotext information, which may Such a system might be used, for example, if a user wishes be used to implement the various functions. to record, e.g., “The Tonight Show With Johnny Carson” in highly compressed form, e.g. MPEG-2 at 200:1 compres 1061. The use of on-line database listings may be used by sion, except during the performance of a musical guest, at the present interface to provide information to be down which time the recording should have a much lower loss, loaded and incorporated in the index entry of the library e.g., MPEG-2 at 20:1, or in analog format uncompressed. A function, and may also be used as part of the intelligent normal VCR could hardly be used to implement such a determination of the content of a broadcast. This information function even manually, because the tape speed (the analogy may further be used for explicitly programming the interface of quality level) cannot generally be changed in mid record by the user, in that the user may be explicitly presented with ing. The present system could recognize the desired special the available choices available from the database. segment, record it as desired, and indicate the specific parameters on the information directory. The recorded infor EXAMPLE 4 mation may then be retrieved sequentially, as in a normal VCR, or the desired selection may be preferentially 1062) Controlled Encryption and Accounting System retrieved. If the interface of the present invention is set to 1063) The present invention also allows for scrambling, automatically record such special requests, the catalog sec encryption and locking of source material, and the receiving tion would then be available for the user to indicate which device selectively implements an inverse process or a partial selections were recorded based upon the implicit request of inverse process for descrambling, decryption or unlocking the user. Because the interface has the ability to characterize of the material, much as the Videocipher series systems from US 2007/0016476 A1 Jan. 18, 2007

General Instruments, and the fractal enciphering methods of ler could then act as the “telecommunication center” and Entertainment Made Convenient Inc. (EMC, and related authorize decryption of the material. companies, e.g., EMC, and Iterated Systems, Inc. The present invention, however, is not limited to broadcasts, and 1069. If the source signal is in digital form, a serial copy instead could implement a system for both broadcasts and management scheme system is preferably implemented. prerecorded materials. In the case of copying from one tape 1070 The present invention is advantageous in this appli to another, such a system could not only provide the herein cation because it provides an advanced user interface for mentioned library functions of the present invention accord creating a program (.e. a sequence of instructions), and it ing to Example 2, it could also be used to aid in copy assists the user in selecting from the available programs, protection, serial copy management, and a pay-per-view without having presented the user with a detailed description royalty collection system. of the programs, i.e., the user may select the choice based on 1064 Such a system could be implemented by way of a characteristics rather than literal description. telecommunication function incorporated in the device, 1071. In the case of encrypted program source material, shown as block 1808 of FIG. 18, or an electronic tag which it is particularly advantageous if the characterization of the records user activity relating to a tape or the like. Such tags program occurs without charging the account of the user for might take the form of a smart card, PCMCIA device, or Such characterization, and only charging the account if the other type of storage device. A royalty fee, etc., could program is viewed by the user. The user may make a viewing automatically be registered to the machine either by tele decision based on the recommendation of the interface communication or registry with the electronic tag, allowing system, or may review the decision based on the title or new viewer options to be provided as compared with present description of the program, or after a limited duration of VCRs. viewing. Security of the system could then be ensured by a 1065. Numerous digital data encryption and decryption two level encryption system, wherein the initial decryption systems are known. These include DES, “Clipper, elliptic allows for significant processing, but not comfort able key algorithms, public key/private key (RSA, etc.), PGP and viewing, while the second level of decryption allows view others. Digital encryption allows a sender to scramble a ing, and is linked to the accounting system. Alternatively, the message so that, with an arbitrary degree of difficulty, the decryption may be performed so that certain information, message cannot be determined without use of a decryption less than the entirety, is available in a first decryption mode, key. while other information comprising the broadcast informa tion is available in a second decryption mode. 1066 An encrypted tape or other source material may be decrypted with a decryption key available by telecommuni 1072 The transmission encryption system may be of any cation with a communication center, remote from the user, type, but for sensitive material, i.e. where mere distortion of in a decryption unit, shown schematically as the decrypt unit the material (e.g., loss of synchronization information and 1806a of FIG. 18. Such an encryption/decryption scheme phase distortion) would be insufficient, an analog multiple requires special playback equipment, or at least equipment Subband transform, with spread spectrum band hopping and with decryption functionality, and thus any usage or digital encryption of various control signals, would provide decrypted data may be registered as a result of the require a system which would be particularly difficult for the user to ment to receive a decryption key. The decryption unit may view without authorization, and could be effectively imple be part of an addressable remote unit for control of the unit mented with conventionally available technology. The frac remotely. tal compression and encryption of the EMC and Iterated Systems, Inc. system is also possible, in instances where the 1067 During acquisition of the electronic decryption key, broadcast may be precompressed prior to broadcast and the a VCR device of an embodiment of the present invention transmission system supports digital data. Of course, if a would indicate its identity or electronic address, and an digital storage format is employed, a strict digital encryption account is charged a fee for Such use. The negotiation for the system of known type may be used, such as those available electronic key is also preferably encrypted. In addition, the from RSA. The implementation of these encryption systems decryption key may be specific for a particular decoder. is known to those skilled in the art. These may include the Such a system could also be used for controlled access National Bureau of Standards (NBS), Verifiable Secret Shar Software, for example for a computer, wherein a remote ing (VSS) and National Security Agency (NSA) encryption account is charged for use of the Software. Information standards, as well as various proprietary standards. communication may be through the Internet or through an on-line service Such as America Online or CompuServe. EXAMPLE 5 1068 Such a system differs from the normal hardware “key” or “dongle' (device which attaches to standard hard 1073). User Interface ware port for authentication and usage limitation) because it 1074. In one embodiment of the present invention, the requires on-line or electronic access for an encryption key, apparatus comprises a program entry device for a VCR or which may offer different levels of use. It also differs from other type of media recording system. The human interface a call-in registration, because of the automatic nature of the element has an infrared device to allow wireless communi telecommunication. This presently described system differs cation between the human interface device and the VCR from normal pay-per-view techniques because it allows, in apparatus proper. The human interface device also includes certain instances, the user to schedule the viewing. Finally, a direct-manipulation type input device. Such as a trackball with an encryption function implemented in the VCR, the or joystick. Of course it is understood that various known or device allows a user to create and distribute custom "soft to-be developed alternatives can be employed, as described ware' or program material. In addition, the present control above. US 2007/0016476 A1 Jan. 18, 2007 56

1075. It is noted that many present devices, intended for ing in case of difficulty. However, forcing all users to always use in computers having graphic interfaces, would advan work in a default mode limits the improvements in produc tageously make use of an input device which is accessible, tivity that may be gained by a data-sensitive processing without the necessity of moving the users hands from the system, and hence this standardization for its own sake is keyboard. Thus, for example, Electronic Engineering Times rejected by the present invention. (EET), Oct. 28, 1991, p. 62, discloses a miniature joystick 1079. The improvements to be gained by using an intel incorporated into the functional area of the keyboard. This ligent data analysis interface for facilitating user control and technique is directed at a different aspect of user interaction operation of the system are more than merely reducing the with a programmable device than certain preferred embodi average number of keystrokes or time to access a given ments of the present invention, in that the input device does function. Initial presentation of all available information to not have a minimal number of keys. While the device a new user might be too large an information load, leading disclosed in EET is intended for use in a full function to inefficiency, increased search time and errors. Rather, the keyboard, the preferred embodiment of the present invention improvements arise from providing a means for access of is directed towards the minimization of the number of keys and availability to functions not necessarily known to the and avoidance of Superfluous keys by provision of a pointing user, and to therefore improve the perceived quality of the device. Of course, the present invention could be used with product. The system to determine the sophistication of the a full function input device, where appropriate, and the user includes a number of storage registers, for storing an joystick of EET (Oct. 28, 1991, p. 62) would be suitable in analysis of each act for each user. A given act is represented this case. in a plurality of the registers, and a weighting system to 1076 The interface of the present invention studies the ensure that even though an act is represented in a number of behavior and moods of the user, in context, during interac registers, it is not given undue emphasis in the analysis. tions to determine the expected user level of that user as well Thus, each act of the user may be characterized in a number as the preferences of the user. These user characteristics may of ways, and each characteristic stored in an appropriate change over time and circumstances. This means that the register, along with a weighting representing an importance system studies the interaction of the user to determine the of the particular characteristic, in relation to other identified skill of the user or his or her familiarity with the operation characteristics and in relation to the importance of the act as and functionality of the system. By determining the skill of a whole. The act is considered in context, and therefore, the the user, the system may provide a best compromise. The stored information relates to the act, the sequence of acts purpose of this feature is to provide a tailored interface prior to the act, acts of the user occur after the act, the results adapted to the characteristics of the user, thus adaptively of the sequence of acts which include the act, and charac providing access to various features in a hierarchical manner teristics of the user which are not “acts', but rather include such that a most likely feature to be used is more easily timing, mouse path efficiency, and an interaction with other accessible than an unlikely feature, but that features can USCS. generally be accessed from all or most user levels. The user 1080. An apparatus for performing a path information or level analysis also allows the system to teach the user of the efficiency determining function is shown schematically in various functions available, particularly when it becomes FIG. 18, and in more detain in FIG. 21. Thus, for example, apparent that the user is being inefficient in the use of the if a characteristic of the user is an unsteady hand while using system to perform a given task. Therefore, the menu struc the cursor control device, e.g. mouse, producing a high ture may also be adaptive to the particular task being frequency or oscillating component, the existence of this performed by the user. When combined with the user level characteristic is detected and quantified by the high fre analysis feature, the user efficiency feature will provide a quency signal component detector 2112, and, depending on preferable interface, with reduced learning time and the amplitude, frequency and duration (e.g. path length), increased usability for a variety of users. may also be detected by the path optimization detector 2105. 1077 Thus, an important concept is that the system has Once this characteristic is detected and quantified, an adap at least one object having a plurality of functions, certain of tive filter may be applied by the main control 1806 to which are unnecessary or are rarely used for various appli selectively remove the detected component from the signal, cations or in various contexts, while these are used with in order to improve the reliability of the detection of other greater frequency in other contexts. Further, based upon characteristics and to determine the intended act of the user. predetermined protocols and learned patterns, it is possible 1081. It should be noted that the various characteristic to predict which functions will be used and which will not filters preferably act in “parallel at each stage of the be used. characteristic recognition, meaning that one characteristic is 1078. Therefore, the system, upon recognizing a context, defined simultaneously with the detection of other charac will reconfigure the availability or ease of availability of teristics, which assists in resolving ambiguities, allows for functions and allow various Subsets to be used through parallel processing by a plurality of processing elements “shortcuts”. Thus, to some extent, the interface structure which improves real-time recognition speed, and allows a may vary from time to time based upon the use of the probability-based analysis to proceed efficiently. Such a system. The prior art apparently teaches away from this "parallel computation system is included in a neural net concept, because it is believed to prevent standardization, computer, and a hardware-implementation of a neural net/ limits the “recordability” of macros and/or instruction sheets fuZZy logic hybrid computer is a preferred embodiment, for casual users and limits the availability of technical which allows fuzzy rules to be programmed to provide Support. Each of these can be addressed, to Some extent by explicit control over the functioning of the system. It is the availability of a default mode (so that users can access preferred that a human programmer determine the basic all information), and because the interface is self-simplify rules of operation of the system, prior to allowing a back US 2007/0016476 A1 Jan. 18, 2007 57 propagation of errors learning algorithm to improve and Sounds. The set is simple, and the broadcast often shows a adapt the operation of the system. head and neck, or full body shot with a bland background, 1082) The adaptive system implemented according to the likely with fewer enduring props. A signal processing com present invention, by detecting a user level, allows a novice puter, programmed for audio and/or video recognition, is user to productively interact with the system while not provided to differentiate between at least the two types with unnecessarily limiting the use of the adaptive interface by an Some degree of efficiency, and with a possibly extended advanced user, who, for example, wishes to move the cursor sampling time, have a recognition accuracy, such that, when quickly without the limiting effects of a filter which slows this information is integrated with other available informa cursor response. tion, a reliable decision may be made. The required level of reliability, of course, will depend on the particular applica 1083. Another example of the use of an adaptive user tion and a cost-benefit analysis for the required system to interface level is a user who repeatedly requests “help' or implement the decision-making system. user instructions, through the explicit help request detector 2115, which causes an output from the current help level 1088 Since the system according to the present invention output 2102; such a user may benefit from an automatic need not display perfect accuracy, the preferred embodiment context-sensitive help system, however Such a system may according to the present example applies general principles interfere with an advanced user, and is unnecessary in that to new situations and receives user or other feedback as to case and should be avoided. This adaptive user interface the appropriateness of a given decision. Based on this level concept is not limited to a particular embodiment of the feedback, Subsequent encounters with the same or similar present invention, Such as a VCR, and in fact, may be data sets will produce a result which is “closer to an optimal broadly used wherever a system includes an interface which decision. Therefore, with the aid of feedback, the search is intended for use by both experienced and inexperienced criterion would be improved. Thus, a user could teach the users. This differs from normal help systems which must be interface through trial and error to record the desired broad specifically requested, or “balloon help' (Apple Computer, cast programs. Thus, the presently described recognition Macintosh System 7.0. 7.1, 7.5) which is either engaged or algorithms may be adaptive and learning, and need not apply disengaged, but not adaptive to the particular situation based a finite set of predetermined rules in operation. For such a on an implicit request or predicted need. In the case of a learning task, a neural network processor may be imple single user or group of users, the interface could maintain a mented, as known in the art. history of feature usage for each user, as in the past user history block 2107, and provide a lower user interface level 1089. The feature extraction and correlation system for those features which are rarely used, and therefore less according to the present invention is shown in FIG. 22. In familiar to the user, through the current user level output this figure, the multimedia input, including the audio signal 2101 and all other available data, are input in the video input 2201. The video portion is transferred to a frame buffer 1084. It should be noted that the present system prefer 2202, which temporarily stores all of the information. All ably detects an identity of a user, and therefore differentiates other information in the signal, including audio, VIR, video between different users by an explicit or implicit identifica text, close caption, SAP (second audio program), and over tion system. Therefore, the system may accumulate infor Scan, is preferably stored in a memory, and analyzed as mation regarding users without confusion or intermingling. appropriate. The frame buffer 2202 may have an integral or separate prefiltering component 2203. The filtered signal(s) EXAMPLE 6 are then passed to a feature extractor 2204, which divides the Video frame into a number of features, including movement, 1085 VCR Programming Preference Prediction objects, foreground, background, etc. Further, sequences of 1086. The device according to the present invention is Video frames are analyzed in conjunction with the audio and preferably intelligent. In the case of a VCR, the user could other information, and features relating to the correlation of also input characteristics of the program material that are the video and other information, e.g., correlation of video desired, and characteristics of that program material which and audio, are extracted. Other information is also analyzed is not desired. The device would then, over time, monitor and features extracted, e.g., audio and close caption. All various broadcast choices, and determine which most extracted features relating to the multimedia input are then closely match the criteria, and thus be identified. For passed to a transform engine or multiple engines in parallel, example, if the user prefers “talk-shows, and indicates a 2205. These transform engines 2205 serve to match the dislike for “situation comedies” (“sitcoms'), then the device extracted features with exemplars or standard form tem could scan the various available choices for characteristics plates in the template database 2206. indicative of one or the other type of programming, and perform a correlation to determine the most appropriate 1090. It should be noted that even errors or lack of choice(s). A sitcom, for example, usually has a "laugh track correlation between certain data may provide useful infor during a pause in normal dialogue. The background of a mation. Therefore, a mismatch between audio and close sitcom is often a confined space (a "set’), from different caption or audio and SAP may be indicative of useful perspectives, which has a large number of “props' which information. For non-video information, exemplars or tem plates are patterns which allow identification of an aspect of may be common or unique. This set and the props, however, the signal by comparing the pattern of an unidentified signal may be enduring over the life of a show. with the stored pattern. Thus, the voice patterns of particular 1087. A talk-show, on the other hand, more often relies on persons and audio patterns of particular songs or artists may actual audience reaction (possibly in response to an be stored in a database and employed to identify a source “applause sign), and not prerecorded or synthesized signal. US 2007/0016476 A1 Jan. 18, 2007

1091. The transformed extracted features and the tem movements aid little in recognition, and may be virtually plates are then correlated by a correlator or correlators 2207. ignored, except in the case where a particular person’s The parallelization of implementation of the transforms and mouth movements are distinctive, e.g., Jim Nabors (“Gomer correlators serves to increase the recognition speed of the Pyle'), and Tim Curry (“Rocky Horror Picture Show”). device. It should be understood that appropriate systems for Thus, the complexity and parallelism in the intermediate parallelization are known in the art. For example, the TMS recognition stages may actually simplify the later stages by 320C80, also known as the TI MVP (Texas Instruments allowing more abstract features to be emphasized in the multimedia Video processor) contains four DSP engines and analysis. Animation poses a special example where audio a RISC processor with a floating point unit on a single die. and image data may be separated, due to the generally Aboard including a TMS320C80 is available from General non-physiologic relation between the image and Soundtrack. Imaging Corp., Billerica Mass., the S/IP80, which may be programmed with ProtoPIPE. In addition, a board including 1095. The pattern recognition function of the present a TMS320C80 is also available from Wintriss Engineering invention could be used, in a VCR embodiment according to Corp., San Diego, Calif. Multiple MVP processors may also the present invention to, e.g., to edit commercials out of a be parallelized for additional computing power. The MVP broadcast, either by recognition of characteristics present in may be used to analyze, in parallel, the multimedia input commercials, in general, or by pattern recognition of specific signal and correlate it with stored patterns in a database. In commercials in particular, which are often repeated numer this context, correlation does not necessarily denote a strict ous times at various times of the day, and on various mathematical correlation, but rather indicates a comparison broadcast channels. Therefore, the system may acquire an to determine the “closeness” of an identified portion of unidentified source signal, which may be, for example, a 30 information with an unidentified portion, preferably includ second segment, and compare this with a database of char ing a reliability indicator as well. For neural network-based acteristics of known signals. If the signal does not match any processing, specific hardware accelerators also available, previously known or identified signals, it is then subject to such as from Nestor, Inc. and Intel. Therefore, since there a characterization which may be the same or different than may be multiple recognizable aspects of the unidentified the characterization of the identified signals. The character data, and various degrees or genericness of the characteristic izations of the unidentified signal are then compared to recognized, it is preferred that at this initial stage of the characteristics to be recognized. If the unidentified signal recognition process that the output of the correlators 2207 be meets appropriate criteria, a presumptive generic character a data set, e.g. a matrix, series of pointers, or other arrange ization is made. This characterization is preferably con ment, so that sufficient information is available for higher firmed by a user later, so that a positively identified signal is level processing to allow application of an appropriate added to the database of identified signals; however, under decision process. Of course, if the characteristic to be certain circumstances no confirmation is required. detected is simple and well defined, and the decision-making 1096 Certain media present a recognizable audio or process may be implemented with a simple correlation Video cue when a commercial break has ended. (E.g. often result, then a complex data set output is not required. In fact, sports events, such as the Olympic Games, will have theme the output of the correlator may have a number of different music or distinctive images). The present device need not forms, based on the context of the recognition process. respond immediately to such cues, and may incorporate a delay, which would store the information while a decision is 1092) If, for example, an exact match to an entire frame being made. In the case of a video tape, the delay may be up is sought, partial match information is not particularly to the time between the time of recording and the time of useful, and is ignored in this process. (Of course, since the playback. Further, the temporary storage medium may be system is “self-learning, the processing results may be independent of the pattern recognition system. Thus, a maintained and analyzed for other purposes). If the system, system provided according to the present invention may on the other hand, is analyzing novel data, a full analysis actually include two independent or semi-independent data would likely be necessary including partial results and low streams: the first serving as the desired signal to be stored, correlation results. retaining visually important information, and the second 1093. The outputs of the correlators are input into an providing information for storage relating to the pattern adaptive weighing network 2208, to produce a probability of recognition system, which retains information important for a match between a given feature and a given template. The the recognition process, and may discard this information recognition is completed in an identifier 2209, which pro after the pattern recognition procedure is complete. duces a signal identifying one or more objects in the video 1097 A system which provides a plurality of parallel data frame input. The identifier 2209 also has an output to the streams representing the same source signal may be advan template database 2206, which reinforces the recognition by tageous because is allows a broadcast quality temporary providing feedback; therefore, if the same object appears storage, which may be analog in nature, to be separate from again, it will be more easily recognized. The template the signal processing and pattern recognition stage, which database 2206 therefore also has an input from the feature may be of any type, including digital, optical, analog or extractor 2204, which provides it with information regarding other known types, which need only retain significant infor the features recognized. It is also noted that, in addition to mation for the pattern recognition, and therefore may be allowing recognition, the parallel transform engines 2205. highly compressed (e.g. lossy compression), and devoid of correlators 2207, and adaptive weighing network 2208 also various types of information which are irrelevant or of little allows the system to ignore features that, though complex, importance to the pattern recognition functions. Further, the do not aid in recognition. temporary storage may employ a different image compres 1094 For example, during dialogue, the soundtrack voice sion algorithm, e.g. MPEG-4, MPEG-2 or MPEG-1, which may correlate with the mouth movements. Thus, the mouth is optimized for retention of visually important information, US 2007/0016476 A1 Jan. 18, 2007 59 while the recognition system may use a compression system the fractal method could be used as a part of the image optimized for pattern recognition, which may retain infor recognition system. Of course, it should be appreciated that mation relevant to the recognition function which is lost in other fractal processing methods are available and may be other compression systems, while discarding other informa likewise employed. tion which would be visually important. Advantageously, however, the analysis and content transmission streams are 1102 Audio data is also compressible by means of fractal closely related or consolidated, such as MPEG-7 and transforms. It is noted that the audio compression and image MPEG-4 recognition functions cannot be performed on the FTC-II board, and therefore an alternate system must be employed 1098. In a particularly advantageous arrangement, the in order to apply the pattern recognition aspects of the compression algorithm is integral to the recognition func present invention. It should also be noted that an even more tion, preparing the data for the pattern matching and char efficient compression-pattern recognition system could be acterization, and therefore is optimized for high throughput. constructed by using the fractal compression method in According to this embodiment, the initial compression may conjunction with other compression methods, which may be include redundant or uncompressed information, if neces more efficient under certain circumstances, such as discrete sary in order to achieve real-time or near real-time recog cosine transform (DCT), e.g. JPEG or modified JPEG or nition, and, thus may actually result in a larger intermediate wavelet techniques. Fractal compression systems are also data storage requirement than the instantaneous data pre available from other sources, e.g. the method of Greenwood sented to the recognition system; however, the term "com et al., Netrologic Inc., San Diego, Calif. See also, Shepard, pression', in this case, applies to the long term or steady J. D., “Tapping the Potential of Data Compression, Military state status of the device, and in a real-time recognition and Aerospace Electronics, May 17, 1993, pp. 25-27. function, the amount of data stored for use in recognition is preferably less than the cumulative amount of data pre 1103) A preferred method for compressing audio infor sented, except during the very initial stages of data acqui mation includes a model-based compression system. This sition and possibly rare peaks. system may retain stored samples, or derive these from the data stream. The system preferably also includes high-level 1099] In the case where a high quality (low loss, e.g. models of the human Vocal tract and Vocalizations, as well broadcast quality) intermediate storage is employed, after a as common musical instruments. This system therefore decision is made as to whether the data should be stored stores information in a manner which allows faithful repro permanently or otherwise further processed or distributed, duction of the audio content and also provides emphasis on the data may be transferred to the appropriate system or the information-conveying structure of the audio signal. Subsystem of the apparatus. Alternatively, the high quality Thus, a preferred compression for audio signals retains, in intermediate storage is retained, and no further processing is readily available form, information important in a pattern performed. In either case, the purpose of this storage is to recognition system to determine an abstract information buffer the source data until the computational latency content, as well as to allow pattern matching. Of course, a resolves any decisions which must be made. dual data stream approach may also be applied, and other known compression methods may be employed. 1100 According to one aspect of the present invention, the source image may be compressed using the so called 1104) Because of the high complexity of describing a “fractal transform, using the method of Barnsley and Sloan, particular signal pattern or group of audio or image patterns, which is implemented and available as a hardware accel in general, the system will learn by example, with a simple erator in product form from Iterated Systems, Inc., Norcross, identification of a desired or undesired pattern allowing Ga., as the Fractal Transform Card (FTC) II, which incor analysis of the entire pattern, and extraction of characteris porates eight fractal transform integrated circuit chips, 1 tics thereof for use in preference determination. MByte of Random Access Memory (RM), and an Intel i80960CA-25 UP, and operates in conjunction with 1105 Barnsley and Sloans method for automatically POEMTM (Iterated Systems, Inc., Norcross, Ga.) software, processing digital image data consisting of image informa which operates under MicroSoft-Disk Operating System tion, disclosed in U.S. Pat. Nos. 5,065,447 and 4,941,193, (MS-DOS). FTC-II hardware compression requires approxi both expressly incorporated herein by reference, consists of mately 1 second per frame, while Software decompression the steps of storing the image data in the data processor, then on an Intel 80486-25 based MS-DOS computer, using generating a plurality of uniquely addressable domain “Fractal Formatter software, can be performed at about 30 blocks from the stored image data, each of the domain frames per second, which allows approximately real time blocks representing a different portion of the image infor viewing. The Fractal Video Pro 1.5 is a video codec for mation Such that all of the image information is contained in WIN, allowing software only playback at 15-30 fps, 70-150 at least one of the domain blocks. A plurality of uniquely bytes/sec. This is a non-symmetrical algorithm, requiring addressable mapped range blocks corresponding to different more processing to compress than to decompress the image. Subsets of the stored image data are created, from the stored The FTC-IV Compression Accelerator Board is presently image data, with each of the Subsets having a unique address. This step includes the Substep of executing, for each available. of the mapped range blocks, a corresponding procedure 1101 This fractal compression method potentially allows upon the one of the subsets of the stored image data which data compression of upwards of 2000:1, while still main corresponds to the mapped range block. Unique identifiers taining an aesthetically acceptable decompressed image are then assigned to corresponding ones of the mapped range result. Further, since the method emphasizes structural blocks, each of the identifiers specifying for the correspond aspects of the image, as opposed to the frequency decom ing mapped range block a procedure and an address of the position used in DCT methods (JPEG, MPEG), elements of corresponding Subset of the stored image data. For each of US 2007/0016476 A1 Jan. 18, 2007 60 the domain blocks, the one of the mapped range blocks Cohen, R., “FullPixelSearch Helps Users Locate Graphics”, which most closely corresponds according to predetermined MacWeek, Aug. 23, 1993, p. 77. criteria is selected. Finally, the image information is repre sented as a set of the identifiers of the selected mapped range 1110 Image processing hardware and systems are also blocks. This method allows a fractal compression of image avail ale from Macron, Nashua N.H., Coreco, St. Laurent, data. In particular, Drs. Barnsley and Sloan have optimized Quebec; Analogic, and others. the match of the domain blocks with the mapping region by 1111) A fractal-based system for real-time video com minimizing the Hausdorff distance. A decompression of the pression, satellite broadcasting and decompression is also data precedes analogously in reverse order starting with the known from Iterated Systems, Inc. and Entertainment Made identifiers and the mapping regions to produce a facsimile of Convenient, Inc. (EMC). In such a system, since the the original image. This system is highly asymmetric, and compressed signal is transmitted, the remote receiving sys requires significantly more processing to compress than to tem need not necessarily complete decompression prior to decompress. Barnsley and Sloan do not suggest a method for the intelligent pattern recognition function of the present using the fractal compression to facilitate image recognition, invention. This system also incorporates anti-copy encryp which is a part of the present invention. tion and royalty and accounting documentation systems. It is 1106 Basically, the fractal method proceeds from an noted that the EMC system does not incorporate the intel understanding that real images are made up of a plurality of ligent features of the present invention. like Sub components, varying in size, orientation, etc. Thus, 1112 A preferred fractal-based system according to the a complex block of data may be described by reference to present information provides the source data preprocessed to the Subcomponent, the size, orientation, etc. of the block. allow easy and efficient extraction of information. While The entire image may thus be described as the composite of much precharacterization information may be provided the Sub-images. This is what is meant by iterative function explicitly, the preferred system allows other, unindexed systems, where first a largest block is identified, and the information to also be extracted from the signal. Further, the pattern mapping is repetitively performed to describe the preferred system provides for an accounting system which entire image. facilitates pay-per-view functions. Thus, the interface of the 1107] The Iterated Systems, Inc. FTC-II or FTC-IV present invention could interact with the standard account board, if applied as a part of a system according to the ing system to allow royalty-based recording or viewing, and present invention, is preferably used in conjunction with a possibly implement a serial-copy recording prevention sys frame-grabber board, such as MatroX, Quebec, Canada, tem. Prior art systems require a user to explicitly select a Image-LC board, or a Data Translation DT 1451, DT2651, program, rather than allow an intelligent system to assist in DT2862, DT2867, DT2861 or DT2871, which may perform selection and programming of the device. The EMC system additional functions, such as preprocessing of the image is described in “EMC Pushes Video Rental By Satellite”, signal, and may be further used in conjunction with an image Electronic Engineering Times, Dec. 2, 1991, p. 1, p. 98. See processing system, such as the Data Translation DT2878. Of also, Yoshida, J., “The Video-on-demand Demand, Elec course, it should be understood that any suitable hardware, tronic Engineering Times, Mar. 15, 1993, pp. 1, 72. for capturing, processing and storing the input signals, up to 1113 Fractal techniques may be used to store images on and including the state of the art, may be incorporated in a a writable mass storage medium, e.g. CD-ROM compatible. system according to the present invention without exceeding The present system may thus be used to selectively access the scope hereof, as the present invention is not dependent data on the CD-ROM by analyzing the images, without on any particular Su System, and may make use of the latest requiring full decompression of the image data. advances. For example, many modern systems provide appropriate functionality for digital video capture, either 1114 Wavelets hold promise for efficiently describing uncompressed, mildly compressed, or with a high degree of images (i.e., compressing the data) while describing mor compression, e.g., MPEG-2. phological features of the image. However, in contrast to wavelet transforms which are not intended to specifically 1108). The Texas Instruments TMS320C80 provides a retain morphological information, the selection of the par Substantial amount of computing power and is a preferred ticular wavelet and the organization of the algorithm will processor for certain computationally intensive operations likely differ. In this case, the transform will likely be more involving digital signal processing algorithms. A system computationally complex and therefore slower, while the employing a parallel TMS320C40 processors may also be actual compression ratios achieved may be greater. used. The Intel Pentium series (or related processors from AMD, National Semiconductor, or other companies), DEC/ 1115 Thus, one embodiment of the device according to Compaq Alpha, SPARC, or other processors intended for the present invention may incorporate a memory for storing desktop computing may, either individually or in multipro a program, before being transferred to a permanent storage cessor configurations, be used to process signals. facility, Such as tape. Such a memory may include a hard disk drive, magnetic tape loop, a rewritable optical disk 1109) A pattern recognition database system is available drive, or semiconductor memories, including Such devices from Excalibur Technologies, San Diego, Calif. Further, as wafer scale memory devices. This is shown diagrammati IBM has had pattern recognition functionality available for cally as the intermediate storage 2210 of FIG. 22. The its DB/2 database system, and has licensed Excalibur's XRS capacity of Such a device may be effectively increased image retriever recognition software for DB/2. See, Lu, C., through the use of image data compression, which may be “Publish It Electronically, Byte, September 1993, pp. proprietary or a standard format, i.e. MPEG-1, MPEG-2 94-109. Apple Computer has included search by sketch and (Motion Picture Experts Group standard employing DCT search by example functions in PhotoFlash 2.0. See also, encoding of frames and interframe coding), MPEG-4 US 2007/0016476 A1 Jan. 18, 2007

(Motion Picture Experts Group standard employing DCT be implemented by various known means. A compressed encoding of frames and interframe coding, as well as model image may also be advantageously used in conjunction with based encoding methods) JPEG (Joint Photographic Experts the image recognition system of the present invention, as Group standard employing DCT encoding of frames), Px64 described above. In Such a case, the compression system (Comité Consultatif International des Telegraph et telephone would retain the information most important in the recog (International telegraph and telephone consultative commit nition function, and truncate the unimportant information. tee) (CCITT) standard H.261, videoconferencing transmis 1121. A further method of performing pattern recogni sion standard), DVI (Digital Video Interactive), CDI (Com tion, especially of two dimensional patterns, is optical pact Disk Interactive), etc. pattern recognition, where an image is correlated with a set 1116 Standard devices are available for processing such of known image patterns represented on a hologram, and the signals, available from 8x8, Inc., C-Cube, Royal Philips product is a pattern according to a correlation between the Electronics (TriMedia), and other companies. Image pro input pattern and the provided known patterns. Because this cessing algorithms may also be executed on general purpose is an optical technique, it is performed nearly instanta microprocessor devices. neously, and the output information can be reentered into an 1117 Older designs include the Integrated Information electronic digital computer through optical transducers Technology, Inc. (IIT, now 8x8, Inc.) Vision Processor (VP) known in the art. Such a system is described in Casasent, D., chip, Integrated Information Technology Inc., Santa Clara, Photonics Spectra, November 1991, pp. 134-140. See also Calif., the C-Cube CL550B (JPEG) and CL950 (MPEG references cited therein. decoding), SGS-Thompson ST13220, STV3200, STV3208 1122. These optical recognition systems are best suited to (JPEG, MPEG, Px64), LSI Logic L64735, L64745 and applications where an uncharacterized input signal frame is L64765 (JPEG) and Px64 chip sets, and the Intel Corp. to be compared to a finite number of visually different i750B DVI processor sets (82750PB, 82750DB). Various comparison frames (i.e., at least one, with an upper limit alternative image processing chips have been available as generally defined by the physical limitations of the optical single chips and chip sets; in board level products, such as storage media and the system for interfacing to the storage the Super Motion Compression and Super Still-Frame Com media), and where an optical correlation will provide useful pression by New Media Graphics of Billerica, Mass., for the information. Thus, if a user wished to detect one of e.g., Personal Computer-Advanced technology (PC-AT, an IBM “David Letterman”, “Jay Leno', or “David Koppel', a created computer standard) bus; Optibase, Canoga Park, number of different planar views, or holograms in differing Calif. (Motorola Digital Signal Processor (DSP) with dedi poses, of these persons would be formed as a holographic cated processor for MPEG); NuVista-- from Truevision correlation matrix, which could be Superimposed as a mul (Macintosh video capture and output); New Video Corp. tiple exposure, stacked in the width dimension, or placed in (Venice, Calif.) EyeC Delivery board for Macintosh NuBus a planar matrix, side by side. The detection system produces, systems (DVI); Intel Corp. ActionMedia II boards for from the uncharacterized input image and the holographic Microsoft Windows and IBM OS/2 in Industry Standard matrix, a wavefront pattern that is detectable by photonic Adapter (ISA the IBM-PC bus standard for 8 (PC) or 16 bit SSOS. (PC-AT) slots); Micro Channel Architecture (MCA) (e.g., Digital Video Interactive (DVI), Presentation Level Video 1123. It is preferred that if multiple holographic images (PLV) 2.0, RealTime Video (RTV) 2.0) based machines; and of a particular characterization are employed, that they each as complete products, such as MediaStation by VideoLogic. produce a more similar resulting wavefront pattern than the holographic images of other characterizations, in order to 1118 Programmable devices, including the Texas Instru enhance detection efficiency. The optical pattern recognition ments TMS320C80 MVP (multimedia video processor) may method is limited in that a holographic image must be be used to process information according to standard meth prepared of the desired pattern to be detected, and that ods, and further provide the advantage of customizability of optically similar images might actually be of a different the methods employed. Various available DSP chips, exem image, if the differences are subtle. However, this method plary board level signal processing products and available may be used in conjunction with electronic digital pattern software are described in more detail in '32-bit Floating recognition methods, to obtain the advantages of both. Point DSP Processors”, EDN, Nov. 7, 1991, pp. 127-146. Methods are also known to electronically write an image to The TMS320C80 includes four DSP elements and a RISC a holographic storage medium, thereby facilitating its use in processor with a floating point unit. a general-purpose image recognition system. Of course, the 1119. It is noted that the present interface does not depend system may also be used to identify talk show guests, such on a particular compression format or storage medium, so as “Richard Gere' or “Cindy Crawford', or these same that any suitable format may be used. The following refer individuals in other contexts. ences describe various video compression hardware: Kim, 1124. If image compression is used, once an image is Y., “Chips Deliver Multimedia, Byte, December 1991, pp. compressed, it need not be decompressed and returned to 163-173; and Donovan, J., “Intel/IBM's Audio-Video Ker pixel, NTSC or other standard transmission or format for nel, Byte, December, 1991, pp. 177-202. storage on tape, and thus the compressed image information 1120. It should also be noted that the data compression may be stored in the same format as is present in the algorithm applied for storage of the received data may be temporary storage medium. Thus, the block labeled inter lossless or lossy, depending on the application. Various mediate processing 2211 of FIG. 22 shows that the inter different methods and paradigms may be used. For example, mediate storage need not retain the information as received DCT (discrete cosine transform) based methods, wavelets, from the frame buffer 2202, and in fact, may prepare it for fractals, and other known methods may be used. These may the feature extractor 2204. In addition, the storage medium US 2007/0016476 A1 Jan. 18, 2007 62 itself need not be normal videotape (S-VHS, VHS, Beta, 8 1129. The present embodiment was constructed and mm, Hi-8) and may be an adapted analog storage technique tested using HyperPADTM, a rapid prototyping package for or a digital storage technique. Various magneto-optical an IBM-PC Compatible Computer. It is, of course obvious recording techniques are known, which can store between that the present embodiment could be incorporated in a 128 MB (33.4") and around 5 GB (11"), uncompressed, which might be Suitable for storing compressed digital or analog commercial VCR machine by those skilled in the art, or be information. Multilayer CD-ROM and short wavelength implemented on many types of general purpose computers (e.g., blue) laser systems allow storage densities of about 3.5 with output screens which allow on-screen feedback for the to 10 Gbytes per disk, allowing storage of over two hours of programming operation. Further, the system of the present MPEG-2 encoded video. embodiment can include a remote-control device which communicates with a VCR through an infrared beam or 1125. It is also noted that the present technology could beams, and can thus exert control over an infrared remote also be applied to any sort of mass storage. Such as for a controlled VCR, or translate the programming information personal computer. In Such a case, a characteristic of the and communicate through an infrared remote control, using computer file, which is analogous to the broadcast program the standard type infrared transmitter. in temporary storage of a VCR, is classified according to Some criteria, which may be explicit, such as an explicit 1130. An IBM PC-AT compatible (MS-DOS, Intel header or identifying information, or implicit, such as a 80286-10 MHz) computer was used to test the two simula document in letter format, or a memorandum, as well as by tions. In order to simulate the use of a remote control device words and word proximity. In particular, Such a recognition in programming the VCR, an infrared device made by system could differentiate various clients or authors based NViewTM was attached to the computer. This device came on the content of the document, and these could be stored in with a keyboard that was used to “teach” a MemorexTM different manners. The text analysis system of a text-based computer storage system is analogous to the program clas Universal Remote so that the desired actions could be sification system of the VCR embodiment of the present obtained. By using a universal remote, the computer could invention. However, there is a further analogy, in that the be controlled by using a remote control. VCR could incorporate optical character recognition of text 1131 The present embodiment incorporates a mouse displayed in the program material, employ voice recogni input device. It is understood that a small trackball with a tion, or directly receive text information as a part of a closed button for selection, mounted on a remote control may also caption or videotext system. Thus, the VCR device accord be employed, and may be preferable in certain circum ing to the present invention could recognize and classify stances. However, a computer mouse is easily available, and programs based on textual cues, and make decisions based on these cues. This might also provide a simple method of the mouse and trackball data are essentially similar for the discriminating program material, for example, if a commer type of task implemented by the user, with trackball perfor cial does not include close caption or Second Audio Program mance being slightly faster. For daily use on a VCR how (SAP), while the desired program does, or vice versa, then ever, a trackball would be a more preferable input device a commercial could be discriminated from a program with because it does not require a hard, flat surface, which is not very little computational expenditure. always available to a user when programming a VCR. Such as in the situation where a person is watching television EXAMPLE 7 while sitting in a chair or sofa. 1126 VCR Interface 1132 A GeniusTM Mouse was used as the input device in the prototype of the interface of the present invention. With 1127 Aparticular VCR interface system according to one the mouse, the user could view all of the choices at once on aspect of the present invention includes an internal clock, the display Screen, and then make a selection from the items four program memory, and the capability to display a on the screen by moving the cursor and then pressing the left graphical color interface. By providing the user with the aforementioned features, this design is a unique implemen mouse button. tation for an instrument to be used for programming an event 1133. The interface of the present example focuses on driven controller via an interactive display. All information attending to the user's needs, and the interface must be that the user needs is displayed on the screen to avoid or modified for each application. By reducing the searching, minimize the unnecessary searching for information. This learning times, and entry times, the mental load is also information includes the current date and current time. minimized. Some tradeoffs are necessary as a result of 1128) A simulation of the AKAI Inc. VCR VS303U subjective and objective data. Because of the difficulty in (on-screen programming) and the interface of the present optimizing a single interface design for all levels of users, a invention, were tested to evaluate users’ performances. The menu system was used in an attempt to satisfy all these user AKAI interface of the prior art, hereinafter referred to as the types. prior art interface, was chosen because users made the fewest errors while using this machine, and no user quit 1134) The interface of the present example reduced the while programming, as compared to three other VCRs number of incorrect recordings by 50%. The severity of the tested, a Panasonic (made by Matsushita, Inc.) PV4962 (Bar errors is unimportant here because one wrong entry will Coder), an RCA brand (formerly Radio Corporation of cause an irretrievable mistake and the user will not record America, Inc.) VKP950 (on-screen programming), Pana the intended program. One study reported that faulty inputs, sonic brand (made by Matsushita Inc.) PV4700 (Display which lead to missing the program, can be reported by Panel). almost every present day owner of a VCR. US 2007/0016476 A1 Jan. 18, 2007

EXAMPLE 8 recorded by the system), changed in time or programming slot, etc. Thus, the interface could present information to the 1135 Programmable Device Interface user, of which he might not be aware, and/or predict an 1136. It is also noted that the interface of the present action based on that information. Such a device could, if set invention need not be limited to audio-visual and multime in a mode of operation that allows such, automatically dia applications, as similar issues arise in various program execute a sequence of instructions based on a predicted mable controller environments. Such issues are disclosed in course of action. Thus, if a user is to be absent for a period, Carlson, Mark A., “Design Goals for an Effective User he could set the machine to automatically record a show, Interface', Electro/82 Proc., Mar. 1, 2001-Mar. 1, 2004; even if the recording parameters are not known with preci Kreifeldt, John, “Human Factors Approach to Medical Instrument Design, Electro/82 Proc., Mar. 3, 2001-Mar. 3, sion at the time of setting by the user. Of course, this 2006: Wilke, William, “Easy Operation of Instruments by particular embodiment depends on the availability of a Both Man and Machine', Electro/82 Proc., Mar. 2, 2001 database of current broadcast schedules, however, Such a Mar. 2, 2004; Green, Lee, “Thermo Tech: Here’s a common database may generally be available, e.g., in an on-line sense guide to the new thinking thermostats. Popular database. Mechanics, October 1985, 155-159; Moore, T. G. and Dart 1142. Such an on-line database system of known type nall, “Human Factors of a Microelectronic Product: The may be used and need not be described in detail herein. Central Heating Timer/Programmer. Applied Ergonomics, Alternately, a printed Schedule of broadcasts may be scanned 1983, Vol. 13, No. 1, 15-23; and “The Smart House: Human into a computer and the printed information deciphered Factors in Home Automation', Human Factors in Practice, (e.g., OCR) to gain access to a database. Other methods may December 1990, 1-36. also be used to access Scheduling information, e.g. access 1137) This generalized system is shown in FIG. 23, in channels on cable systems, as well as other broadcast which the sensor array 2301 interfaces with a microproces information identifying future and imminent programming. sor 2302 with a serial data port 2302a, which transmits Together, these methods allow semiautonomous operation, sensor data to a control 2303. The control 2303 further guided by programming preferences rather than explicit interfaces or includes a data pattern recognition system 2304 programs, where Such explicit instruction is absent. and an interface and programming console 2305 according 1143) The smart screens according to the present inven to the present invention, using the aforementioned intelli tion may be implemented as follows. The controller may be, gent features and adaptive pattern recognition techniques. for example, an Apple Power Macintosh 8100/110 AV The control 2203 controls the plant 2306, which includes all computer, operating under Macintosh 7.5 operating system. the controlled actuators, etc. The Hypercard 2.3 software may be used to implement the screen interface, which incorporates the above-described EXAMPLE 9 features, which is generally compatible with the Hyperpad 1138) Adaptive Graphic Interface software described above. HyperCardTM is mentioned due to its capabilities to reference external programs, thus allowing 1139. A “smart screen” aspect according to the present interfacing to various software and hardware devices. A invention is further explored in the present example. This more global Scripting language. Such as Frontier by User aspect of the present invention allows the interface to Land Software Inc., may also be used, especially where low anticipate or predict the intent of the user, to provide, as a level hardware control of interfaced devices, such as a VCR, default user choice, the most likely action to be taken by the multimedia adapter, or the like is desired. Apple AppleScript user of the programmable device as a default, which may be may also be used. The Quicktime format may be used to either accepted or rejected by the user, without inordinate store and recall data, however, many acceptable formats delay to the user. The intelligent selection feature may also exist. The input device is an Apple Desktop Bus (ADB) automatically choose an option and execute the selected mouse (Apple Computer Inc., Cupertino, Calif.), and the option, without further intervention, in cases where little or output display is an 8 bit or 24 bit graphics color adapter no harm will result. Examples of such harm include a loss connected to, e.g., a 14" color monitor. In addition, various of data, a Substantial waste of the user's time and an parameters concerning the use of the interface are stored in inappropriate unauthorized allocation of computational the computer's memory, and a non-volatile mass storage SOUCS. device, such as a hard disk drive, or Electrically Erasable 1140. When a user regularly applies the VCR device, for Programmable read Only Memory (EEPROM) or Erasable example, to record a particular television show which Programmable Read Only Memory (EPROM), as well as appears weekly on a given television channel, at a given battery backed Random Access Memmory (RAM) could time, on a given channel. Such an action could be immedi also be used. ately presented to the user as a first option, without forcing 1144. A more modern implementation might employ, for him to explicitly program the entire sequence. Likewise, if example, a single or dual Pentium II 450 MHz workstation, the user has already entered Such a command, the presented running Microsoft Windows NT 4.0 (or Windows 2000, choices could include a second most likely selection, as well when available). The hardware is a matter of choice, includ as the possibility of canceling the previously entered com ing memory, monitor, pointing device, graphic display card, mand. Video capture card, mass storage options, and the like. 1141 Further, if an entire television programming guide Preferably, a hardware codec is provided, for example a for a week or month is available as a database, the interface Media 100, Inc. Broadway device. The software may be, for could actively determine whether the desired show is pre example, Microsoft Visual Basic 5.0 or other suit ale devel empted, a repeat (e.g., one which has been previously opment language. US 2007/0016476 A1 Jan. 18, 2007 64

1145 Intel Pentium-based platforms may also be used, action or operation may be realized. This predicted operation preferably in IBM-PC compatible implementations. Intel is, in the context of the current user interface State, the most 80860 and/or Intel 80960 processor platforms may also be probable next action to be taken by the user. used. 1151. The predicted operation is based on: the identity of 1146 Alternatively, other Apple Power PC, Macintosh the user, if more than one user operates the interface and (MC680X0 series) or IBM Power PC implementation may machine, the information already entered into the interface be used, providing the advantage of increased processing during the present programming session, the presently avail power over Motorola 680X0 derivatives. The specific Power able choices for data entry, settings for the use of the PC employed may be any version, including desktop system machine, which may be present as a result of a "setup versions available from Apple and IBM and embedded operation, settings saved during a prior session, and a versions from IBM and Motorola. These PowerPC proces database of programming choices. In the case of a Hyper sors may also be provided in a parallel processing imple Card Script, the interface Software calls another program mentation. Further, custom implementations of Power PC which has access to the necessary data in the memory, as hardware optimized for the relevant computational tasks well as access to any remote database which may be may be employed. necessary for implementation of the function. Using a pre 1147. Of course, other systems, including DEC Alpha dictive technology, such as Boolean logic, fuZZy logic, and HP 9000 systems may also be employed, as well as neural network logic, or other type of artificial intelligence, SPARC, MIPS, and other available RISC systems. While a most probable choice may be presented to the user for his RISC systems, possibly supplemented with DSP hardware, approval, or another alternative choice may be selected. are presently preferred because of their efficiency in execut Further, a number of most probable choices may be pre ing the pattern recognition tasks, Complex Instruction Set sented simultaneously or in sequence, in order to improve Computer (CISC)., hybrid and other known processing the probability that the user will be immediately or quickly systems may be employed. The Texas Instruments presented with an acceptable choice. If multiple choices are TMS320C80 coloines a Reduced Instruction Set Computer presented, and there is limited room on the display, two (or (RISC) processor, Arithmetic logoc Unit (ALU) and four more) similar choices may be merged into a single menu DSP processors on a single chip, and is therefore a preferred selection, which may be resolved in a secondary menu processor for implementing various aspects of the system, screen, e.g., a Submenu or dialog box. especially mathematical processing including DCT and cor 1152 FIG. 24 shows a system for correlating a user's relations. preferences with a prospective or real-time occurrence of an 1148. According to the present invention, the interface event. The input device 2401, which is a remote control with may perform comparatively simple tasks, such as standard a pointing device. Such as a trackball, provides the user's graphic user interface implementation with optimized pre input to the control 2402. The program is stored in a program sentation of screen options, or include more complex func memory 2403, after it is entered. The control 2402 controls tionality, such as pattern recognition, pattern matching and a plant 2404, which is a VCR. The control also controls an complex user preference correlations. Therefore, hardware on-screen programming interface 2405, through which the requirements will range from basic 68040, 80486, Pentium, user interactively enters the program information. Each Power PC, MIPS, SPARC, Digial Equipment Corp. (DEC, program entry of the user is Submitted to the user history now Compaq Computer Corp.) Alpha, or other micropro database and preferences module 2406, which may also cessors which are used to perform visual or audio interface receive explicit preference information, input by the user functions, to much special purpose processors for imple through the input device 2401. The prospective and real time event characterization unit 2407 uses any and/or all relevant mentation of complex algorithms, including mathematical, information available in order to determine the character of neural network, fuZZy logic, and iterated function systems a signal input, which is a video signal, from the signal (fractals). receiver 2408. A signal analyzer 2409 provides a prelimi 1149. It should be noted that, while many aspects of the nary analysis and characterization of the signal, which is intelligent interface according to the present invention do input to the prospective and real time event characterization not require extremely high levels of processing power, and unit 2407. The prospective and real time event character therefore may be provided with inexpensive and commonly ization unit 2407 also interacts and receives an input from a available computing hardware, other aspects involve com telecommunication module 2410, which in turn interacts and plex pattern recognition and advantageously employ pow receives information from an on-line database 2411. Auser erful processors to achieve a short processing latency. Both preference and event correlator 2412 produces an output simple and complex interface systems, however, are relating to a relatedness of an event or prospective event and included within the scope of the present invention. Process a user preference. In the event of a high correlation or ing may be distributed in different fashions, so that complex relatedness, the control 2402 determines that the event or functionality may be implemented with relatively simple prospective event is a likely or most likely predicted action. local hardware, with a substantial amount of required pro The prospective event discussed above refers to a scheduled cessing for a high level of functionality performed centrally, event, which is likely to occur in the future. The character and for a large number of users. ization unit also has a local database 2413 for storing 1150. From the stored information regarding the prior use schedule information and the like. of the interface by the user, including prior sessions and the 1153. In the particular context of a videotape, one con immediate session, and a current state of the machine sideration of the user is the amount of time remaining on the (including a received data stream and information relating to tape. Generally, users wish to optimally fill a tape without the data stream previously stored), a predicted course of splitting a program, although the optimization and non US 2007/0016476 A1 Jan. 18, 2007

splitting parameters may vary between users. Therefore, the relating to clock function. The interface would then, if the length of the tape and the amount and character of other clock is not properly set, and if there is no telecommunica items on the tape are also factors to be employed in tion or other external means for automatically determining determining a most desired result. With respect to this issue, the exact time, present the user with a menu selection to set the interface may maintain a library function which allows the proper time. Of course, if the correct time is available to the identification of a partially filled tape for recording under the apparatus in Some form, this could be automatically given circumstances. The interface may also optimize a obtained, and the internal clock updated, without interven playback by selecting a tape containing a desired sequence tion. These same sources may be used to Verify the accuracy of materials. of an internal clock. Further, if a reliable external clock 1154. The intelligent interface may also be used as a part system is available, an internal clock may be dispensed with of an educational system, due to its ability to adapt to the or ignored. Time may also be inferred based on the regular level of the user and dynamically alter an information schedules of broadcasts, e.g., the 11:00 p.m. news begins at presentation based on the “user level”, i.e. the training status 11:00 p.m. If the user does not have access to a source of the of the user, and its ability to determine areas of high and low exact time, the step of correcting the time may be deferred, performance. Likewise, the intelligent interface according to although at some point the user should be reminded to verify the present invention may also be used in a business envi the clock information. The user may thus be able to override ronment for use by trained individuals who require relatively a machine-generated request or attempt to correct the time static Software interface design for consistence and “touch data. typing with memorized keystroke or mouse click 1159 If the machine has access to an external source of sequences. In this case, the intelligent functionality is seg the exact time, it would then preferably access this source regated into a separate user interface structure, such as an first. Such sources of exact time include a telephone con additional “pull down menu' or other available screen nection to a voice line which repeats the time. The computer location. While the interface always monitors user perfor would then perform a speech recognition algorithm which mance, the impact of the analysis of the user is selectively would be used to determine the time. Such a speech recog applied. User analysis may also be used for performance nition algorithm could also be used as a part of the user evaluation according to an objective criteria, based on interface for other purposes, i.e. a speech recognition system continuous monitoring. In a network environment, user is not supplied solely for obtaining time information. Alter profile and evaluation may be made portable, stored so as to natively, a modem or communication device could be used be accessible from any networked device the user may to obtain the time in digitally coded form over a network, interact with, from office computers to thermostats to pho which would alleviate the need for speech recognition tocopying machines to coffee machines. capabilities for this function. An on-line connection could also be used in order to obtain information concerning EXAMPLE 10 television scheduling. 1155) Intelligent Adaptive VCR Interface 1160) A further method for obtaining accurate time infor 1156. A user interacting with the device intends to record mation is to access a video signal which contains the desired a particular program, “Married With Children’ (Fox, Sun time information. For example, many cable broadcasting day, 9:00 p.m., etc.) on its ever occurrence. This intent, systems have a channel which continuously broadcasts the however, is to provide a full library of episodes, and not to time in image form. The interface tunes this channel, and duplicate episodes. The particular program is subject to the acquires a representation of the screen image, thereafter occurrence of reruns, syndicated distribution, time shifting performing a character recognition algorithm to capture the of performance, preview scenes and advertisements. Fur time information. This character recognition algorithm ther, various actors appearing in the particular program also could also be used to obtain or capture information regard appear in other capacities and roles on television. Therefore, ing programming schedules, stock prices, and other text after this intent is elucidated, the interface scans available information which may appear on certain cable broadcast directories of programming to determine when “Married channels. With Children' will be broadcast. In addition, to the extent 1161 Thus, the interface, in obtaining necessary infor possible, all channels may be monitored, in the event that the mation, employs Such available data source access methods directories or erroneous or incomplete. as speech recognition, character recognition, digital tele 1157. It is noted that the interface may be quite effective communication means, radio wave reception and interpre if it is used for a number of applications, such as television, tation, and links to other devices. radio, desktop computer, and even kitchen and HVAC system. For example, preferences for processing MTV or 1162. In interacting with the apparatus, the user first identifies himself/herself to the machine, which can occur in other music video information may be directly relevant to a number of ways. This step may be dispensed with, or at processing of radio or other music reproduction devices, and least trivialized, if only one user regularly interacts with the Vice versa. apparatus. Otherwise, such identification may be important 1158. At some point in the process, preferably prior to in order to maintain the integrity of the user profiles and Substantive programming input, the interface performs a predictive aspects of the interface. An radio frequency self-diagnostic check to determine whether the machine is transponder (RF-ID), infrared transponder (IR-ID) system set up and operating correctly. This would include a deter may automatically determine the user based on a devices, mination of whether the clock has been set and thereafter which may be concealed in a piece of jewelry or wristwatch. operating continuously. Of course, the clock could have, in The user may also be identified by Voice pattern recognition, practice, a battery to minimize the occurrence of problems speaker independent voice recognition, video pattern recog US 2007/0016476 A1 Jan. 18, 2007 66 nition, fingerprint, retinal scan, or other biometric evalua inaccurate or inappropriate. The most likely choices are then tion. An explicit entry of the user identity may also be those programs that begin at the selected time. If the user had employed, wherein the user types his/her name on a key input the channel or network, instead of starting time, then board or selects the name or unique identifier from a the presented choices would be the broadcast schedule of the “pick-list'. The interface, upon identifying the user, channel, e.g. channel 5 or Fox, for the selected day. retrieves information regarding the user, which may include 1166 The user then selects one of the available choices, past history of use, user preferences, user Sophistication, which completes the programming sequence. If no database patterns of variation of user, which may be based on, e.g., of broadcasts is available, then the user explicitly defines all time, mood, weather, lighting, biometric factor or other parameters of the broadcast. When the programming is factors. completed, the interface then updates its user database, 1163 Thus, after completing system diagnostics, includ prompts the user to set the VCR to record, by, e.g., inserting ing the time-check function referred to above, the system a blank or recordable tape. next determines or predicts the desired function of the user. 1167) If the predicted desire of the user is of no help, or In this regard, if more than one user has access to the system, the user seeks to explicitly program the system, a manual the user identifies himself to the interface, in a user identi program entry system is available. Where there is no useful fication step 1701 or an analogous action, which may be a prediction of the user, the interface may request a training coded entry, or a selection from the menu. If the interface session, which may be a general inquiry, or specifically has voice recognition capability, then the user may be directed to immediately forthcoming broadcasts, or both. recognized by his voice pattern, or merely by stating his 1168. In this case, after a failure to predict a desired name. The interface then accesses the memory for a profile program, the user then proceeds to explicitly program the of the past use of the machine by the user, which may VCR interface to record “Married with Children on Fox at include the entire prior history, relevant abstracts of the 9:00 p.m. on Sunday evening. If a database is available, it history, or derived user preferences, as shown in the per might also show that “Married with Children' is also sonalized startup based on user profile step 1702, which syndicated in re-runs, and therefore various episodes may be information is also stored and used in the past user history available on other channels at other times. Thus, during the determining element 2107. These choices differ in the Subsequent session, both the premier showing and re-run of amount of storage necessary in order to retain the desired “Married With Children” would be available predicted information. choices, along with the 11 o'clock News on NBC. 1164 Thus, if the user has only used the VCR to record, 1169 The user having demonstrated a preference for e.g., the National Broadcasting Company (NBC) 11 o'clock “Married with Children', the interface then characterizes the news, i.e., record all days from 11:00 p.m. to 11:30 p.m. on program. This includes, for example, a characterization of NBC, in the past, the most likely current predicted choice the soundtrack, the background, foreground, actors and would be the NBC 11 o'clock news. If the interface were to actresses present, credits, etc. The interface then attempts to present a number of choices, having lower probability, then correlate the features present in the reference selection with it interprets the recording history to be “news’ based on a other available selections. This comparison may be with a database of broadcast information. Therefore, a prediction of preformed database, providing immediate results, or pro lower probability would be American Broadcasting Com spectively, after entry of the reference selection. Of course, pany (ABC) or Central Broadcasting Company (CBS) news a number of correlation functions may proceed simulta at, e.g., 11:00 p.m., and the NBC news at, e.g., 5:00 p.m. In neously, and various choices may be merged to form a a cable television system, there may be a number of NBC compound reference selection, any ambiguity in which to be affiliated news alternatives, so that these alternatives may be later resolved. Further, as various “episodes of the refer investigated first before other networks or the like are ence selection occur, the system appends and integrates the presented as likely choices. In addition, where a video feed most recent occurrence with the stored reference informa is unavailable, a text feed from the internet or an on-line tion, thus updating the reference database. service may be acquired as a probable alternative. 1170. When an occurrence is identified, it is immediately 1165 Thus, a number of likely choices, based on intel buffered, until such time as the particular episode may be ligently determined alternatives, as well as adaptation based compared against previously stored episodes. If two identi on determined user preferences, are initially presented to the cal broadcasts occur simultaneously, one may be selected, user, along with a menu selection to allow rejection of these i.e., the one with the best reception. When the episode is predicted choices. In this case, the user selects the “reject’ identified, if it is new, the buffered broadcast information is selection, and the system presents the user with a next permanently stored; if it is previously stored, the buffer is predicted desired menu choice. Since the user history, in this flushed and the occurrence is further ignored as a “hit”. case, does not provide for another choice of particularly high Since the apparatus is now not responding to a direct probability, the user is prompted to explicitly choose the request, it may then perform various housekeeping func program sequence by day, time, channel, and duration. The tions, including updating databases of broadcasts and the user then enters the starting time for recording according to like. This is because, although the apparatus is preferably the methods described above. The interface then searches its highly trained upon manufacture, a large number of new databases regarding the user and broadcast listings to present broadcasts are always being created and presented, so that a most likely choice given that parameter, as well as all the apparatus must constantly maintain its “awareness of available alternatives. In this case, the user history is of little data types and trends, as well as update its predicted help, and is not useful for making a prediction. In other preferences of the user(s). cases, the system uses its intelligence to “fill in the blanks'. 1171 Based on input from the user, other programming which could, of course, be rejected by the user if these are including the same actors and/or actresses may be processed, US 2007/0016476 A1 Jan. 18, 2007 67 e.g., recorded. For example, Katey Segal periodically system itself is Sophisticated, it provides simplicity, with appears on "Jay Leno' as a musical guest, and therefore may inductive reasoning and deductive reasoning for the user. be recorded in these appearances. 1177 Thus, a minimum of user interaction is required even for complex tasks, and nearly full automation is EXAMPLE 11 possible, as long as the user and apparatus are able to 1172 Intelligent Adaptive VCR Interface communicate to convey a preference. As a further embodi ment according to the present invention, the interface system 1173 Another example of the use of the present program will stored transmitted data, and subsequently review that ming system allows a hybrid request which does not corre data, extracting pertinent information. The stored data may spond to any single broadcast Schedule entry. In this case, if then be deleted from the storage medium. In this regard, the the user instead wishes to record weather reports on all system may be self learning. channels, the interface may be of further help. The interface controls a plurality of tuner elements 2502 of a video signal 1178. It is noted that various algorithms and formulae for reception device 2501, so that a plurality of broadcasts may pattern recognition, correlation, data compression, trans be simultaneously received. Using the mass storage and forms, etc., are known to those skilled in the art, and are possibly image data compression described above, a plural available in compendiums, such as Netravali, Arun N., and ity of broadcasts may also be recorded simultaneously in the Haskell, Barry G., “Digital Pictures Representation and intermediate storage 2503. The mass storage may be mul Compression’, Plenum Press, New York (1988); Baxes, tiple VCRs, optical storage, magnetooptical storage, mag Gregory A., “Digital Signal Processing, A Practical Primer', netic storage including disk (e.g. single disks, multimedia Prentice-Hall, Englewood Cliffs, N.J. (1984); Gonzalez, compatible disks, RAID, etc.) tape (QIC, 8 mm, 4 mm, etc.). Rafael C., “Digital Image Processing. Addison-Wesley, Preferably, the archival recording medium is recordable Reading, Mass. (1987), and, of a more general nature, Press, DVD or possibly recordable CD-ROM. William H. et al., “Numerical Recipes in C The Art of 1174 The optical recording tape produced by ICI, Inc., or Scientific Computing, Cambridge University Press, 1988. other card or tape optical storage medium might also be a EXAMPLE 12 useful storage medium for large Volumes of data, as might be generated by recording multiple video signals. The 1179 Intelligent Adaptive VCR Interface known implementations of the ICI product system best 1180 A further example of the use of the advanced suited for commercial or industrial use and not for individual intelligent features of the present invention is the use of the COSU U.S. system to record, e.g., "live” musical performances. These 1175. In any case, the interface 2506 accesses its associ occur on many “talk’ shows, such as “Tonight Show” (NBC, ated database 2413 to determine, at a given time, which 11:30 p.m. to 12:30 p.m., weeknights), “Saturday Night channels are broadcasting “news’. The interface system Live” (NBC 11:30 p.m. to 1:00 a.m. Saturday-Sunday), and might also randomly or systematically monitor or scan all or other shows or “specials’ such as the “Grammy Awards'. a portion of the available broadcasts for “special reports”. The interface, if requested by the user to record such The interface system then monitors these channels for performances, then seeks to determine their occurrence by, indicia of a “weather' information content broadcast. For e.g., analyzing a broadcast schedule; interacting with the example, the newscaster who appears to report the weather on-line database 2411; and by reference to the local database on a given show is usually the same, so that a pattern 2413. When the interface determines with high probability recognition system 2505 of the video frame could indicate that a broadcast will occur, it then monitors the channel(s) at the presence of that newscaster. In addition, the satellite the indicated time(s), through the plurality of tuners 2502. photographs, weather radar, computer generated weather The system may also autonomously scan broadcasts for forecast screens, etc. are often similar for each broadcast. unexpected occurrences. Finally, news segments, such as “weather often appear at 1181. In the case of pay-per-view systems and the like, the same relative time in the broadcast. Using this informa which incorporate encrypted signals, an encryption/decryp tion, the interface system selects certain broadcast segments tion unit 2509 is provided for decrypting the transmitted for retention. signal for analysis and viewing. This unit also preferably 1176) This retention begins at a beginning of a news allows encryption of material in other modes of operation, segment. Such as “weather, stop recording during commer although known decryption systems without this feature cials, and continues after return from break, on all selected may also be employed with the present system. During the channels. In order to assist in making accurate decisions, the monitoring, the interface system acquires the audio and monitored broadcasts may be stored in a temporary storage Video information being broadcast, through the signal medium until a decision is made, and thereafter transfer the receiver 2408, and correlates this information with a known recording to a more permanent storage medium if that be profile of a “live musical performance', in the preference appropriate. It is noted that the system of the present and event correlator 2412. This must be distinguished from invention is intelligent, and may therefore “learn' either music as a part of e.g., a Soundtrack, as well as “musicals' explicitly, or through training by example. Therefore, if the which are part of movies and recorded operas, if these are system made an error during the process, the user may define not desired by the user. Further, music videos may also be the error of the system, e.g., a Substitute newscaster or undesirable. When the correlation is high between the broad rearrangement of news segments, so that the interface sys cast and a reference profile of a “live musical performance'. tem has a reduced likelihood of making the same error again. the system selects the broadcast for retention. In this case, Thus, while Such a system is inherently complex, it poses the information in the intermediate storage 2503 is trans significant user advantages. Further, while the interface ferred to the plant 2507, which includes a permanent storage US 2007/0016476 A1 Jan. 18, 2007

device 2508. The intermediate storage 2503 medium is used have an architecture optimized for the efficient and rapid to record a “buffer segment, so that none of the broadcast execution of repetitive numeric calculations, are desirable is lost while the system determines the nature of the broad for certain pattern recognition tasks, and may be provided as cast. This, of course, allows an extended period for the a tightly coupled parallel processing array to increase determination of the type of broadcast, so that, while real throughput. time recognition is preferred, it is not absolutely necessary 1185. A known board containing a DSP is the in order to gain the advantages of the present invention. The MacDSP3210 by Spectral Innovations Inc., containing an buffer storage data, if not deleted, also allows a user to select AT&T digital signal processor and an MC68020 CISC a portion for retention that the interface system has rejected. processor, and which uses the Apple Real-time Operating 1182. Thus, while it is preferable to make a determination System Executive (A/ROSE) and Visible Cache Operating in real time, or at least maintain real time throughput with a System (VCOS). It is preferred that the processors employed processing latency, it is possible to make an ex post facto be optimized for image processing, because of their higher determination of the nature of the broadcast program. By throughput in the present image processing applications, to using an available delay, e.g., about 5 to about 300 seconds, process the video signals, and more other signal processors or longer, the reliability of the determination can be greatly to analyze the audio signals. Of course, general purpose increased as compared to an analysis of a few frames of processors may be used to perform all calculations. An array video data, e.g., about 15 to about 300 mS. An intermediate processor which may be interfaced with a Macintosh is the reliability will be obtained with a delay of between about Superserver-C available from Pacific Parallel Research Inc., 300 to about 5000 mS. As stated above, the storage system incorporating parallel Inmos Transputers. Such an array for this determination need not be uncompressed nor loss processor may be suitable for parallel analysis of the image less, so long as features necessary to determine the character segment and classification of its attributes. of the broadcast are present. However, it is preferred that for 1186 Pattern recognition processing, especially after pre broadcast recording intended for later viewing, the storage processing of the data signal by digital signal processors and be as accurate as possible, so that if a compression algorithm image compression engines, may also be assisted by logical is implemented, it be as lossless as reasonable given the inference engines, such as FUTURE (Fuzzy Information various constraints. The MPEG-2 standard would be appli Processing Turbo Engine) by The Laboratory for Interna cable for this purpose, though other video compression tional Fuzzy Engineering (LIFE), which incorporates mul systems are available. tiple Fuzzy Set Processors (FSP), which are single-instruc 1183. In a preferred situation, approximately 5 minutes of tion, multiple data path (SIMD) processors. Using a fuzzy broadcast material is analyzed in order to make a determi logic paradigm, the processing system may provide a best fit nation of the content. This broadcast material is stored in two output to a set of inputs more efficiently than standard media. First, it is stored in a format acceptable for viewing, computational techniques, and since the presently desired Such as video tape in a videotape recorder, or in digital video result requires a “best guess', rather than a very accurate format, e.g., uncompressed, MPEG-2. Second, it is received determination, the present interface is an appropriate appli in parallel by the computer control, where the data is subject cation of this technology. to a number of recognition and characterization processes. 1187. As noted above, these processors may also serve These are performed in parallel and in series, to produce a other functions such as voice recognition for the interface, or stored extracted feature matrix. This matrix may contain any extracting text from video transmissions and interpreting it. type of information related to the broadcast material, includ It is also noted that, while some of these coprocessing ing an uncompressed signal, a compressed signal, a highly engines are now costly, these costs are decreasing and the processed signal relating to information contained in par present invention therefore includes the use of sophisticated ticular frames and abstract features, spatially and temporally present designs as well as future devices which may be used dissociated from the broadcast signal, yet including features to perform the stated functions. The continued development included in the broadcast which relate to the content of the of optical computers may also dramatically reduce the cost broadcast. of implementing this aspect of the present invention, how 1184. One possible method incorporates one or more ever, the present state of the art allows the basic functions to digital signal processor based coprocessor elements, which be performed. See attached appendix of references, incor may be present on, e.g., Nubus cards in the Macintosh porated herein by reference, detailing various optical com Quadra 950, Apple Power PC, PCI card in Pentium-based puting designs. MS-DOS/Windows 3.1, 3.11, 95, 98, NT computers (or 1188 A real time operating system may be employed, of Macintosh PCI-based computers), other Power PC based computers. These elements may be based on C-Cube CL550 which there are a number of available examples. Some older (JPEG compression), Analog Devices ADSP-21020, Analog examples include SPOX DSP operating system, IBM's Devices ADSP-21060, AT&T (formerly American Tele Mwave operating system and AT&T’s VCOS operating phone and Telegraph Co.) DSP32C, AT&T DSP3210, AMD system. These operating systems, and possibly others, are to 29000 series, Motorola DSP 96000ADS, Texas Instruments be supported by Microsoft Inc.'s Windows 95 operating TMS320C40, TMS320C80, IBM Mwave, or other known system Resource Manager function. devices. Other devices are also available from Analog 1189. It is noted that various methods are available for Devices, AT&T, DSP Group, Motorola, NEC, SGS-Thom determining a relatedness of two sets of data, Such as an son, Sharp, Texas Instruments, Zilog, Zoran, and other image or a representation of an image. These include the vendors. See, EDN, May 11, 1995, pp. 40-106; Bursky, D., determination of Hausdorff distance, fuzzy correlation, “Improved DSPICs Eye New Horizons”, Electronic Design, arithmetic correlation, mean square error, neural network Nov. 11, 1993, pp. 69-82. DSP systems, which generally “energy minimization, covariance, cross correlation, and US 2007/0016476 A1 Jan. 18, 2007 69 other known methods, which may be applied to the raw data user level interface of the present invention. However, the or after a transformation process, Such as an Affine trans following better serves to define this aspect of the invention. formation, a Fourier transformation, a Gabor transformation, a warping transformation, a color map transformation, and 1195 The input device, in addition to defining a desired the like. Further, it is emphasized that, in image or pattern command, also provides certain information about the user recognition systems, there is no need that the entire image be which has heretofore been generally ignored or intentionally correlated or even analyzed, nor that any correlation be removed. With respect to a two-dimensional input device, based on the entirety of that image analyzed. Further, it is Such as a mouse, track all, joystick, etc., this information advantageous to allow redundancy, so that it is not necessary includes a velocity component, an efficiency of input, an to have unique designations for the various aspects of the accuracy of input, an interruption of input, and a high data to be recognized, nor the patterns to be identified as frequency component of input. This system is shown sche matching the uncharacterized input data. matically in FIG. 21, which has a speed detector 2104, a path optimization detector 2105, a selection quality detector 1190) The MSHELL from Applied Coherent Technology 2106, a current programming status 2108, an error counter is a software system that runs on a Mercury MC3200 array 2109, a cancel counter 2110, a high frequency signal com processor, in conjunction with a Data Translation DT2861 or ponent detector 2112, an accuracy detector 2113 and a DT2862. The NDS1000 Development System from Nestor, physio-dynamic optimization detector 2114. In addition, Inc., provides image recognition Software which runs on a FIG. 21 also shows that the interface also uses a past user PC compatible computer and a Data Translation DT2878. history 2107, an explicit user level choice 2111 and an explicit help request 2115. 1191. The C-Cube CL550 is disclosed in “C-Cube CL550 JPEG Image Compression Processor. Preliminary Data 1196. This list is not exclusive, and is somewhat depen Book, August 1991, and addendum dated Nov. 20, 1991, and dent on the characteristics of the specific input device. For products incorporating the CL550 include the JPEG Vdeo a mouse, trackball, or other like device, the velocity or speed Development it (ISA bus card with Chips and Technologies component refers to the speed of movement of the sensing PC video 82C9001A Video Window Controller), and the element, i.e. the rotating ball. This may also be direction C-Cube CL550 Development Board/PC for ISA Bus sensitive, i.e., velocity vector. It is inferred that, all other (CL550, for use with Truevision TARGA-16 or ATVista things being equal, the higher the Velocity, the more likely cards) or for NuBus (Macintosh). The so-called C-Cube that the user “knows’ what he is doing. “CL950' is a MPEG decoder device. Such a device as the CL950 may be particularly useful for use in the present VCR 1197) The efficiency of input refers to two aspects of the for reproducing compressed program material, which may user interface. First, it refers to the selection of that choice be compressed by the present apparatus, or may be used for which most simply leads to the selection of the desired decompressing pre-compressed program material. Other selection. For example, if “noon” is an available choice along with direct entry of numbers, then the selection of MPEG-1 and MPEG-2 encoding and decoding devices are “noon” instead of “12:00 p.m.” would be more efficient. The known. second aspect of efficiency has to do with the path taken by 1192. It is noted that all functions of a VCR would also the user in moving a graphic user interface cursor or input be facilitated by the use of such powerful processors, and device from a current position to a desired position. For thus it is not only these advanced functions which are example, a random curve or Swiggle between locations is enabled by these advanced processors and coprocessors. It is less efficient than a straight line. This effect is limited, and also noted that these image recognition functions need not must be analyzed in conjunction with the amount of time it necessarily all be executed local to the user, and may in fact takes to move from one location of a cursor on the screen to be centralized with resultant processed data transmitted to another, if the speed of movement is very rapid, i.e. less than the remote user. This would be advantageous for two rea about 400 mS for a full screen length movement, or less than sons: first, the user need not have an entire system of about 300 mS for small movements, then an inefficiency in hardware localized in the VCR, and second, many of the path is likely due to the momentum of the mouse and hand, operations which must be performed are common to a momentum of the rolling ball, or a physiological arc of a number of users, so that there is a net efficiency to be gained. joint. This aspect is detected by the physio-dynamic opti mization detector 2114. Thus, only if the movement is slow, EXAMPLE 13 deliberate, and inefficient, should this factor weigh heavily. It is noted that arcs of movement, as well as uncritical 1193 Intelligent Adaptive VCR Interface damping of movement around the terminal position may be more efficient, and a straight path actually inefficient, so that 1194 The interface of the present invention incorporates the interface may therefore calculate efficiency based on a an intelligent user interface level determination. This func complex determination, and act accordingly where indi tion analyzes the quality of the user input, rather than its cated. content. Thus, this differs from the normal interface user level determination which requires an explicit entry of the 1198 Thus, an “efficient’ movement would indicate an desired user level, which is maintained throughout the user who may work at a high level, and conversely, an interface until explicitly changed. The present interface may inefficient movement would indicate a user who should be incorporate the “smart screen” feature discussed above, presented with simpler choices. The efficiency of movement which may, through its analysis of the past user interaction is distinguished from gestures and path dependent inputs, with the interface predict the most likely predicted user input Such as drawing and painting. These may be distinguished function. Thus, the predictive aspects of the present inven based on machine status or context. Further, the interface tion may be considered a related concept to the intelligent may recognize gestures in many contexts. Therefore, ges