US0075.99963B2

(12) United States Patent (10) Patent No.: US 7,599,963 B2 Fernandez (45) Date of Patent: Oct. 6, 2009

(54) NETWORK-EXTENSIBLE 6,980,596 B2 12/2005 Wang et al. RECONFIGURABLE MEDIA APPLIANCE 7,102,669 B2 9/2006 Skow 2001/0010493 A1* 8/2001 Script et al...... 340,546 (76) Inventor: Dennis S. Fernandez, 1175 Osborn 2001/0043219 A1 11/2001 Robotham et al. Ave., Atherton, CA (US) 94027 2002/0053083 A1* 5/2002 Massey ...... T25/46 (*) Notice: Subject to any disclaimer, the term of this patent is extended or adjusted under 35 U.S.C. 154(b) by 113 days. (Continued) (21) Appl. No.: 11/203,645 FOREIGN PATENT DOCUMENTS (22) Filed: Aug. 11, 2005 JP A-H-08-279922 10, 1996 e ----9 (65) Prior Publication Data US 2008/0163287 A1 Jul. 3, 2008 (Continued) OTHER PUBLICATIONS Related U.S. Application Data Gleicher et al., “A Framework for Virtual Videography”, Interna (63) yof application No. 10/448,202, filed on tional Symposium on Smart Graphics, online), Jun. 11-13, 2002 ay ZRS, retrieved on Mar. 3, 2009). Retrieved from Internet:

cellular 30S 304 ServiceDevice PDA Camera

37 33 Media Repository PCWorkStati N 398 32 Networked Camera BRM Service GPs service - 100 309

30 -1 Security Service Biometric Service 10 313 News 311 Renderer 32 Sports Private US 7,599,963 B2 Page 2

U.S. PATENT DOCUMENTS SCRI, 2002 Pro DVDUsage & Products / Brands Report http://www. scri.com/2002prodvd.html. 2002/0054244 A1* 5, 2002 Holtz et al...... 348,722 SCRI, Broadcast/Pro Video Brand Awareness and Ratings http:// 2002/017.0062 A1 11/2002 Chen et al. www.scri.com/brands2002.html. 2002fO190991 A1* 12/2002 Efrain et al...... 345/475 SCRI, 2002-2004 Global Streaming Media Marketplace Trends 2003/0O85992 A1 5/2003 Arpa et al. Report http://www.scri.com/sc sm 2002trends.html. 2003, OO86686 A1* 5/2003 Matsui et al...... 386,52 Tensilica Inc., Xtensa Architecture and Performance, Sep. 2002 2003/0098.957 A1 5, 2003 Haldiman www.tensilica.com/Xtensa white paper.pdf. 2003/030993 A1 7/2003 Mendelevitch et al. Pixim Inc., Pixim D2000 Video Imaging System for Advanced 2003. O149569 A1 8, 2003 Jowitt et al. CCTV Cameras Publication PB-D2000-2003-02 Mountain View, 2003/0185301 A1* 10, 2003 Abrams et al...... 375,240.12 CA 94043. 2003. O187924 A1* 10, 2003 Riddle ...... 709.204 Pixim Inc., Pixim D1000 Video Imaging System for CCTV Cameras 2003,0224857 A1* 12, 2003 Baerlocher .. ... 463f42 Publication PB-D1000-2003-02 Mountain View, CA 94043. 2004/OOO3400 A1 1/2004 Carney et al. Pixim Inc., Pixim Digital Pixel System Technology Backgrounder, 2004/0027368 A1 2/2004 Snyder et al. 2003 http://www.pixim.com/technology/technologyphtml. 2004/0027369 A1 2/2004 Kellock et al. Pixim Inc., Platform Tools, 2003 http://www.pixim.com/products/ 2004/0061804 A1* 4/2004 Favrat et al...... 348,554 platform.html. 2004/0103225 A1* 5/2004 McAlpine et al...... 71 Of 52 Patrick Mannion, EE Times, ZigBee gets push from chips and 2004/0105004 A1* 6/2004 Rui et al...... 348.159 startups http://www.commsdesign.com/story/OEG2003.0416S0023 2004/O151242 A1 8/2004 Chang Apr. 16, 2003. 2004/O154043 A1 8, 2004 Roman Junko Yoshida, EE Times, LSI Logic bids for H.264 market with 2004/0193483 A1* 9, 2004 Wolan ...... TO5/14 VideoLocus acquisition http://www.eetimes.com/story/ 2004/0218902 A1* 11/2004 Yanagita ...... 386,52 OEG2003O416SOO27. 2004/022O798 A1 11/2004 Chi et al. Carlos Garza, RIAA, Music Industry Integrated Identifers Project 2004/0226051 A1 11/2004 Carney et al. (MI3P) and MPEG-21 Jan. 2003. 2005/0091305 A1 4/2005 Lange et al. Kari Jaaskelainen, Strategic Questions in the Development of Inter 2006/0038923 A1 2/2006 Dinwiddie active Television Programs Dissertation, ILMARI design publica tions. FOREIGN PATENT DOCUMENTS SMPTE, The SMPTE Journal Abstracts of Papers, 2003. MPTE, The SMPTE Journal Abstracts of Papers, Jan. 1999. JP A-H-10-210365 8, 1998 MPTE, The SMPTE Journal Abstracts of Papers, Feb. 1999. JP A-2002-319270 10, 2002 MPTE, The SMPTE Journal Abstracts of Papers, Apr. 1999. JP A-2003-006052 1, 2003 MPTE, The SMPTE Journal Abstracts of Papers, May 1999. JP 2004-022716 1, 2004 MPTE, The SMPTE Journal Abstracts of Papers, Jun. 1999. MPTE, The SMPTE Journal Abstracts of Papers, Jul. 1999. OTHER PUBLICATIONS MPTE, The SMPTE Journal Abstracts of Papers, Aug. 1999. Mey et al., “A Multimedia Component Kit' Ioneline). Sep. 1993, MPTE, The SMPTE Journal Abstracts of Papers, Sep. 1999. retrieved on Mar. 3, 2009). Retrieved from Internet:

ITV The Medium of All Mediums? www.interactivehq.org/coun Junko Yoshida, EE Times Philips' Silicon Hive to develop IP cores cils/amc/iTVStudy.pdf. http://www.eetimes.com/story/OEG20030314S0013 Mar. 16, 2003. Jerry Bennington, Secretary ATVEF & CableLabs Executive Con Bermai. 7200 Series Wireless Systems-on-Chip IEEE 802.11a with Sultant, USA Challenges in Building a New Television Paradigm. 802.11b Interface for Combo Solutions 2002. Jane Black, BusinessWeek Hollywood's Digital Love/Hate Story. Junko Yoshida, EE Times Camera-phones takehold as digital imag Willsearch, Inc., Millenium Multimedia Leader. ing platform http://www.eetimes.com/story/OEG20030210S0025 Jessica Hundley, Director's Guide to , Acting for DV Feb. 10, 2003. http://www.Sonyusadvcam.com/content/printer 36.shtml. Bob Tourtellotte, Reuters Imax, Lucas TeamUp for Giant 'StarWars' Dean Takahashi, Mercury News, S1,700 remote built to control home Movie Sep. 9, 2002. of the future. Imax Ltd., Imax The 15/70 Filmmaker's Manual 1999. Intel Corporation, Intel Technology Journal Interoperable Home Legal Infrastructure for Certification Authorities and Secure Elec Infrastructure vol. 06, Issue 04, Nov. 15, 2002, ISSN 1535-766X. tronic Commerce Digital Signature Guidelines Aug. 1, 1996. David L. Hayes, Fenwick & WestLLPAdvanced Copyright Issues on Tobias Hollerer, John Pavlik, Columbia University Situated Docu the Internet. mentaries: Embedding Multimedia Presentations in the Real World Drew Dean, SRI International Computer Science Laboratory Digital IEEE 1999. Rights Management: A Contrarian's View. Adobe A Manager's Introduction to Adobe eXtensible Metadata Platform, The Adobe XML Metadata Framework. Berkeley Conference on Law and Technology of DRM Systems Raymond Chan, Emily Reyna, Matt Rubens, Annie Wu, Stanford DRM Technology Tutorial Feb. 27, 2003. University Pornography and Technology http://www.stanford.edu/ Edward W. Felten, Dept. of Computer Science, Princeton University ~mrubenspornsite2/technology.htm. DRM Black Boxes, and Public Policy. Edrolfo Leones, Director of Global Media Asset Stragegies. The Walt John Erickson, HP Labs, Vermont Policy enforcement vs. the free Disney Company, "Applying Metadata Practices' http://www. flow of information. uemedia.com/eam/pdfLeonesEAM2003.pdf. Drian A. Lamacchia, Microsoft Corporation DRM Policy and Rights “DTV Application Software Environment Level 1 (DASE-1). Part Expression on the Trusted Platform of the Future. 1: Introduction, Architecture, and Common Facilities, ATSC Stan Richard D. Rose, Connecting the docs: Navigating the laws and dard, Advanced Television System Comitte. http://www.atsc.org/ licensing requirements of the Internet music revolutio. standards/a100.zip. Semiconductor Business News, Toshiba develops fuel cell for note “DTV Application Software Environment Level 1 (DASE-1). Part books http://www.siliconstrategies.com/story/ 2: Declarative Applications and Environment, ATSC Standard http:// OEG2003O3O4SOO48. www.atsc.org/standards/a100.Zip. Dean Takahashi, Mercury News Sony chip to transform video-game DTV Application Software Environment Level 1. (DASE-1), Part 3: industry. Procedural Applications and Environment http://www.atsc.org/stan George Leopold, EE Times Bill would add copy protection to DTV dards/a100.zip. receivers, Sep. 20, 2002. “DTV Application Software Environment Level 1 (DASE-1), Part Matthew Gast, AirMagnet Seven Security Problems of 802.11 Wire 4: Application Programming Interface http://www.atsc.org/stan less Dec. 2002, Mountain View, CA 94043. dards/a100.zip. Michael A. Hiltzik, Digital Cinema Take 2 MIT Technology Review, “DTV Application Software Environment Level 1 (DASE-1). Part Sep. 2002. 5: Zip Archive Resource Format http://www.atsc.org/standards/ Sarnoff Corporation, About Sarnoff Silicon Strategies http://www. a100.zip. Sarnoff.com/products services/semiconductor ip? DTV Application Software Environment Level 1 (DASE-1) Part 6: silicon strategies/about.asp. Security; http://www.atsc.org/standards/a100.Zip. Junko Yoshida, EE Times Interface issues loom for full-featured DTV Application Software Environment Level (DASE-1) Part 7: phones http://www.eetimes.com/story/OEG20030217S0016 Feb. Application Delivery System—Arm Binding http://www.atsc.org/ 17, 2003. standards/a100.zip. Semiconductor Business News, Sarnoff offers digital video manipu DTV Application Software Environment Level 1 (DASE-1) Part 8: lation COS http://www.semibiznews.com/story/ Conformance; http://www.atsc.org/standards/a100.Zip. OEG20030 130SO009 Jan. 30, 2003. U.S. Appl. No. 10/448,202, filed May 28, 2003, Fernandez, Dennis. Semiconductor Business News, Pictos rolls out new CMOS image Video-On-Demand: Taping Deeper Wells, Chaos Media Networks, sensor http://www.semibiznews.com/story/OEG20030 130S0009 www.chaosmedianetworks.com Jan. 2002, Santa Clara, CA. Mar. 3, 2003. S. Appl. No. 1 1/831,758, filed Jul. 31, 2007, Fernandez, Dennis. Consumer Electronics Association, 5 technologies to watch Oct. S. Appl. No. 1 1/831,777, filed Jul. 31, 2007, Fernandez, Dennis. 2002. S. Appl. No. 1 1/836,064, filed Aug. 8, 2007, Fernandez, Dennis. Atheros Communications, Building A Secure Wireless Network S. Appl. No. 1 1/836,658, filed Aug. 9, 2007, Fernandez, Dennis. How Atheros Defines Wireless Network Security Today And In The S. Appl. No. 1 1/836,668, filed Aug. 9, 2007, Fernandez, Dennis. Future 2003. .S. Appl. No. 1 1/934,678, filed Nov. 2, 2007, Fernandez, Dennis Loring Wirbel, EE Times Consumer, wireless could be optical saviors S. Appl. No. 1 1/934,707, filed Nov. 2, 2007, Fernandez, Dennis. http://www.commsdesign.com/story/OEG20030326S0018 Mar. 26, S. Appl. No. 1 1/934,712, filed Nov. 2, 2007, Fernandez, Dennis. 2003. S. Appl. No. 12/114, 160, filed May 2, 2008, Fernandez, Dennis. Robert Keenan, EE Times ChipWrights optimizes processor for digi S. Appl. No. 12/238,686, filed Sep. 26, 2008, Fernandez, Dennis. tal cameras http://www.eetimes/com/story/OEG20030210S0045 S. Appl. No. 1 1/585,508, filed Oct. 23, 2006, Fernandez, Dennis. Feb. 10, 2003. cited by examiner U.S. Patent Oct. 6, 2009 Sheet 1 of 9 US 7,599,963 B2

|— U.S. Patent

punOS U.S. Patent Oct. 6, 2009 Sheet 3 of 9 US 7,599,963 B2

?unfilZ U.S. Patent

909G09

U.S. Patent Oct. 6, 2009 Sheet 5 Of 9 US 7,599,963 B2

O cy r

U.S. Patent Oct. 6, 2009 Sheet 6 of 9 US 7,599,963 B2

707 ?07 : 3OnpOud-?SOd Infi!--#3 U.S. Patent Oct. 6, 2009 Sheet 7 of 9 US 7,599,963 B2

|09 eun61-I9 U.S. Patent Oct. 6, 2009 Sheet 8 of 9 US 7,599,963 B2

eun61-I9

U.S. Patent Oct. 6, 2009 Sheet 9 of 9 US 7,599,963 B2

InOS30 eun61-I1 ?u?uuJ???Oel3uueO-??O US 7,599,963 B2 1. 2 NETWORK-EXTENSIBLE FIG.3a is a diagram illustrating network-extensible recon RECONFIGURABLE MEDIA APPLIANCE figurable media appliance communicating with other net worked services and devices, according to an embodiment of CROSS-REFERENCE TO RELATED the present invention. APPLICATION FIG.3b is a diagram illustrating network-extensible recon figurable media appliances communicating over a network This application is a continuation of the U.S. patent appli with a server, according to an embodiment of the present cation Ser. No. 10/448,202 filed on May 28, 2003. invention. FIG. 4 is a flow diagram illustrating a method for sensing BACKGROUND 10 according to an embodiment of the present invention. FIG. 5 is a flow diagram illustrating a method for filling in 1. Field a template according to an embodiment of the present inven Invention relates to electronic media devices and process tion. ing methods, particularly to networked reconfigurable media FIG. 6 is a flow diagram illustrating a method for tagging appliance. 15 audio and/or video representation with location and/or direc 2. Related Art tion information. Conventional digital video media appliances rely on exter FIG. 7 is a flow diagram illustrating a method for transfer nal computing resources for post-processing of recorded ring data and/or instructions from off-appliance source to audio or video signals, since limited on-appliance editing and on-appliance memory. filtering capabilities may render Such tasks impractical. Fur thermore, limited on-appliance Software extensibility and DETAILED DESCRIPTION limited communication with external resources leave conven tional digital media appliances as standalone audio or video FIG. 1a is an architectural diagram illustrating network recording tools limited by factory-loaded on-appliance pro extensible reconfigurable media appliance 100 according to cessing capacity. 25 an embodiment of the present invention. Media appliance 100 Accordingly, there is need for network-extensible and eas comprises media unit 101, controller 108, communication ily reconfigurable media appliance capable of communicat unit 103, and power module 116. ing over networks and allowing for extension of on-appliance Media unit 101 comprises audio and/or video (A/V) sensor audio or video processing Software and tagging of recorded 30 120 for sensing incoming audio and/or video. Sensed video is audio or video signals. stored in memory 110 using video format Such as Digital Video Disc (DVD), PAL Digital Video Cassette (PAL DVC), SUMMARY PAL or NTSC Laserdisc, 24P HD, 3/4-inch, MPEG-2, MPEG-4 (DV-25, DV-50, IMIX, ISMA, etc.), H.264, AVI, Network-extensible reconfigurable media appliance 35 DV, DVCAM, DVCPRO, DVCPRO-25/50/100, VHS, senses incoming audio and/or video, and encodes and stores D-VHS, W-VHS, Digital-8, Digital-S, D1, D2, D5 HD, Beta in media appliance memory or alternatively relays over net cam SX, Digital , other digital ENG format, Motion work to recipient. On-appliance digital audio and/or video JPEG, any other HDTV format, NTSC, PAL, HDD/RAID/ effects and filters process audio and/or video data stored in Disk Arrays, and/or other format for encoding video (speci memory, as well as incoming audio and/or video stream on 40 fications for describing these formats are herein incorporated the-fly. Media appliance dynamically stores, modifies, by reference). updates and deletes on-appliance set of digital effects and Media unit 101 optionally comprises biometric module filters, providing mobile extensible reconfigurable effects 106. Biometric module 106 comprises finger-print scanner, studio. Media appliance communicates wirelessly and/or retinal scanner, and/or other element for collecting a biomet over-wire with other media appliances, computers, security 45 ric Sample, and stores scanned biometric data and/or result of systems, video storage, Global Positioning System (GPS) biometric identification process in memory 110. For services, Internet, cellular services and/or personal digital example, a data structure is stored comprising a digital rep assistants (PDA) providing seamless integration of captured resentation of collected biometric sample for authorization audio and/or video stream with off-appliance resources and/ based on comparison with previously-stored biometric iden or services. Audio and/or video, wireless, biometric and GPS 50 tifier. Biometric module 106 optionally couples with a micro input and/or output as well as on-appliance acceleration array chip for genetically-based identification. detector render media appliance Suitable for security appli Media unit 106 optionally comprises reconfigurable logic cations. Extensible reconfigurable on-appliance effects and and/or software 122 for performing programmable audio and/ filters studio render media appliance for entertainment and or video sensing, or for conversion to or from audio and/or Video production or editing applications. 55 video formats. Controller 108 comprises microprocessor 123 (such as one BRIEF DESCRIPTION OF DRAWINGS from the Intel Centrino processor family, the specification of which is herein incorporated by reference), and optionally FIG. 1a is an architectural diagram illustrating network comprises co-processor 124, Digital Signal Processing (DSP) extensible reconfigurable media appliance according to an 60 unit 125, array processor 126, and/or reconfigurable logic embodiment of the present invention. 127. Controller 108 performs audio and/or video processing FIG. 1b is an architectural diagram illustrating network on audio and/or video data residing in memory 110. Option extensible reconfigurable media appliance according to an ally in real-time manner, controller 108 performs on-the-fly embodiment of the present invention. audio processing and/or on-the-fly video processing on FIG. 2 is a diagram illustrating memory element of net 65 incoming encoded audio data and/or incoming encoded video work-extensible reconfigurable media appliance according to data prior to storage of resulting processed audio data and/or an embodiment of the present invention. resulting processed video data in memory 110. US 7,599,963 B2 3 4 Controller 108 is implemented in Application Specific digital representation of incoming Sound in memory 110 Integrated Circuit (ASIC) blocks, synthesizable intellectual using audio format Such as Audio Interchange File Format property cores, cell processors, reconfigurable logic blocks, (AIFF), MPEG Layer 3 (MP3), and/or other format for Field Programmable Gate Arrays (FPGAs), Tensilica's encoding audio information. XTensa chip architecture and/or instruction set, Single or 5 I/O module 111 preferably has audio and video outputs. Multiple Instruction Single or Multiple Data (S/MIS/MD) I/O module 111 preferably communicates with on-appliance architecture signal processing chips, Sony "Cell' chip, and/or display or screen unit 114 and on-appliance speaker 115 for other architecture for performing audio and/or video process displaying video and generating audio. Optionally, display 1ng. unit 114 comprises a teleprompter for displaying visual Controller 108 and/or A/V sensor 120 may perform color 10 prompts (such as text and/or pictures). space conversion, brightness, white-balance, backlight com Optionally, I/O module 111 communicates wirelessly, pensation, gain control, activity detection, motion detection, wired, over cellular network, over LAN and/or over WAN motion tracking, gamma correction, sharpening, multi-frame (such as Internet), to send and/or receive GPS data, Digital noise reduction, depth estimation, 2-D bad-pixel correction, Rights Management (DRM) meta-data, audio and/or video Video compression, video stabilization, digital pan, digital 15 plugins, and/or other instructions and/or data for processing tilt, digital Zoom, and/or mosaicing for building panoramic and/or tagging of audio and/or video data. Optionally, I/O images from Successive frames. module 111 has video and audio inputs for receiving audio Communication unit 103 comprises radio-frequency (RF) and video signals from external audio and/or video source transceiver 128 for communicating via radio waves (e.g. over Such as a camera, a PDA, a media repository, a satellite, a cellular or other wireless network), and/or network controller security service, a DRM service, a biometric service, a GPS 129 for communicating via a wired and/or wireless network service, a PC or workstation, a cellular service or cellular (e.g. local area network (LAN), wide area network (WAN). device, or other device or service communicating with media wireless fidelity (WiFi) network, etc.). appliance 100. Media appliance 100 optionally has network Communication unit 103 optionally comprises subscriber controller 117 for communicating with other devices and/or information module (SIM) unit 130 and/or smart card unit for 25 services over a network. storage and/or retrieval of information about a user (Such as FIG. 2 shows memory 110 according to a preferred user preference, Subscribed service, permission, account embodiment of the present invention. Memory 110 comprises information, etc.), and/or for allowing usage of media appli Dynamic Random-Access Memory (DRAM), Static Ran ance 100 by one or more users. dom-Access Memory (SRAM), high-speed Flash memory, Communication unit 103 optionally comprises GPS mod 30 and/or removable memory (e.g. removable flash memory ule 112 for receiving GPS data over satellite. Optionally, GPS card such as MultiMediaCard). Memory 110 stores audio and module 112 is a micro GPS transponder implemented in video data 201. single chip or chipset. Optionally, memory 110 stores software instructions and Communication unit 103 optionally comprises accelera data implementing billing 202 and/or business methods. Such tion detector 113 (such as a gyroscope, a single-chip acceler 35 as a time-based pay-per-view and/or micro-billing feature. ometer or other element for detecting acceleration) for deter For example, memory 110 stores a data structure comprising mining orientation and/or acceleration of media appliance a field describing a viewing (such as a home-viewing of a 1OO. Video clip of video stream) and/or a field indicating an amount Communication unit 103 optionally comprises reconfig to be charged for the viewing and/or a field identifying a party urable logic or software 131 for performing programmable 40 to be charged. protocol translation, format conversion, network packet pro Optionally, memory 110 stores meta-data and/or instruc cessing, network packet compression and/or decompression, tions for implementing DRM 203 (e.g. Disney Media Asset communication encryption and/or decryption, and/or other Management (MAM) format), Resource Definition Frame communication processing. work (RDF) implementation such as Adobe's XMP (eXten Power module 116 provides power for media appliance 45 sible Metadata Framework), or other scheme for managing 100, and comprises AC and/or DC source, portable recharge meta-data. For example, an XMP packet data structure com able battery, fuel cell (e.g. direct methanol fuel cell, etc.), prising a header, an XML meta-data, a trailer, and a padding and/or other source for providing electrical power. Option field is employed. Optionally, memory 110 stores data and/or ally, media appliance 100 employs MICA microsensor plat instructions for implementing DRM according to a Right form for low-power wireless sensor networks, herein incor 50 Expression Language Data Model, for example employing porated by reference. Extensible Rights Markup Language (XrML). Optionally, Optionally, media appliance 100 architecture conforms to memory 110 stores meta-data and/or instructions for imple Advanced Telecommunication Computing Architecture (Ad menting proposed Global Release Identifier Syntax (GRID), vancedTCA), herein incorporated by reference. for example employing a data structure having an Identifier FIG.1b is a diagram illustrating network-extensible recon 55 Scheme, an Issuer Code, a Release Number, and a Checksum. figurable media appliance 100 according to one embodiment Optionally, memory 110 stores instructions and/or data of the present invention. Light or video sensor 102 senses 204 for performing digital authentication, encryption, incoming image stream and stores digital representation in decryption, key generation, digital signing, digital water memory 110. Preferably, sensor 102 is a complementary marking, and/or other instructions for performing security metal oxide semiconductor (CMOS) image sensor. Option 60 and/or privacy related computation on audio and/or video ally, sensor 102 is integrated with an image preprocessor. data, DRM data, billing data and/or conditions, sensitive per Optionally, sensor 102 comprises integrated two-chip set Sonal data, or other data residing in media appliance 100 such as Pixim D1000 or D2000 Video Imaging System chip and/or communicated to or from media appliance 100. For sets. Sensor 102 optionally comprises a partition for post example, memory 110 stores a data structure comprising a image processing steps. Alternatively, sensor 102 is a charge 65 field describing an encryption (and/or decryption) key, and coupled device (CCD) or an Active Pixel Sensor (APS) further stores instructions for encrypting a video stream using imager. Audio sensor 104 senses incoming Sound and stores the encryption (and/or decryption) key. US 7,599,963 B2 5 6 Optionally, memory 110 stores instructions and/or data stream, and a field for indicating whether a public perfor 205 for performing identity recognition (such as facial rec mance (and/or display) right is granted for the identified ognition, emotion recognition, Voice recognition, and/or video clip of video stream. Other digital rights can be imple other patternor identity recognition) on video data 201 and/or mented analogously. DRM meta-data and/or instructions on incoming video signal. For example, memory 110 stores a 5 optionally support secure promotion, sale, delivery, distribu data structure comprising an identifier for a database against tion, and/or usage tracking of digital content. Optionally, which image recognition is to be performed, for example a execution environment is partitioned into kernel Versus user database of faces for recognizing faces in a crowd. The data space and/or into standard versus trusted partitions according base may be stored (partially or completely) internally on to Microsoft's Next-Generation Secure Computing Base media appliance 100 or reside externally on a server. As 10 (NGSCB). another example, memory 110 stores a data structure com Media appliance 100 optionally inserts, deletes, and/or prising a feature extracted from a video stream and/or video modifies a label in an RDF (e.g. XMP) tag describing a media clip (using image extraction instructions stored in memory Segment. 110), and the extracted feature is used for a database query or Media appliance 100 optionally implements content is sent to a server for further handling. 15 authenticity, device authentication, and/or user authentica Optionally, memory 110 stores instructions and/or data for tion. Content authenticity comprises digital watermarking, performing authoring 206 and/or digital video editing (e.g. digital fingerprinting, and/or other technique for content linear or non-linear editing), compositing, and/or special authentication. For example, memory 110 stores instructions effects, such as Apple's Final Cut Prosoftware. For example, for reading an identifier describing a source of a video clip memory 110 stores a data structure comprising a bit rate and/or video stream, wherein the identifier is embedded in a associated with the encoding of a video clip and/or video digital watermark within the video clip and/or video stream. stream. As another example, memory 110 stores a data struc As another example, memory 110 stores a data structure ture comprising author information, genre information, title, comprising a field identifying one or more authorized sources characters, actors, genre, story, activities, viewer demograph for downloading video clips and/or video streams. Device ics, locations, Scenes, backgrounds, props, objects, setpieces, 25 authentication comprises Smartcards, public key certificates, or other information pertaining to a video clip and/or video and/or device for performing authentication. User authenti Stream. cation comprises biometrics using biometric module 106. Optionally, memory 110 stores instructions and/or data for passwords, and/or other technique for performing user tagging 207 the digital representation of a sensed scene authentication. (video stream and/or video clip) with meta-data. For example, 30 Media appliance 100 optionally implements, in software memory 110 stores a data structure comprising time, media (e.g. residing in memory 110) and/or hardware, an abstraction appliance location (such as provided by GPS module 112), layer between application and display, such as DVB (Digital media appliance orientation and/or media appliance accelera Video Broadcast) and/or MHP (Multimedia Home Platform) tion (such as provided by acceleration detector 113), multi abstraction layers. Specifications for incorporating the DVB lingual features (allowing for translation, Subtitles, Voice 35 and MHP formats are herein incorporated by reference. over, etc.), cues to a theater automation system (such as FIG.3a shows networked media appliance 100 communi instructions for house lights to go up, half-way up, or down, or cating with other device and/or service, according to a pre instructions to open or close curtains, etc.), instructions for ferred embodiment of the present invention. Communication allowing or disallowing content (such as trailers or promo with other device and/or service proceeds via direct network tional clips) to play next to other similar content, information 40 connection, Internet, WiFi, IEEE 802.11, IEEE 802.16, IEEE indicating suitability of content for different audiences such 802.15.4, ZigBee specification, cellular, Bluetooth, Universal as children, information indicating any promotional offers, Serial Bus (USB). Apple's FireWire, and/or other communi products and/or services (such as advertisements, product cation channel or protocol. Communication is optionally catalogs and/or coupons for products and/or services), infor encrypted, authenticated and/or digitally signed, preferably mation allowing for organizing and/or managing meta-data 45 with encryption engine 204 implemented in memory 110, or available to advertisers and/or service providers, and/or other alternatively with encryption engine 204 implemented in con information describing, identifying and/or relating to con troller 108. tent. Media appliance 100 optionally communicates with media DRM meta-data and/or instructions optionally comprise repository 307 for downloading and/or uploading video and/ flags for implementing rights and/or limitations of reproduc 50 or audio clips, video and/or audio meta-data Such as author tion, rights and/or limitations of public performance, rights information, genre information, title, characters, actors, and/or limitations of display, rights and/or limitations of dis genre, story, activities, demographics, locations, Scenes, tribution, rights and/or limitations of importation, rights and/ backgrounds, props, objects, set pieces, etc. or limitations of transmission or access, rights and/or provi Media appliance 100 optionally communicates with DRM sions under Digital Millennium Copyright Act (DMCA), 55 service 308 for downloading and/or uploading DRM meta rights and/or limitations of caching, rights and/or limitations data. Optionally, media appliance 100 generates a message of browsing, rights and/or limitations of storage, rights and/or indicating an infringement and/or other violation of digital limitations of transfer such as burning to Compact Disk (CD) rights, according to a set of DRM rules, such as copying or DVD, rights and/or limitations of referring or linking or without permission, broadcasting without permission, etc. framing, rights and/or limitations of streaming or download 60 For example, memory stores a data structure comprising a ing, rights and/or limitations of advertising, or other rights field identifying a video clip and/or video stream, and an and/or limitations and/or provisions. For example, memory indicator of a violation of a DRM rule, such as an act of 110 stores a data structure comprising a field identifying a broadcasting the video clip and/or video stream without per video clip or video stream, and a field for indicating whether mission. a reproduction right is granted for the identified video clip of 65 Media appliance 100 optionally communicates with secu Video stream. In another example, memory 110 stores a data rity service 309 to upload security information such as video structure comprising a field identifying a video clip or video and/or audio record of scene, identity recognition data as US 7,599,963 B2 7 8 computed by identity recognition instructions 203, GPS data residing on computer 303). For example, memory 110 stores as provided by GPS module 112, directional data as provided a data structure comprising a field indicating the time of last by acceleration detector 113, and/or to download security synchronization of media appliance 100 with computer 303 information Such as location to watch, identity data to store (or media repository manager or program manager residing for matching against images, and/or voice audio signature to on computer 303). Communication proceeds wirelessly and/ store for matching against audio clips. For example, media or via a cradle (coupled to computer 303) into which media appliance 100 sends a data structure to security service 309. appliance 100 is placed for synchronization. In one embodi wherein the data structure comprises a field identifying a ment, media appliance 100 comprises a user interface offer person, and a field identifying the location of the media appli ing a synchronization button (hard button on media appliance ance 100 at the time the person is sensed by media appliance 10 100 and/or soft button displayed in media appliance's 100 100. Optionally, media appliance 100 couples to police graphical display), activation of which causes described data authority for providing live and/or recorded footage and/or synchronization. triggering alarm and calling police according to built-in Media appliance 100 optionally communicates with PDA media appliance intelligence for identifying potential danger 304, cellular service and/or device 305, and/or other mobile ous and/or Suspicious conditions. 15 service and/or device for displaying video and/or audio data. Media appliance 100 optionally communicates with bio Media appliance 100 optionally communicates with other metric service 301 to upload biometric information obtained networked media appliance 306 for exchanging video and/or by biometric module 106, and/or to download biometric sig audio clips and/or for collaborating in the production of a nature for matching against incoming biometric data. media project, wherein a media appliance is assigned a token Media appliance 100 optionally communicates with GPS (number, String, etc.), statically or dynamically, for identify service 302, such as GPS satellites, to receive GPS informa ing the media appliance. Media appliance 100 optionally tion. For example, if media appliance 100 moves into a communicates with other networked media appliance 306 to restricted area, as indicated by GPS service 302 and/or by enable video-conferencing and/or multi-way collaboration, information residing on media appliance 100 and/or obtained for example, in business meetings, real estate transactions, remotely, GPS unit 112 activates an alert. For example, 25 distance learning, sports, fashion shows, Surveillance, train memory 110 stores a data structure comprising a field iden ing, games, tourism, etc. For example, memory 110 stores a tifying a restricted geographical area, and media appliance data structure comprising a field for describing a group of 100 generates an alarm when location of media appliance collaborating media appliances 100, and a field identifying 100, as indicated by GPS service 302, falls within the media appliance 100 itself among the group of collaborating restricted geographic area. 30 media appliances. Media appliance 100 optionally communicates with news FIG.3b is a diagram illustrating network-extensible recon service 310 and/or other objective information service. In one figurable media appliances communicating over a network embodiment, media appliance 100 receives a data structure with a server, according to an embodiment of the present from news service 310, the data structure representing a digi invention. One or more client media appliances 330 commu tal template and comprising a field identifying a location, and 35 nicate over a network 331 with server 332. Network 331 is a one or more fields identifying elements to be covered by combination of one or more wired and/or wireless networks reporter (Such as a person to interview, a particular place to such as the Internet, a LAN, a WAN, a satellite network, or point out to viewers, other news reporters covering the same other network for communication. In one embodiment, server news story, etc.). 332 is a news server, having a script or digital template for Media appliance 100 optionally communicates with sports 40 producing a news program. Server 332 delegates the record broadcasting network, game-show broadcasting network, ing or streaming of various predetermined pieces of audio and/or other gaming or competition-related network 311. In and/or video footage to the various media appliance clients one embodiment, media appliance 100 receives a data struc 330, wherein the recorded or streamed pieces will serve to ture from sports broadcasting network 310, the data structure fill-in the server 332 script or digital template for producing comprising a field identifying one or more competing parties, 45 the news program. In another embodiment, server 332 is a a field identifying a location of the competition, and a field server for sports or other competition, having a script or indicating the competition schedule. digital template for producing a sports program or a program Media appliance 100 optionally communicates with pri for other competitive activity. Server 332 delegates the vate service 312. In one embodiment, media appliance 100 recording or streaming of various predetermined pieces of receives a data structure from movie production source or 50 audio and/or video footage to the various media appliance network 310, the data structure comprising a field identifying clients 330, wherein the recorded or streamed pieces serve to one or more movie or media production, a field identifying a fill-in the server 332 script or digital template for producing location of the production, a field indicating the production the sports (or other competition) program. schedule, a field indicating one or more scenes, and a field In one embodiment, I/O module 111 presents a user inter indicating one or more cast or staff members. 55 face (UI), comprising a combination of hard (physical) but Media appliance 100 optionally communicates with ren tons and/or soft (graphical) buttons for accessing and using derer 313 to display video data. Renderer 313 comprises a billing functions, DRM functions, authentication, identity cinema or movie theater, television receiver, computer dis recognition, digital editing of media, and/or other services as play, IMAX display, a Digital Audio Broadcast (DAB) broad shown in FIG. 3a and described above. For example, a view caster, a satellite broadcaster, a digital TV, a High Definition 60 (for example comprising a button) is presented via display TV (HDTV), a PDA and/or cellular phone (or other mobile 114 to allow approval of a billing associated with the viewing device display). of video data. As another example, a view is presented via Media appliance 100 optionally communicates with a per display 114, allowing selection of one or more audio and/or sonal computer (PC) and/or workstation 303 and/or other video data for submission or transmission to a server 332, computing device for synchronization of data residing on 65 Such as a news server or a sports server, as described above. media appliance 100 with computer 303 (optionally interfac Selection of a presented audio and/or video data designates ing with media repository manager and/or program manager the selected data for Submission or transmission to the server. US 7,599,963 B2 10 Optionally, interfaces and media appliances are physically external variable (such as a combination of one or more of separate, wherein through an interface a user can tap into a weather, temperature, stock quotes, time of day or year, view pool or one or more media appliances to view available audio ing location, amount of money left in viewer's account, or any and/or video data, and/or select one or more available audio other external variables), based on biometric sensing of and/or video for submission or transmission to a server 332, viewer, based on the result of an identity or emotion recog as described above. As another example, a view is presented nition procedure on viewer (Such as distinguishing between at server 332 for approving the inclusion of a submitted or happiness, sadness, excitement, apathy, interest in a particu transmitted audio and/or video data into a script or a digital lar aspect of the presentation and/or other emotions or indi template for a news or sports program, wherein the audio cations of interest exhibited by viewer), based on real-time and/or video data is submitted by a media appliance client 330 10 input from viewer or from larger audience (such as deliberate to server 332, as described above. viewer decision of which script or template path to take next, FIG. 4 is a flow diagram illustrating a method for sensing provided via an input device or detected by the presentation according to one embodiment of the present invention. The module), or based on other variables. Such non-linear tem method begins with pre-production 401. Pre-production plate or Script allows for example for the production and comprises employing 402 a script and/or storyboard flow 15 presentation of a PG-rated, R-rated, or X-rated version of a chart, or employing 403 a digital template 403. A portion of given movie depending on the audience (for example a parent this front-end may be implemented automatically or manu may elect to view the R-rated version of the movie while ally in Software, comprising analysis, design, development, electing a PG-rated presentation for the children). As another production, implementation or evaluation of Script, story example, a wedding template or script may allow for different board, and/or digital template. Optionally, frames and/or presentations based on whether the bride's family or the scenes are labeled (via meta-data) according to Script, story groom's family is viewing. As another example, a mystery board, or digital template in use. presentation may offer alternate endings, based on viewer A script or storyboard is downloaded over a wired and/or input or external variables as described above. wireless network, made available via removable storage (e.g. Media appliance 100 senses 404 video and/or audio and memory card and/or disk), or is alternatively created on media 25 stores a digital representation in memory 110. Optionally, appliance. A digital template describes how to construct a multiple audio and/or video streams are sensed, either by the Video and/or multimedia document by sensing (i.e. 'shoot same media appliance or by collaborating media appliances, ing’ or recording) and assembling individual scenes and/or wherein synchronization is provided for the multiple streams, segments in particular order, and is downloaded over a wired in the form of meta-data tags describing related Scenes and/or and/or wireless network or created on media appliance. Alter 30 streams and/or frames, and/or in the form of meta-data natively, user of media appliance 100 may decide not to describing time stamps relating different scenes and/or consult a script, storyboard, or digital template, and proceed streams. For example, memory 110 stores a data structure directly to sensing 404. comprising one or more fields identifying one or more related One example of a template is a template for insurance Video scenes and/or streams and/or frames, and a field indi inspection of vehicle accidents, wherein the template indi 35 cating the nature of the relation (for example indicating that cates “slots’ for video clips, taken from various angles, of the the video scenes and/or streams and/or frames represented vehicles involved in the accident, as prescribed by an insur different viewing angles of the same sensed object). ance company. Media appliance 100 then post-produces the stored digital Optionally, media appliance 100 adaptively guides media representation, using controller 108 and/or audio or video appliance operator in making discretionary decisions to take 40 plugin stored in memory 110. alternate script paths and/or alter flow of script (or storyboard The post-produced digital representation is then stored 406 or digital template) or generally deviate from the script, for in memory 110 (or in other storage medium Such as optional example when dealing with emergency conditions and/or on-appliance hard-disk or storage tape for storing data), dis events which do not occuraccording to script. Such guidance played 407 on on-appliance display unit 114, and/or sent for may employ non-deterministic scripts, according to logic 45 off-appliance display and/or exhibition (e.g. for IMAX dis specified using Bayesian modeling, neural networks, fuZZy play according to IMAX 15/70 format, or for Texas Instru logic, and/or other technique for making decisions under ments DLP (Digital Light Processing) format), or for digital complex conditions and/or under incomplete information. remastering according to IMAX's DMR (Digital Remaster For example, in one embodiment a cast member in a script is ing) format, or for satellite distribution (e.g. to Digital Audio described by fuzzy attributes, such as “a female actor with at 50 Broadcast (DAB) distribution scheme to DAB enabled least five years drama experience” in leading role (instead of devices such as PDAs, cellular phones, personal audio and/or or in addition of identifying the lead role actor by name). Video players, or other devices for presenting audio and/or Then, in case the lead actor canceling her engagement, video). Optionally, communication of media appliance 100 instructions employing fuZZy logic perform a search for with other devices and/or services complies with ATSC actors matching the fuzzy attributes to dynamically recom 55 DASE (Advanced Television Systems Committee Digital TV mend one or more candidates to fill the role. Application Software Environment) architecture, incorpo Optionally, digital template or script is non-linear, allow rated herein by reference. ing for one or more branching points. A branching point FIG.5 is a flow diagram illustrating a method for optionally allows the script and/or template to flow in more than one filling-in a template according to a preferred embodiment of path. For example, Scene (or clip or stream) A can be followed 60 the present invention. Starting 501 with a template, sense 502 by scene B or scene C, depending on which branch of the a first scene according to the template, and fill-in 503 sensed branching point following A is taken. For a viewer, a media scene in template. If no additional scene is desired 505, finish presentation prepared according to Such non-linear template 506, else 504 proceed to step 502 and repeat until done. or script allows for a multiplicity of presentations comprising Template is stored in memory 110 comprising suitable format different scene (or clip or stream) orderings. For a viewer, the 65 such as the Advanced Authoring Format (AAF). decision of which of the alternate paths to follow in a branch FIG. 6 is a flow diagram illustrating a method for optionally ing point can be viewer selected, randomly chosen, based on tagging audio and/or video representation with information US 7,599,963 B2 11 12 contained in a meta-data structure. Upon sensing 601 a scene, I claim: the digital representation of the sensed scene is tagged 602 1. Network-extensible non-linear digital television pro with meta-data. Meta-data comprises time, media appliance gramming reconfigurable media appliance with storage location (such as provided by GPS module 112), media appli medium for digitally storing non-linear digital television ance orientation and/or media appliance acceleration (such as media program comprising one or more branching points in provided by acceleration detector 113), multi-lingual features Such program for cellular telephone having non-linear digital (allowing for translation, Subtitles, Voice-over, etc.), cues to a television multi-media stream processing applications com theater automation system (such as instructions for house prising: lights to go up, half-way up, or down, or instructions to open a cell processor comprising a first cell processing unit and or close curtains, etc.), instructions for allowing or disallow 10 a second cell processing unit, Such first and second cell ing content (such as trailers or promotional clips) to play next processing units enabled to process one or more non to other similar content, information indicating Suitability of linear digital television programming media streams in content for different audiences such as children, information serial or parallel using synchronized meta data tags to indicating any promotional offers, products and/or services provide extensible reconfigurable effects studio for cel (such as advertisements, product catalogs and/or coupons for 15 lular telephone or digital television application that pro products and/or services), information allowing for organiz duces seamlessly synchronized integration of such pro ing and/or managing meta-data available to advertisers and/or cessed multiple media streams comprising non-linear service providers, and/or other information describing, iden digital television programming; and tifying and/or relating to content. Tagging may be done per a memory for storing Such multi cell processing unit pro scene, per frame, per audio and/or video stream (e.g. when cessed multiple media streams, said memory compris multiple streams are present), or per other defined segment of ing a first media portion partitioned in a kernel space in audio and/or video. For example, a video scene is tagged with a storage medium; meta-data comprising a field identifying the language used in and a second media portion partitioned in a user space in the video scene. As another example, a video stream is tagged the storage medium; a non-linear digital television pro with meta-data comprising a field indicating a warning 25 gram being produced electronically via a renderer for against viewing by children. audio or video presentation using the first and second FIG. 7 is a flow diagram illustrating a method for transfer portions according to a non-linear digital television pro ring data and/or instructions from off-appliance source to gramming template or storyboard, such that the digital on-appliance memory. After determining 701 off-appliance television program presentation branches or flows from Source. Such as external repository (for templates, plugins, 30 the first portion to the second portion according to the DRM data, encryption keys, media clips, security data, bio non-linear digital television programming template or metric data, GPS data, etc.), proceed by transferring 702 data storyboard, such non-linear digital television program and/or instructions from determined off-appliance source to ming branching or flowing enabling a group of network on-appliance memory 110. extensible media appliances comprising one or more In one embodiment, media appliance 100 is a member of a 35 cellular telephone or digital television to collaborate to distributed group of media appliances 100, for example in a reconstruct a covered event effectively by Synchronizing distributed network of media appliances 100 and/or in a peer Such branching or flowing audio or video presentations to-peer configuration of media appliances 100. A media that are collected wirelessly from distributed media appliance 100 dynamically joins and/or leaves a distributed appliances, such that the group of one or more cellular group of media appliances 100, in parallel and/or serially with 40 telephone or digital television media appliances collabo other media appliances 100. Alternatively, media appliance rate or synchronize to provide wirelessly network-ex 100 initiates a distributed group of media appliances 100, tensible non-linear digital television programming; the allowing for other media appliance's 100 to dynamically join network-extensible reconfigurable media appliance and/or leave the group. In one embodiment, the group of comprising a cellular telephone configured programma media appliances 100 collaborates to cover an event, such as 45 bly to render non-linear digital television programming a sporting event, a public political event (e.g. a rally), a family using the cell processor and the memory to perform event (e.g. a wedding), or other event. Media appliances 100 programmable protocol translation, packet processing tag sensed audio and/or video data as described above (e.g. or format conversion of Such rendered non-linear digital with GPS information, time stamps, DRM meta-data, or other television programming. information previously described), allowing reconstruction 50 2. The apparatus of claim 1 wherein the memory further of covered event from the audio and/or video data collected comprising: by distributed media appliances 100. Memory 110 stores a third media portion partitioned in a trusted space in the instructions and/or data for initiating, joining, leaving and/or storage medium, Such that the program presentation querying the status of or information about Such a distributed branches or flows from the first or second portion to the group of media appliances 100. 55 third portion according to the non-linear digital televi Foregoing described embodiments of the invention are sion programming template or storyboard. provided as illustrations and descriptions. They are not 3. Network-extensible non-deterministic digital television intended to limit the invention to precise form described. In programming reconfigurable media appliance system with particular, it is contemplated that functional implementation on-appliance memory for accessing non-deterministic digital of invention described herein may be implemented equiva 60 television media programming for cellular telephone having lently inhardware, software, firmware, and/or other available non-deterministic digital television multi-media stream pro functional components or building blocks, and that networks cessing comprising: may be wired, wireless, or a combination of wired and wire a cell processor comprising a first cell processing unit and less. Other variations and embodiments are possible in light a second cell processing unit, Such first and second cell of above teachings, and it is thus intended that the scope of 65 processing units enabled to process one or more non invention not be limited by this Detailed Description, but deterministic digital television programming media rather by Claims following. streams in serial or parallel using synchronized meta US 7,599,963 B2 13 14 data tags to provide extensible reconfigurable effects reconstruct a covered event effectively by Synchronizing studio for cellular telephone or digital television appli Such branching or flowing program presentations that cation that produces seamlessly synchronized integra are collected wirelessly from distributed media appli tion of such processed multiple media streams compris ances, such that the group of one or more cellular tele ing non-deterministic digital television programming; phone or digital television media appliances collaborate and or synchronize to provide wirelessly network-extensible a memory for storing Such multi-cell processing unit pro non-deterministic digital television programming; the cessed multiple media streams, said memory compris network-extensible reconfigurable media appliance ing a first memory element comprising a pre-produced comprising a cellular telephone configured programma portion; and a second memory element comprising a 10 bly to render digital television programming using the post-produced portion; Such that a news, sports, movie, cell processor and the memory to perform program game, family or political audio or video digital televi mable protocol translation, packet processing or format sion program is presented electronically using the pre conversion of Such rendered non-deterministic digital and post-produced portions according to a non-deter television programming. ministic digital television programming template or sto- 15 4. The system of claim 1 wherein the memory further ryboard, Such that the digital television program presen comprising: tation branches or flows from the first portion to the a third memory element comprising a third portion, Such second portion according to the non-deterministic digi that the program presentation branches or flows from the tal television programming template or storyboard. Such pre- or post-produced portion to the third portion non-deterministic digital television programming according to the non-deterministic digital television pro branching or flowing enabling a group of network-ex gramming template or storyboard. tensible media appliances comprising one or more cel lular telephone or digital television to collaborate to k k k k k