US007667704B2
(12) Ulllted States Patent (10) Patent N0.: US 7,667,704 B2 Hogle (45) Date of Patent: Feb. 23, 2010
(54) SYSTEM FOR EFFICIENT REMOTE 2002/0085020 A1 7/2002 Carroll, Jr. PROJECTION OF RICH INTERACTIVE USER 2002/0105546 A1 8/2002 KuntZ INTERFACES 2002/0130904 A1 9/2002 Becker et al. (75) Inventor: Francis Hogle, Bellevue, WA (US) 2002/0180803 A1 12/2002 Kaplan et 31' 2002/0196268 A1 12/2002 Wolffet al. (73) Assignee: Microsoft Corporation, Redmond, WA 2002/0199190 A1 12/2002 Su (US) 2003/0061279 A1 3/2003 Llewellyn et al. 2003/0078972 A1 4/2003 Tapissier et al. ( * ) Notice: Subject' to any disclaimer, the term of this 2003/0079038 A1 4/2003 Robbin et a1‘ patent 1s extended or adjusted under 35 U.S.C. 154(b) by 463 days. (21) Appl. N0.: 11/095,255 (Continued) (22) Filed: Mar. 30, 2005 FOREIGN PATENT DOCUMENTS
(65) Prior Publication Data EP 1376331 1/2004 US 2006/0227141 A1 Oct. 12, 2006
(51) Int- 0- OTHER PUBLICATIONS G06T 13/00 (2006.01) (52) US. Cl...... 345/473; 345/589; 345/156; IEEE Std 802.l5.l IEEE Standard for Information technology- Tele 345/169; 725/112; 709/227 communications and information exchange between systems- Local (58) Field of Classi?cation Search ...... 345/419, $91 Ilnetrigocllimn grea “grow-(iii? rcfcgllliremjlnis PMEPIHSI _ lI‘e ess e llln'l ccess on T0 an ySlC ayer See a licatiosgli/igigig?’1i§e6;;:r2i111;2{O709/227 Speci?cations for Wireless Personal Area Networks (WPANs), 2002, PP P 1'3" pp. 49 and 61.*
( 56 ) R e 1'erences Ct1 e d (Continued) U.S. PATENT DOCUMENTS Primary ExamineriKimbinh T Nguyen 5,809,497 A 9/1998 Freund et al. (74) Attorney, Agent, or FirmiLee & Hayes, PLLC 6,084,582 A 7/2000 Qureshi et al. 6,342,907 B1 1/2002 Petty et al. (57) ABSTRACT 6,366,914 B1 4/2002 Stern 6,452,609 B1 9/2002 Katinsky et al. 6,460,058 B2 10/2002 Koppf?u et al' An exemplary method of communicating between a host 6,731,312 B2 5/2004 Robbm device and a rendering device for controlling a user interface * 1A2‘; §emth of the rendering device includes generating messages, encod ’ ’ annu et a1‘ """""""" " 709/206 in the messa es in a buffer as blobs of data and communi 7 203 678 B1 4/2007 P 1 1 131 g g 7’246’ 134 B1 7/2007 K121133158 e ' cating the blobs of data to the rendering device. Various other 7’283’l2l B2 10/2007 Adan et a1 ' exemplary methods, devices, systems, etc., are also disclosed. 2002/0015042 A1* 2/2002 Robotham et al...... 345/581 2002/0078467 A1 6/2002 Rosin et al. 16 Claims, 6 Drawing Sheets
EXEMPLARV COMPUTING ENVIRONMENT m 830 620 (ROM EU. BIOS @ Processing Unit 590 (RAM £32 Operating Video 695 System EA Adapter 0mm“ 4 > Peripheral Application Interface Programs
User Input Olher Program Intertaoe Modules £15 Program Bat?37
"""°‘“°" Stem Prams Modules Data m e13 JWAN 652 .. 1 544 645 646 $47 685 ’ Keyboard A linations US 7,667,704 B2 Page 2
US. PATENT DOCUMENTS “Registering an Application Under the Start Menu Category”, retrieved on Apr. 9, 2009 at <
Developers
Rendering I Device US. Patent Feb. 23, 2010 Sheet 2 of6 US 7,667,704 B2
Synchronous Communication Protocol Timeline 200 Host Rendering Device Ati Host Create RB ' Ati NetworkHRi CD Ati NetworkRH i
Draw Solid Fill G)
Draw Circle (9
Set Visual Content 5
5 Visible US. Patent Feb. 23, 2010 Sheet 3 of6 US 7,667,704 B2
Asynchronous Communication Protocol Timeline E
Host Rendering Device
Create RB Ati NetworkHR i @ Draw i T Ate Solid Fill ~> RD Draw T i Circle SetVisual me n+1 Content ~> @ Destroy T i RB és, ' T Visible Ati NetworkRH i Frame n+2
Fig. 3 US. Patent Feb. 23, 2010 Sheet 4 of6 US 7,667,704 B2
Exemplary Asynchronous Communication Protocol Timeline m Host Rendering Device to-—— 6) CreateDraw RB S°“dF"' E @ Circle AtBaich 5g @ SetC Visualt t E G) on e28 AtBatch Network 5 Destroy i @El tr —. I At Network i
4.-.-.-"...... 4
Fig. 4
US 7,667,704 B2 1 2 SYSTEM FOR EFFICIENT REMOTE cating the blobs of data to the rendering device. Various other PROJECTION OF RICH INTERACTIVE USER exemplary methods, devices, systems, etc., are also disclosed. INTERFACES BRIEF DESCRIPTION OF THE DRAWINGS RELATED APPLICATIONS Non-limiting and non-exhaustive examples are described With reference to the folloWing ?gures, Wherein like reference This application is related to US. patent application Ser. numerals refer to like parts throughout the various vieWs No. 11/095,758 entitled, “Enabling UI template customiZa unless otherWise speci?ed. tion and reuse through parameteriZation”, to Glein, Hogle, FIG. 1 is a diagram of an exemplary system that includes Stall, Mandryk and Finocchio, ?led on Mar. 30, 2005, (Which developers, a server, a host and one or more renderers. is incorporated by reference herein); US. patent application FIG. 2 is a timeline for a synchronous communication Ser. No. 11/095,764 entitled “System and method for protocol for a rendering device receiving instructions for a graphical user interface from a host device. dynamic creation and management of lists on a distance user FIG. 3 is a timeline for an asynchronous communication interface”, to Ostojic, ?led on Mar. 30, 2005, (Which is incor protocol for a rendering device receiving instructions for a porated by reference herein); and US. patent application Ser. graphical user interface from a host device. No. 1 1/ 095,746 entitled “Context menu navigational method FIG. 4 is a timeline for an exemplary asynchronous com for accessing contextual and product-Wide choices via remote munication protocol for a rendering device receiving instruc control”, to Ostojic, Glein and Sands, ?led on Mar. 30, 2005, 20 tions for a graphical user interface from a host device. (Which is incorporated by reference herein). FIG. 5 is a diagram of an exemplary system that includes various components for a host computer and a rendering TECHNICAL FIELD device Wherein communication may occur according to an exemplary protocol. 25 FIG. 6 is a diagram illustrating an exemplary computing Subject matter disclosed herein relates generally to com environment, Which may be used to implement various exem munication of one or more messages for instructing a graphi plary technologies described herein. cal user interface. DETAILED DESCRIPTION BACKGROUND 30 In the description that folloWs, various exemplary meth ods, devices, systems, etc., are presented. User interface systems typically face a trade-off betWeen FIG. 1 shoWs an exemplary system 100 that includes a high interactivity via local program code and projection to, developer environment 110, a server 120 and a client envi typically, remote rendering devices, Where netWork band 35 ronment 130. Of course, many such client environments may Width impedes their ability to provide high interactivity. be in communication With the server 120. Developers in the Many systems simply accept degraded interactivity or quality developer environment 110, as represented by Workstations When using a rendering device or “renderer”. Other systems 112, 112', develop applications for distribution or use by the explicitly provide snippets of helper code to execute on ren server 120. In turn, the server 120 alloWs the host 132 in the dering devices. Still others adopt an explicit host/renderer 40 client environment 130 to doWnload such applications, coop model for user interface, forcing developers to design tWo erate With such applications, etc. Whole pieces of softWare that communicate With each other. The host 132 in the client environment 130 hosts one or Finally, some approaches simply demand higher perfor more rendering devices 134, 136. As described herein various mance rendering device hardWare and connectivity to provide exemplary methods, devices, systems, etc., provide for e?i richness in such scenarios. 45 cient rendering of a user interface on a rendering device in communication With a host. Such a user interface may be These systems have signi?cant drawbacks. Degrading directed by native applications of a host or by non-native quality affects end user results. Requiring snippets of helper application. Such non-native application may be controlled to code complicates the programming and interaction models. some degree by a remote server. Communication betWeen the Renderer/host computing forces each application developer 50 host and the one or more rendering devices 134, 136 may be to develop explicitly for each possible rendering device, unidirectional, bi-directional or multi-directional. Particular thereby limiting the reach or prohibitively increasing the cost examples, discussed herein refer to unidirectional and bi of many solutions. Requiring higher performance rendering directional but may also be implemented in multi-directional device hardWare and connectivity also carries prohibitive manner Where more than one host exists. In some instances, cost. 55 rendering device to rendering device communication may Subject matter presented herein addresses these and other occur. issues, for example, by judicious selection of renderer and In the exemplary client environment 130, the rendering host runtime components, communications protocol design device 134 represents a smaller rendering device such as a and programming model. handheld device or a controller for a “smart” appliance, secu 60 rity system, environmental system, etc. The rendering device 136 represents a larger rendering device such as a TV, a SUMMARY projector, a monitor, etc., Which may be used to vieW TV shoWs, movies, neWs broadcasts, sports, etc. Other exemplary An exemplary method of communicating betWeen a host rendering devices may be picture frame devices, set top device and a rendering device for controlling a user interface 65 boxes, cell phones, game boxes and the like. While a render of the rendering device includes generating messages, encod ing device is generally not intended to serve as a substitute for ing the messages in a buffer as blobs of data and communi a personal computer; various exemplary methods, protocols, US 7,667,704 B2 3 4 etc., presented herein may be implemented for communica ality as Well. The renderer is generally softWare that can tion between a host and a personal computer. In such receive instructions (e.g., from a user interface frameWork) examples, the host may be a personal computer or other and, in turn, direct hardWare to cause the display of a user computer. interface, user interface components, etc. (e. g., graphics, ani While some rendering devices may have memory or stor mations, etc.). Conventional renderers typically support one age capacity in excess of a typical personal computer, the or more standards such as, but not limited to, DirectX graph processing resources are usually dedicated to special purpose ics (e.g., DirectDraW, Direct3D), OpenGL graphics (Open tasks. For example, a set top box may have RAM to capture Graphics Library), or GDI graphics (Graphical Device Inter HDTV frames and a high capacity hard drive to store full face). length movies yet have limited ability to run business appli Referring again to the timeline 200 of FIG. 2, the timeline cations that run on most personal computers. To construct a 200 indicates When various events occur on the host side and set top box to run such business applications Would increase on the rendering device side, Where the host and rendering cost of the set top box and potentially complicate program device communicate via a netWork, a Wireless link, a cable, ming of dedicated audio-visual tasks. etc. As already mentioned, rendering devices come in all types The events correspond to rendering tasks for display of a of shapes and siZes With varying capabilities to perform many user interface by the rendering device. The tasks originating different types of tasks. To require manufactures to beef-up With the ho st include: (1) create render builder (RB); (2) draW resources to facilitate the job of a host, is unrealistic, espe solid ?ll; (3) draW circle; (4) set visual content; and (5) cially in the home market Where incremental cost can be a destroy render builder (RB). signi?cant factor in implementation. Instead, in such circum 20 The render builder creation task (1) and destruction task (5) stances (e.g., Whether home or other), the host device should basically serve as a start point and an end point, respectively, generally be capable of bearing the bulk of the burden When for one or more draWing or related messages (e. g., tasks presentation of or interaction With a user interface on a ren (2)-(4)), Which in the synchronous communication protocol dering device is required. timeline 200 are communicated in series of communications Referring again to the rendering device 136, such a device 25 from the host to the rendering device. may present a user interface that Works Well at a distance of The rendering device receives instructions or metadata for about ten feet and may be controlled via a remote control. A the tasks (l)-(5) and processes the instructions (e.g., using a typical remote control is smaller and easier to use than a renderer) upon receipt via the netWork. A delay exists conventional keyboard and mouse; hoWever, it generally pro betWeen the host sending the instructions or metadata vides a more limited form of user input (e.g., due to feWer 30 (throughout this description “or” generally means “and/ or”), keys or buttons). And While a greater vieWing distance pro Which is represented by Atl- NetWorkHR. The host Atl- Host has vides a more comfortable experience, it can necessitate fea a delay and the rendering device has a delay Atl- RD. Further, tures that provide a visual design style to ensure clarity, coher Where the rendering device sends information to the host, ence, and readability. In addition, a user may expect the user another delay occurs Atl- NetWorkRH. Thus, for the example of interface to have display quality equivalent to that of a con 35 FIG. 2, each task has four associated delays: Atl. NetWorkHR, ventional TV or HDTV Where frames are displayed smoothly, Atl- NetWorkRH, Atl- Host, and Atl- RD. i.e., consecutive frames are displayed at a particular frame Host resources may be increased or optimiZed to diminish rate that provides the user or vieWer With an uninterrupted, Atl- Host and netWork tra?ic may be reduced or optimiZed to glitch-free presentation of media. diminish Atl- NetWorkHR and Atl- NetWorkRH; hoWever, as A user may also expect the rendering device like the device 40 described herein, rendering device capabilities are not part of 136 to provide a dynamic, animated experience. A user may a solution to increased performance for user interfaces. Thus, also expect that input Will be easy and make her experience Atl. RD is considered rendering device dependent and a limi simpler, not more complicated. A user may also expect appli tation. cations to be more convenient, simpler to learn, and easier to Referring again to the timeline 200, the rendering device use than applications controlled by the keyboard or mouse. 45 may actually display the circle after receipt of the “set visual Various exemplary methods, devices, systems, etc., content” task and processing of that task by the rendering described herein rely on asynchronous communication, device. At the time of display, seven netWork delays, four host Which may, for example, reduce risk of interruptions, delays and four rendering device delays have occurred. Fur glitches, dropped frames, delayed frames, etc ., by a rendering ther, Where netWork tra?ic is signi?cant or other host tasks device. As explained in more detail beloW, an exemplary 50 many, probability of additional delays exist. Thus, a compel asynchronous communication protocol reduces netWork time ling need exists to reduce the number of netWork communi and can provide a better user vieWing experience When com cations betWeen the host and the rendering device. pared to synchronous communication protocols. A typical FIG. 3 shoWs a typical asynchronous communication pro synchronous communication protocol is presented With tocol timeline 300 in relationship to the same tasks as pre respect to FIG. 2, folloWed by a typical asynchronous com 55 sented in FIG. 2. HoWever, the timeline 300 also shoWs munication protocol, FIG. 3, and various exemplary asyn frames displayed by the rendering device (e. g., Frame n, chronous communication protocols (e.g., FIG. 4). Frame n+1 and Frame n+2). These frames represent addi FIG. 2 shoWs a typical synchronous communication pro tional tasks to be performed by the rendering device that may tocol timeline 200 for communication betWeen a host and a be interrupted, shifted or not displayed due to execution of rendering device. As communications occur from rendering 60 host communicated tasks. As mentioned, such con?icts can device to host as Well, the timeline 200 exhibits bi-directional tarnish a user’s experience. communication. In general, the host may rely on a “frame The render builder creation task (1) and destruction task (5) Work” that issues messages for a “renderer” of a rendering basically serve as a start point and an end point, respectively, device. The frameWork typically acts as an intermediate for one or more draWing or related messages (e. g., tasks betWeen an application and a communication layer associated 65 (2)-(4)), Which in the asynchronous communication protocol With a communication interface. The frameWork may be dedi timeline 300 are communicated in series of communications cated to user interface tasks or it may provide other function from a host to a rendering device. US 7,667,704 B2 5 6 The asynchronous communication protocol represented by As the exemplary asynchronous communication protocol the timeline 300 uses feWer rendering device to host commu reduces the number of communications and the amount of nications than the synchronous protocol represented by the processing time required by the rendering device, there is less timeline 200. In particular, the number is greatly reduced by chance that video Will be interrupted or otherWise compro the fact that the rendering device only sends one communi mised by the batch of tasks. Overall, this alloWs the rendering cation to the host, Which occurs after execution of all host device to provide the user With a better vieWing experience. requested tasks. However, the host still sends ?ve separate Graphical user interfaces are often constructed out of many communications to the rendering device. Thus, the total num small parts. In a message oriented model, each message may ber of communication is six as compared With 10 for the be an update to one aspect of one part. For a single logical synchronous timeline 200. change, an application or applications may require making Having explained a typical synchronous communication several changes to various parts of a user interface, preferably protocol and a typical asynchronous communication, con at one time (e.g., substantially instantaneously, in an anima sider the exemplary asynchronous communication protocol tion sequence, etc.). If these changes are not performed presented With respect to the timeline 400, Which, as shoWn, atomically, the user may see unintended partial states. For provides for bi-directional. In other examples, unidirectional example, consider the timeline 300 of FIG. 3 and the timeline or multi-directional communication may occur using an 400 of FIG. 4. In the timeline 300 of FIG. 3, the messages exemplary communication protocol, as appropriate. In the arrive over the course of multiple frames. In terms of a user’ s example of FIG. 4, the exemplary asynchronous communi visual experience, for example, unless another technique or cation protocol only requires one communication from the method is employed to ensure atomicity on the renderer side, ho st to the rendering device and one communication from the 20 the intervening frames Will effectively present on the user rendering device to the ho st, Which may be optional. Thus, the interface partially updated states Which may not be visually render builder creation task (1) and destruction task (5) basi desirable. To avoid such deleterious behavior, for the timeline cally serve as messages that hold, therebetWeen, one or more 300 of FIG. 3 and associated protocol, one could implement draWing or related messages (e. g., tasks (2)-(4)). In contrast to renderer side management of atomicity, Which, in turn, Would the protocols of the timeline 200 and the timeline 300, the 25 increase implementation complexity and cost (e.g., for the creation task (1) and the destruction task (5) are, in the asyn rendering device). In contrast, the exemplary timeline 400 of chronous communication protocol timeline 400, communi FIG. 4 and associated exemplary protocol, together With vari cated in a single communication from the host to the render ous exemplary methods, can place the burden on the ho st side, ing device, along With tasks (2)-(4). Of course, other Which can also alloW for use of lighter-Weight rendering arrangements are possible. 30 devices. As described herein, an exemplary asynchronous commu An exemplary method may arrange messages (e.g., com nication protocol system can enable relatively “chatty” or mands or tasks) into one or more atomic units. Each unit may “rich” messages (e.g., plurality of messages) to be sent in a be a message or a batch of messages. Such an exemplary “chunky” manner. The exemplary asynchronous communi method can provide for atomicity. For example, a particular cation protocol may even disalloW all or certain synchronous 35 set of draWing commands may provide a certain user experi calls, thus amortiZing latency cost over large numbers of ence When executed on a rendering device and if not arranged batched messages (i.e. carrying chatty communications in or grouped into an atomic unit, a risk may exist for degrading chunky Wire transactions). Thus, an exemplary communica the quality of the user experience. Thus, an exemplary method tions model may be substantially 100% asynchronous. may logically group messages based on a desired result for a 40 user. Further, such an exemplary method may account for In the example of FIG. 4, the number of communications display of other information or media such as video Where a from the host to the rendering device is reduced by preparing frame rate or a frame-to-frame integrity should be maintained a batch, Which has a delay AtBatch Host, instead of separate to provide the best user experience. and distinct individual delays as in the aforementioned syn An exemplary method may arrange messages (e.g., com chronous and asynchronous examples. Further, the rendering 45 mands or tasks) for graphics or animations together With device may process the communicated batch instructions or audio (e.g., beeps, sounds, music, etc.). For example, mes metadata as a batch With a delay AtBatch RD. In general, batch sages for an animation combined With a “Wooshing” sound processing, Whether by the host or the rendering device, is may be communicated in as a batch of messages. Thus, as more e?icient than processing of tasks serially Whereby serial described herein, messages related to audio, other control of processing is delayed and governed by serial communication 50 a rendering device, etc., may be communicated using various of the tasks. exemplary methods, protocols, etc. For example, Where a In the example of the timeline 400, the host process the ?ve rendering device controls an appliance (e. g., a refrigerator), a tasks in a substantially uninterrupted manner, Which results in graphic, a sound and a temperature control signal may be sent a substantial time saving When compared to serial processing as a batch ofmessages. of the ?ve tasks Where communication With a rendering 55 The exemplary asynchronous communication protocol device occurs after processing of each individual task. As may be referred to as using phases. For example, the timeline shoWn, the host requires a period of time from tO to tf, Which 400 includes a host batch processing phase, a communication is normally less than the sum of the individual task processing phase from host to rendering device and a rendering device times as shoWn in the timeline 300 of FIG. 3. batch processing phase. In the example of the timeline 400, the ho st may encode the 60 The exemplary asynchronous communications protocol ?ve tasks in a netWork buffer. The process of encoding a batch described With reference to the timeline 400 may be used for of information in a netWork buffer or communication buffer is communications betWeen any suitable sender and receiver. typically more e?icient than the sum of times required to For example, a frameWork and a renderer may include an encode a series of individual tasks in a netWork buffer or exemplary communications protocol layer Where the proto communication buffer. Thus, a reduction in time occurs on 65 col layer is an abstraction that provides services or operations the host side via the exemplary asynchronous communication for of transfer information betWeen the frameWork and the protocol. renderer and optionally one or more intermediaries. An exem US 7,667,704 B2 7 8 plary communication protocol optionally alloWs for exten In the exemplary system 500, the exemplary communica sions, for example, one or more modules that de?ne message tion protocol may be used to connect a high-performance segments and that may de?ne associated processing rules. messaging model to an instance of the renderer 530 running An exemplary communications protocol uses a message as on the rendering device 506. Typically, the renderer 530 is a a basic unit Where a communication may be a message or a softWare interface for graphics/animation hardWare of the batch of massages. An exemplary communications protocol rendering device 506. The renderer 530 includes various lay may operate according to a messaging model, Which may be ers such as an object layer 532 that manages objects. Man considered a messaging layer cooperative With a protocol agement of objects may include creation of objects, naming layer. An exemplary protocol layer optionally controls mes of objects, accessing of objects, etc. sages in terms of number of messages sent, types of messages Features provided in a “graphics” language, speci?cation sent, batch siZe, etc. or interface generally form a basis for the animation layer 536 An exemplary communications protocol speci?es a mes and the graphics layer 538. Consider, for example, the Open saging model and a graphics model for use in rendering user Graphics Library (OpenGL), Which is a speci?cation de?ning interfaces or one or more components thereof on a rendering a cross-language, cross-platform application program inter device Where messages originate from or are intermediate to face (API) for Writing applications that produce 2D or 3D a host device. Such messaging model or graphics model may computer graphics. The OpenGL interface includes about include objects for performing tasks or representing features 250 different function calls Which canbe used to draW various related to a user interface. An exemplary host device may graphics (eg from simple primitives to complex three-di include a framework that alloWs an application to success mensional scenes constructed from simple primitives). fully render user interfaces, user interface components, etc., 20 OpenGL is also used in the video games industry as are other on a rendering device Whereby communication betWeen the speci?cations, for example, DirectX (e.g., DirectDraW, host device and the rendering device occurs according to an Direct3D, etc.). HTML or other markup language-based exemplary communication protocol (e.g., the asynchronous instructions may be used for graphics, etc. communications protocol as described With respect to the As described herein, an exemplary renderer typically timeline 400 of FIG. 4). 25 includes an application program interface (API) for de?ning While the exemplary asynchronous communication proto 2D and 3D graphic images (e.g., for use in a graphics layer col described With reference to the timeline 400 may be 538) that may provide some degree of compatibility across implemented using equipment such as that shoWn in FIG. 1, hardWare and operating systems and specify a set of messages FIG. 5 shoWs a more detailed exemplary system that may (e.g., suitable for use as messages). Speci?c graphics/anima implement the exemplary asynchronous communication pro 30 tion capabilities exposed via such anAPI may include hidden tocol of FIG. 4. surface removal, alpha blending, antialiasing, texture map The exemplary system 500 includes a host computer 504, a ping, pixel operations, vieWing and modeling transforma rendering device 506 and a communication link(s) 520 Where tions, atmospheric effects or other capabilities. Such graphic the dashed line 540 represents communications betWeen the rendering capabilities can typically be used to create high host computer 504 and the rendering device 506. The host 35 quality still and animated three-dimensional color images computer 504 includes an application 508 and a framework (consider, e.g., the animation layer 536). In general, a render 51 0 that operates in conjunction With the application 508. The ing device Will include a renderer With at least some basic frameWork 510 includes various frameWork components animation capabilities. such as, but not limited to, a graphical user interface (GUI) As already mentioned, the communication layer 534 of the abstraction layer 512, a scheduling layer 514, an object layer 40 515, other services 516 and a communications layer 518. The renderer 530 communicates With the communication layer rendering device 506 includes a renderer 530, Which may be 518 of the frameWork 510. The communication layer 534 may alloW for communication With devices other than the host softWare, hardWare, a combination of softWare and hardWare, etc. The rendering device 506 may be a remote device, for computer 504 and may optionally alloW for communication With other frameWorks. The rendering device 506 may example, located remote from the host computer 504. As 45 shoWn in FIG. 5, the renderer 530 includes an object layer include other layers not shoWn, such as, but not limited to, a 532, a communications layer 534, an animation layer 536, driver layer that describes one or more speci?c softWare inter and a graphics layer 538. faces for hardWare used in rendering and a hardWare layer that The exemplary system 500 may use a high-performance describes hardWare that produces a display. asynchronous messaging model that describes logical render 50 Referring to the exemplary frameWork 510 of the ho st ing and interaction for a remote user interface associated With computer 504 shoWn in FIG. 5, the frameWork 510 includes the rendering device 506. Exemplary protocol elements used the GUI abstraction layer 512 that typically provides a set of in communication betWeen the host computer 504 and the APIs for various GUI elements. The GUI abstraction layer rendering device 506 may include objects to maintain an 512 may interact With the object layer 515 as objects are active connection and facilitate various communication pat 55 created, named, etc., or other layers. The other services 516 terns. Exemplary protocol objects may include a broker may include draWing services, markup language (e.g., XML) object as a root object in the protocol; a context object that parsing services, directional navigation services, data binding represents a logical computation site; a render port callback services, command interpreter services, animation control object for callback from a render port in runtime (the render system services, UI description template model services, UI port may be referred to as a message port, typically for ren 60 layout system services, etc. derer related messages); a data buffer callback object for In general, the frameWork 510 alloWs an application 508 to reporting on the status of bulk data transfers; and a data buffer request presentation of various information or graphics by the object that represents bulk data on a remote node. The exem rendering device 506, for example, to a user of the rendering plary system 500 may operate according to the exemplary device 506. The frameWork 510 may alloW for such requests communication protocol described With respect to the time 65 to more than one rendering device. In general, the exemplary line 400 of FIG. 4 Where the protocol uses one or more of such system 500 can provide for high-?delity remote projection of exemplary objects or other objects. user interfaces or user interface components by leveraging an US 7,667,704 B2 10 exemplary protocol that can batch messages and communi may include various layers such as a buffer layer (e.g., for cate such batches (e.g., as communications) in an asynchro basic framing of protocol interactions) and an object messag nous manner. ing model layer (e.g., for lifetime management and commu The exemplary system 500 or components thereof may nication and where messages are carried in buffers and always implement various exemplary methods. Of course, other sys refer to an object instance). With respect to buffers, message tems or components may be used to implement various exem buffers (e.g., a message or a batch of messages) may be plary methods (see, e.g., devices of FIGS. 1 and 6). An exem passed to an object layer directly for processing and discarded plary method may include communicating one or more when processing returns, data buffers may be reserved for messages from an application executing on a ho st computer to various content and registered with an object layer upon a rendering device wherein the communicating uses an exem arrival, and buffers may be symmetric and bidirectional plary asynchronous communication protocol. For example, where the same model for ho st device to rendering device and an application may issue messages and a framework may vice-versa. The messaging model may include object imple form a batch of the messages and communicate the batch of mentation instances on host device and rendering device. All messages as a communication to a rendering device where, objects may optionally exist in a per-session table, have a upon receipt of the batch, the rendering device performs unique handle and be associated with an implementation rendering according to the messages. An exemplary frame “class”. An instance of an object may be referred to by a work may receive one or more messages from the remote handle on the wire and an asymmetric ownership model may device in response to receipt of the batch of messages (or any be supported. In general, a messaging model may provide that message), execution of one or more of the messages, render a host device does not need to wait for individual replies (i.e., ing according to one or more of the messages, user interaction 20 e?icient asynchrony). with the rendering device in relationship to information in the In general, asymmetric ownership refers to a master/ slave batch, etc. ownership model for device resources. In an exemplary own A priori knowledge of the rendering capabilities of the ership model that may be implemented with various exem rendering device may allow an exemplary framework to plary techniques presented herein, host side management of cooperate with the application or limit the application to a 25 protocol-visible object may occur. For example, a host side certain set of messages and to control the number of messages manager may manage the lifetimes of all protocol-visible or types of messages in a batch. A priori knowledge of the objects, regardless of which end they live on (e.g., rendering current state of the rendering device may allow an exemplary device/renderer end or host device/framework end). Such an framework to schedule messages or batches of messages exemplary ownership model can provides bene?ts such as (e. g., consider the scheduling layer 514), for example, to 30 allowing object creation and destruction to be asynchronous avoid interrupting or interfering with a user experience of a (not just messages to the objects), which can provide for user of the rendering device. A priori knowledge of the com higher performance; allowing easy “reconstitution” of a ren munication link’s characteristics (e.g., bandwidth, usage, derer state after connectivity loss or device reset (e. g., without etc.), may allow an exemplary framework to schedule or resetting the host); and relieving the renderer implementation otherwise control batches of messages. While various fea 35 of the need for complex allocation and management code for tures are mentioned individually for an exemplary frame such resources, which, in turn, can reduce cost. work, a plurality of such features may be optionally combined With respect to messages, a messaging model may provide for scheduling, control, providing a suitable user experience, for messages as siZed blobs of data that may be associated etc. with a unique message number where numbering is option As already mentioned, an exemplary protocol may include 40 ally per-class. A messaging model may provide for a non-Zero various objects: a broker object, a context object, a renderport destination object handle where instance messages are sent to callback object (or a message port callback object), a data object’s handle and static messages are sent to class’s object buffer callback object, a data buffer object, etc. These objects handle and provide for a message number. A messaging may form part of a messaging model. An exemplary protocol model may provide for optional payload, for example, packed may also include additional de?nitions layered atop such 45 at the end. The already mentioned broker object may be a root messaging de?nitions or objects to form a graphics model. class used for bootstrapping an exemplary protocol that exists For example, an exemplary protocol may include a graphics with a well-known handle. Such a broker may include only model that includes a render builder object to hold a packet of static methods for basic services, for example, class discovery drawing commands (e.g., graphic, animation, etc., com (e.g., creates a class instance for the speci?ed name if avail mands), a window object that represents a presentation con 50 able), instance creation (e.g., creates an object instance for the tainer, a visual object that represents a discrete coordinate speci?ed class handle, supports passing an optional “con space, a device callback object for reporting on the status of a structor” message), and instance destruction (e. g., destroys an device, a device object that represents a rendering technology object by handle). In general, such messaging model objects driver, a surface object that represents a logical container for are not used directly by an application requesting rendering of image bits, a surface pool object that represents a physical 55 a user interface (U I) or UI component on a rendering device. container for image bits, a gradient object for modi?cation of An exemplary method uses an exemplary protocol for color space of other primitives in a container scope, an ani communicating messages or information from an application mation manager object that serves for animation control, an to a renderer of a rendering device. For purposes of illustrat animation callback object that reports on an animation, and an ing such an exemplary method, various objects of aforemen animation object that describes a logical animation sequence. 60 tioned messaging models and graphics models are used in As discussed herein, a message may be a command, informa presenting exemplary protocol traf?c. In particular, the tion, etc. sample trace that follows (Sample Trace 1) corresponds to a Thus, an exemplary protocol may use a messaging model single wire message to create a new RenderBuilder object. and a graphics model. The messaging model may act to sim The actual message sent is a static CreateObject call to the plify implementation of a core protocol, allow for extensions, 65 Broker where the new trace is emitted to exhibit processes allow for suitable decoupling of host device and rendering such as parameters to the constructor (invoked by the Broker device communication timing, etc. The messaging model after it creates the instance). Such a trace (Sample Trace l) US 7,667,704 B2 1 1 12 may occur, for example, in the exemplary system 100 trace may be available Where the trace represents a buffer of between one of the rendering devices 134, 136 of FIG. 1 as protocol data being submitted for transfer to the renderer. driven by an application executing on the host device 132. Another sample trace (Sample Trace 3) folloWs that is trace of messages sent to a renderer by an exemplary frameWork Where a request Was made to change focus betWeen tWo GUI Sample Trace 1 buttons. More speci?cally, the folloWing batch describes the state and animation changes required to shift focus from one (new RenderBuilder [class 1000003], constructor MsglfCreate button to another (e. g., focus on a GUI button on a rendering as MessagePtr) device of FIG. 1), for example, Where a button is composed of cat = InScene 16 bytes total multiple layers and a continuous pulsating overlay animation. send Broker [class 1000000] static MsglfCreateObj ect Sample Trace 3 involves a user pressing an arroW key on a idObjectClass = RH:1000003 remote control, so the ?rst feW messages relate to input state idObjectNeW = RH:5000146 (eg hide the cursor and ensure proper focus) and the rest of msgConsttuction = [blob MessagePtr] 215548588 +16 bytes blob data the messages relate to visual rendering state and animations: 40 bytes total
In this sample trace, the siZe of the constructor matches the Sample Trace 3 siZe of the BLOB data for the msgConstruction parameter to send FormWindoW: 100001 F Ms g1 6iForceMouseIdle 20 the CreateObject call. In this sequence, the total amount of fIdle = True messaging bytes sent is 40, not 56. While BLOB may refer to 16 bytes total a binary large object, siZe of such data can depend on various send FormWindoW: 100001 F Ms g1 5iTakeFocus 12 bytes total factors and can be of the siZe of the examples or of other siZes; send Animation:7000062 Msg4iStop thus, the term BLOB may not alWays refer to a “large” object. 12 bytes total While data is generally binary data as communicated, in 25 send Visual: 1 000 177 Msg23iSetVisible various examples, data may optionally be other than binary fVisible = False data. 16 bytes total send Visual: 1 000 177 Ms gOfChangeDataBits When a batch of messages is communicated to a renderer nValue = 0 (e. g., on a remote rendering device), a callback may be added nMask = 7340039 so the application side messaging code can knoW When the 30 20 bytes total send Visual: 1 000 177 Ms g22iSetContent buffer Was actually processed. Thus, the last message in any rbContent = RenderBuilder:0 batch buffer may be a ForWardMessage call instructing the 16 bytes total renderer to send the enclosed message back to the application send Visual: 1 000 1 85 Msg23iSetVisible side. In this example, the enclosed message payload is an fVisible = True 35 16 bytes total OnBatchProcessed call back into the messaging runtime send Visual: 1 000 1 85 Ms gOfChangeDataBits itself (see Sample Trace 2). nValue = 1 nMask = 7340039 20 bytes total send AnimationBuilder [class 100000F] static Msg27iBuildAlphaAnimation 40 Sample Trace 2 viSubject = Visual: 1000185 idAnimation = RH:7000142 (encode RenderPortCallback MsgOfOnBatchProcessed as MessagePtr) 20 bytes total target = RH:0 send Animation:7000142 MsglfAddCallback uBatchCompleted = 64 20 bytes total ch = AnimationCallback:5 20 bytes total send Context [class 1000002] static MsgZfForWardMessage 45 send Animation:7000142 Msgl OiS etRepeatCount idContextDest = ctx:1 cRepeats = —1 msgRetum = [blob MessagePtr] 215548920 16 bytes total +20 bytes blob data send Animation:7000142 Msg6iSetStopCommand 40 bytes total cmd = MoveToEnd 16 bytes total 50 send Animation:7000142 MsgZfAddKeyframe As the message runtime assembles and sends the current idxKeyframe = 0 batch, it may trace information about the batch as a Whole and ?TimeSec = 0 the individual buffer or buffers being used to carry it (e.g., 20 bytes total “MESSAGE BATCH 64” and “SENDING BATCH send AnimationBuilder [class 100000F] static Msg3iSetSCurve ani = Animation:7000142 BUFFER: id:RH:0, 912 bytes”). 55 idxKeyframe = 0 When sending a buffer containing a batch of messages an ?Scale = 1 24 bytes total identi?cation (e.g., an instance id) may alloW multiple buffers send AnimationBuilder [class 100000F] static Msg16iSetFloat to be linked together for large batches. A buffer may include ani = Animation:7000142 bulk binary data (image pixels for a surface, Waveform data idxKeyframe = 0 for a sound, etc). An instance id for a batch or a buffer may ?Value = 0 60 24 bytes total alloW messages to refer to data of the batch or buffer in send Animation:7000142 MsgZfAddKeyframe subsequent rendering commands. In various examples, a idxKeyframe = 1 buffer may include messages and data, Which may optionally ?TimeSec = 1.5 be used by one or more tasks related to a message or a 20 bytes total send AnimationBuilder [class 100000F] static Msg3iSetSCurve subsequent message. 65 ani = Animation:7000142 The Sample “send” Traces 1 and 2 represent actual proto idxKeyframe = 1 col messages being placed in a buffer. A “Sending Buffer” US 7,667,704 B2
-continued -continued
Sample Trace 3 Sample Trace 3
flScale = 1 5 20 bytes total 24 bytes total send Context [class 1000002] static Msg2iForWardMessage send AnimationBuilder [class 100000F] static Msg16iSetFloat idContextDest = ctx:1 ani = Animation:7000142 msgRetum = [blob MessagePtr] 215548920 idxKeyfraIne = 1 +20 bytes blob data ?Value = 0.627451 40 bytes total 24 bytes total 10 MESSAGE BATCH 64 send Animation:7000142 MsgZiAddKeyfraIne [SENDING BATCH BUFFER: id = RH:0, 912 bytes] idxKeyfraIne = 2 ?TimeSec = 3 20 bytes total In the Sample Trace 3, messages only dreW from surfaces Send A1?1mat1?nBuI1lder [Class 1000001:] 5mm Msg3isetsculw that Were previously sent to the renderer. If messages Were yrl?glinzjooomz 15 traced for a transition that required new graphical resources, ?scale : 1 more traf?c Would have been devoted to surface allocation, 24 bytes total loading pixel data from a bulk data buffer and extreme cases, send AnimationBuIilder [class 100000F] static Msg16iSetFloat Surface p001 Compaction gm =Ammanon'7oool4z The batch of Sample Trace 3 stopped the pulsating focus idxKeyfraIne = 2 . . - ?valu? : 0 20 ammatron on the old focused button. When this occurred, the 24 bytes total renderer sent a noti?cation back to the framework that the 56nd A111mat1°n17000142 Msg5iPlay animation Was stopped, triggering some generic handling on 12 bytes total the application side that decided it Was done With that par (neW RenderBuilder [class 1000003], constructor MsgIiCreate ______as M?ssag?ptr) t1cular an1mat1on object. Th1s 1n turn created another, small cat = InScene 25 batch of messages instructing the renderer to destroy the old 16 bytes total I I animation, as presented in Sample Trace 4. send Broker [class 1000000] static MsgIiCreateObJect idObjectClass = RH:1000003 idObjectNeW = RH:5000146 msgConstruction = [blob MessagePtr] 215548588 +16 bytes blob data 30 Sample Trace 4 40 bytes total I I I send Surface:30000EE Msg0_DraWGrid (‘115F956 A111mat19n57000062) rb : R6nderBuild6n5000146 send Broker [class 1000000] static MsgOiDestroyObject ?XlPXl : 20 idObjeet = RHI7000062 ?XZPXl : 231 16 bytes total ?ylpxl : 12 35 (encode RenderPortCallback MsgOiOnBatchProcessed as MessagePtr) ?YZPxl = 41 target = RH:0 rcfDestPxl = {x = 0,Y = 0, Width = -1, Height = -1} uBatchCompl?M = 65 48 bytes total 20 bytes total 56nd Visualz1000176 MSgZZiSHCOHWHt send Context [class 1000002] static Msg2iForWardMessage rbContent = RenderBuilder:5000146 idCOHtFXtDFSt = “X11 16 bytes total msgRetum = [blob MessagePtr] 215515328 send RenderBu1lder.5000146- . MsgOiClear 40 +20 b y tes blob data 12 bytes total 40 bytes total send Surface:10000C6 MsgOiDraWGrid MESSAGE BATCH 65 rb : R6nd6rBuild6rz5000146 [SENDING BATCH BUFFER: id = RHIO, 96 bytes] ?XIPxl = 20
?XZPxl = 231 . . . ?YlPxl : 12 45 As described herein an exemplary protocol burlds on a flYZPxl = 41 messaging model to describe a logical rendering and interac ICiD?StPXl = {X = 0,Y = 0, WIdLh = -1,H@1ght= -1} tion model for high-?delity user interface associated With a 48 bytes total rendering device Which may be located remotely from a host send Visual:1000184 Msg22iSetContent ’ rbContent = RenderBuilder:5000146 Computer' I I 16 bytes total 50 An exemplary protocol may 1nclude var1ous layers such as Send ?lgdfrBglidfr 15000146 Msgoiclear functionality-speci?c modules that implement services, that 86nd surfawloooocgy es 0 a MSgOiDIaWGIId expose programming. lnterface. as objects/messages,. and that rb : RenderBuilder:5000146 use a primary set of objects related to a graphics model. ?X1Pxl= 20 An exemplary graphics model for an exemplary protocol SXZPX] = 231 55 may include a device object, a presentation container object, [13511:]? a surface pool object (e.g., represents coherent storage for rcfD6StPX] : {X : OIY : OI Width : _1I Height : _1} bitmap data), a surface object (e.g., individual draWable 48 bytes total chunk of a surface pool), a visual object (e.g., logical con S?nd “5113111000185 Msgzzis?tcontent tainer for draWing primitives on the screen, oWn coordinate rbContent = RenderBuilder:5000146 - d d - - - 16 bytes total 60 space, organize as a tree roote in a presentatlon container), (dispos? Rend6rBuild6rI5000146) and a render builder object (e.g., a contalner for 11st of draW send Broker [class 1000000] static MsgOfDestroyObject ing primitives being assembled). The exemplary graphics lld6obbl?ct = RIHI5000146 model may include an animation manager (e.g., for coordi ytes tota ...... (encode RenderPortCallback MsgOiOnBatchProcessed as MessagePtr) gates processlng of_ammanons) and an ndlvlqual funcnon target : RHIO 65 lnstance for an1mat1on. Extender graphics objects may be uBatchCompleted = 64 included such as a rasteriZer object (e.g., helper object used for preparing surface data) and a gradient object (e.g., a color US 7,667,704 B2 15 16 channel modi?er object that can modify other primitives equally at both 24 fps and 60 fps, being late by even 1/60 s on Within its scope Where traditional ‘gradient’ functionality is a single frame of 24 fps video Will produce a visible “hiccup” achieved by drawing a solid ?ll inside). Other objects may that users may perceive as a loss of quality. Thus, an exem include a desktop manager object (e.g., for integration With a plary protocol may provide for an increase in quality even at desktop), a form object for desktop-oriented presentation frame rates less than 60 fps. As frame rate increases, an container behavior) and a variety of callback interfaces. exemplary frameWork typically has less room for error, As already mentioned, messages may pertain to task other thereby increasing the need for message control. Again, con than graphics or animation. Further, a batch of messages may trol on the host side can reduce burdens on the rendering include a mix of tasks such as, but not limited to, graphics device side, Which, in itself, may alloW a rendering device to tasks, audio tasks, appliance control tasks, etc., as appropriate improve display quality. or desired. For example, a batch of messages may pertain to An exemplary protocol may be used for communicating control of an appliance (e.g., a refrigerator, thermostat, secu updates, changes, neW user interfaces, neW user interface rity camera, game unit, etc.), a graphic, a sound or other task. components, presentation of video, etc. An exemplary protocol optionally supports extensions. For example, an extension to an exemplary protocol may provide Exemplary Computing Environment features associating With remoting, i.e., instructing a remote The various examples may be implemented in different user interface. Such an extension may provide for full-screen computer environments. The computer environment shoWn or partial screen operations depending on the state of a ren in FIG. 6 is only one example of a computer environment and dering device. For example, communication of a message or is not intended to suggest any limitation as to the scope of use or functionality of the computer and netWork architectures messages may differ for instructing a rendering device’s 20 graphical user interface in a full-screen mode and instructing suitable for use. Neither should the computer environment be a desktop computer’s graphical user interface. If instructions interpreted as having any dependency or requirement relating are being communicated to a desktop computer, then exten to any one or combination of components illustrated in the sions may be loaded on the host computer to provide different example computer environment. functionality or communication of messages. 25 FIG. 6 illustrates an example of a suitable computing sys An exemplary protocol optionally alloWs for communica tem environment 600 on Which various exemplary methods tion of messages from a host device to a rendering device that may be implemented. Various exemplary devices or systems accounts for a user’s media experience With respect to the may include any of the features of the exemplary environment rendering device. For example, Where such a user is consum 600. The computing system environment 600 is only one ing TV media at a frame rate of approximately 60 fps, then 30 example of a suitable computing environment and is not communication may avoid transmission of a batch of mes intended to suggest any limitation as to the scope of use or sages that Would risk interrupting the display of one or more functionality of the invention. Neither should the computing frames. Further, many animations (e. g., fade-in, fade-out, environment 600 be interpreted as having any dependency or enlarge, etc.) require a certain frame rate to ensure that a user requirement relating to any one or combination of compo perceives such animations as intended, typically smoothly. 35 nents illustrated in the exemplary operating environment 600. Where a communication may risk interrupting a frame or Various exemplary methods are operational With numerous delaying display of a frame, then an exemplary frameWork other general purpose or special purpose computing system (e. g., the frameWork 510) may account for such risk and environments or con?gurations. Examples of Well knoWn communicate in smaller batches or With other adjustments. computing systems, environments, and/ or con?gurations that Of course, in the TV example and the animation example, the 40 may be suitable for implementation oruse include, but are not host device sending the message or batch of messages may limited to, personal computers, server computers, hand-held also be providing the animation or the TV media. On such a or laptop devices, multiprocessor systems, microprocessor basis, an exemplary frameWork may act accordingly to pro based systems, set top boxes, programmable consumer elec vide a user With a smooth, expected vieWing experience. tronics, netWork PCs, minicomputers, mainframe computers, Thus, an exemplary frameWork may operate to ensure an 45 distributed computing environments that include any of the optimal frame rate With respect to display of information on a above systems or devices, and the like. For example, the rendering device’s user interface. exemplary system 100 of FIG. 1 may use a remote computer An exemplary frameWork optionally sends one or more to generate information for display of a U1 Wherein the dis messages in a batch Where the timing for communication of played Ul operates in conjunction With a remote control or the batch to a rendering device ensures that changes to the 50 other input device. user interface of the rendering device Will not interrupt a Various exemplary methods, applications, etc., may be frame of an animation or other video display. For example, described in the general context of computer-executable reception of the batch, execution of the batch messages (e. g., instructions, such as program modules, being executed by a execution of commands, transfer of information, etc.) or dis computer. Generally, program modules include routines, pro play may occur betWeen frames of such an animation or other 55 grams, objects, components, data structures, etc., that per video display. Consider an application executing on a host form particular tasks or implement particular abstract data device that requests display of a dialog box on a user interface types. Various exemplary methods may also be practiced in of a rendering device Where the user interface is currently distributed computing environments Where tasks are per displaying video at a frame rate of 60 fps. An exemplary formed by remote processing devices that are linked through frameWork on the host device may control messages in a 60 a communications netWork or other communication (e.g., batch manner, timing manner or other manner to ensure that infrared, etc.). In a distributed computing environment, pro display of the dialog box by the rendering device does not gram modules may be located in both local and remote com interrupt display of the video. puter storage media including memory storage devices. In general, for some rendering devices, a user may expect With reference to FIG. 6, an exemplary system for imple a user interface associated With such a rendering device to 65 menting the various exemplary methods includes a general present uncompromised television video, Which has a frame purpose computing device in the form of a computer 610. rate of 60 HZ. While users can typically discern glitches about Components of computer 610 may include, but are not lim US 7,667,704 B2 17 18 ited to, a processing unit 620, a system memory 630, and a computer readable instructions, data structures, program system bus 621 that couples various system components modules and other data for the computer 610. In FIG. 6, for including the system memory 630 to the processing unit 620. example, hard disk drive 641 is illustrated as storing operating The system bus 621 may be any of several types of bus system 644, application programs 645, other program mod structures including a memory bus or memory controller, a ules 646, and program data 647. Note that these components peripheral bus, and a local bus using any of a variety of bus can either be the same as or different from operating system architectures. By Way of example, and not limitation, such 634, application programs 635, other program modules 636, architectures include Industry Standard Architecture (ISA) and program data 637. Operating system 644, application bus, Micro Channel Architecture (MCA) bus, Enhanced ISA programs 645, other program modules 646, and program data (EISA) bus, Video Electronics Standards Association 647 are given different numbers here to illustrate that, at a (VESA) local bus, and Peripheral Component Interconnect minimum, they are different copies. A user may enter com (PCI) bus also knoWn as MeZZanine bus. mands and information into the computer 610 through input Computer 610 typically includes a variety of computer devices such as a keyboard 662 and pointing device 661, readable media. Computer readable media can be any avail commonly referred to as a mouse, trackball or touch pad. able media that can be accessed by computer 61 0 and includes Other input devices (not shoWn) may include a microphone, both volatile and nonvolatile media, removable and non-re joystick, game pad, satellite dish, scanner, or the like. These movable media. By Way of example, and not limitation, com and other input devices are often connected to the processing puter readable media may comprise computer storage media unit 620 through a user input interface 660 that is coupled to and communication media. Computer storage media includes the system bus 621, but may be connected by other interface both volatile and nonvolatile, removable and non-removable 20 and bus structures, such as a parallel port, game port or a media implemented in any method or technology for storage universal serial bus (U SB). A monitor 691 or other type of of information such as computer readable instructions, data display device is also connected to the system bus 621 via an structures, program modules or other data. Computer storage interface, such as a video interface 690. In addition to the media includes, but is not limited to, RAM, ROM, EEPROM, monitor 691, computers may also include other peripheral ?ash memory or other memory technology, CD-ROM, digital 25 output devices such as speakers and printer, Which may be versatile disks (DVD) or other optical disk storage, magnetic connected through a output peripheral interface 695. cassettes, magnetic tape, magnetic disk storage or other mag The computer 610 may operate in a netWorked environ netic storage devices, or any other medium Which can be used ment using logical connections to one or more remote com to store the desired information and Which can accessed by puters, such as a remote computer 680. The remote computer computer 610. Combinations of the any of the above should 30 680 may be a personal computer, a server, a router, a netWork also be included Within the scope of computer readable PC, a peer device or other common netWork node, and typi media. cally includes many or all of the features described above The system memory 630 includes computer storage media relative to the computer 610. The logical connections in the form of volatile and/ or nonvolatile memory such as read depicted in FIG. 6 include a local area netWork (LAN) 671 only memory (ROM) 631 and random access memory 35 and a Wide area netWork (WAN) 673, but may also include (RAM) 632. A basic input/output system 633 (BIOS), con other netWorks. Such netWorking environments are common taining the basic routines that help to transfer information place in o?ices, enterprise-Wide computer netWorks, intra betWeen elements Within computer 610, such as during start nets and the Internet. up, is typically stored in ROM 631. RAM 632 typically con When used in a LAN netWorking environment, the com tains data and/ or program modules that are immediately 40 puter 610 is connected to the LAN 671 through a netWork accessible to and/ or presently being operated on by process interface or adapter 670. When used in a WAN netWorking ing unit 620. By Way of example, and not limitation, FIG. 6 environment, the computer 610 typically includes a modem illustrates operating system 634, application programs 635, 672 or other means for establishing communications over the other program modules 636, and program data 637. WAN 673, such as the Internet. The modem 672, Which may The computer 610 may also include other removable/non 45 be internal or external, may be connected to the system bus removable, volatile/nonvolatile computer storage media. By 621 via the user input interface 660, or other appropriate Way of example only, FIG. 6 illustrates a hard disk drive 641 mechanism. In a netWorked environment, program modules that reads from or Writes to non-removable, nonvolatile mag depicted relative to the computer 610, or portions thereof, netic media, a magnetic disk drive 651 that reads from or may be stored in a remote memory storage device. By Way of Writes to a removable, nonvolatile magnetic disk 652, and an 50 example, and not limitation, FIG. 6 illustrates remote appli optical disk drive 655 that reads from or Writes to a remov cation programs 685 as residing on the remote computer 680 able, nonvolatile optical disk 656 such as a CD ROM or other (e.g., in memory of the remote computer 680). It Will be optical media (e.g., DVD, etc.). Other removable/non-remov appreciated that the netWork connections shoWn are exem able, volatile/nonvolatile computer storage media that can be plary and other means of establishing a communications link used in the exemplary operating environment include, but are 55 betWeen the computers may be used. not limited to, magnetic tape cassettes, ?ash memory cards, Although various exemplary methods, devices, systems, digital versatile disks, digital video tape, solid state RAM, etc., have been described in language speci?c to structural solid state ROM, and the like. The hard disk drive 641 is features and/or methodological acts, it is to be understoodthat typically connected to the system bus 621 through a data the subject matter de?ned in the appended claims is not nec media interface such as interface 640, and magnetic disk drive 60 essarily limited to the speci?c features or acts described. 651 and optical disk drive 655 are typically connected to the Rather, the speci?c features and acts are disclosed as exem system bus 621 a data media interface that is optionally a plary forms of implementing the claimed subject matter. removable memory interface. For purposes of explanation of the particular example, the magnetic disk drive 651 and the The invention claimed is: optical disk drive use the data media interface 640. 65 1. A method of communicating betWeen a host device and The drives and their associated computer storage media a rendering device for controlling a user interface of the discussed above and illustrated in FIG. 6, provide storage of rendering device, the method comprising: US 7,667,704 B2 19 20 generating control messages by a user interface framework 11. The method of claim 1 Wherein the messages comprise executing on the host device, Wherein the control mes one or more graphics or animation commands. sages comprise at least one animation command; 12. The method of claim 1 Wherein the messages comprise reducing a number of control messages based on a current one or more commands associated With a softWare interface state of the rendering device to avoid interfering With a for graphics or animations. user experience With the rendering device; 13. The method of claim 1 Wherein one or more of the encoding, by the user interface framework, the control messages rely on at least one member selected from a group messages in a buffer as blob data; and consisting of a render builder object to hold a packet of communicating the blob data as one control message to the draWing commands, a WindoW object that represents a pre rendering device, thereby sending control messages as a sentation container, a visual object that represents a discrete batch of control messages, Wherein the rendering device coordinate space, a device callback object for reporting on the displays frames of images at a frame rate and renders status of the rendering device, a rendering device object that according to the control messages of the blob data With represents a rendering technology driver, a surface object that out disrupting the display of frames at the frame rate. represents a logical container for image bits, a surface pool 2. The method of claim 1 Wherein the generating comprises object that represents a physical container for image bits, a receiving one or more commands issued from a remote con gradient object for modi?cation of color space in a container trol to the rendering device and, in response to the one or more scope, an animation manager object that serves for animation commands, generating the control messages by an applica control, an animation callback object that reports on an ani tion executing on the host device. mation, and an animation object that describes a logical ani 3. The method of claim 1 Wherein communication betWeen 20 mation sequence. the ho st device and the rendering device occurs asynchro 14. The method of claim 1 comprising a plurality of buff nously. ers. 4. The method of claim 1 Wherein communication occurs 15. A computer storage medium having computer-execut according to a communication protocol that comprises a able instructions for performing a method on a host device, graphics model for rendering graphics and animations. 25 the method comprising: 5. The method of claim 1 Wherein the buffer comprises a in response to the one or more commands, generating netWork buffer, and Wherein the method further comprises: messages by an application executing on the host device after communicating the blob data as one control message and reducing a number of messages based on a current to the rendering device, obtaining, by the host device, state of a rendering device, Wherein the application information about a characteristic of the communication 30 relies on a user interface frameWork executing on the betWeen the host device and the rendering device; and host device to generate the messages and to encode the changing a number of control messages to include in the messages in a buffer, and further Wherein the messages blob data in a subsequent communication of blob data to comprise: the rendering device. one or more messages originating from the application 6. The method of claim 1 further comprising communicat 35 executing on the host computer in communication ing data from the rendering device to the ho st device by With a remote device executing a renderer; encoding a callback control message as part of the blob data, one or more animation commands; Wherein the callback control mes sage instructs the rendering one or more graphics commands; device to send said data to the host device. one or more commands associated With a softWare inter 7. The method of claim 1 further comprising encoding 40 face for graphics or animations; additional information in the buffer unrelated to the messages a last message Which comprises a call instructing the and optionally related to one or more subsequent messages. rendering device to send a copy of the last message 8. The method of claim 1 further comprising, after the back to the host device; communicating, generating additional messages, encoding encoding the messages in the buffer as blob data, Wherein the additional messages in the buffer as blob data, and com 45 the buffer comprises a netWork buffer; municating the blob data to the rendering device prior to communicating the blob data to the rendering device receipt of a message by the host device sent from the render according to a communication protocol that comprises a ing device. graphics model for rendering graphics and animations, 9. The method of claim 1 Wherein the communication Wherein the rendering device displays frames of images occurs according to a communication protocol that comprises 50 at a frame rate and renders according to the blob data at least one member selected from a group consisting of a Without disrupting display of frames at the frame rate; broker object as a root object in the protocol, a context object after the communicating, generating additional messages that represents a logical computation site, a render port call by the application executing on the host device, encod ing the additional messages in the buffer as blob data, back object for callback from a render port in runtime, a data buffer callback object for reporting on the status of bulk data 55 and communicating the blob data to the rendering device prior to receipt of the last message by the host device transfers, and a data buffer object that represents bulk data on sent from the rendering device. a remote node. 16. The method of claim 15, Wherein the encoding of the 10. The method of claim 1 Wherein a last message com messages in the netWork buffer comprises a batch process. prises a call instructing the rendering device to send a mes sage to the host device. * * * * *