declarative techniques in Distributed Media Center Jari Kleimola and Petri Vuorimaa Telecommunications Software and Multimedia Lab Helsinki University of Technology June 5, 2007 Outline DMC Compound Model Remote UIs Interaction Loop XProc Conclusion W3C Workshop on Declarative Models of Distributed Web Applications June 5-6 2007, Dublin Application Scenario Home UI devices Another Home UI devices UI devices DMC DMC Target devices Target devices Web content Target devices Problem: Unified control and monitoring of networked Targets UI devices • media devices + apps • PCs + laptops • media services + content • phones, PDAs • home automation • TVs, legacy remotes Possible solution: DMC = Distributed Media Center • remote control hub • runs in a PC / STB / gateway 5.6.2007 [email protected] DMC Architecture Browser Midlet thin client application user actions Browser HID UIs TV renderer decoupled IO Midlet IR Remote application Midi device LAN / internet LAN / internet LAN / internet markup RFB custom input - XHTML - XForms - JPG fragm. - eg. bytecode - HTTP REST / XPointer - CSS, SVG - MIDPML - markup (REX) - REX - JavaScript - OSC, MIDI, IR Concrete UI Localization, Personalization, Branding (XML), optional DMC LT S Abstract UI (XML) X Models (XML + DOM, state variables) UPnP oBIX Zeroconf MIDP Web other generics adapter adapter /Bonjour adapter adapter adapters adapter CE devices home control other DMCs cell phones shares media apps XML files media apps automation Macs stores games XMLDB Targets home control VoD presentation automation live streams home studio 5.6.2007 videoVOIP [email protected] DMC Architecture Browser Midlet thin client application user actions Browser HID UIs TV renderer decoupled IO IR Remote Midi device LAN / internet LAN / internet LAN / internet markup RFB custom input - XHTML - XForms - JPG fragm. - eg. bytecode - HTTP REST / XPointer - CSS, SVG - MIDPML - markup (REX) - REX - JavaScript Concrete UI Localization, Personalization, Branding (XML) DMC LT S Abstract UI (XML) X Models (XML + DOM, state variables) Adapters generics CE devices home control other DMCs cell phones shares media apps XML files media apps automation Macs stores games XMLDB Targets home control VoD presentation automation live streams home studio 5.6.2007 videoVOIP [email protected] Compound Model Distributed into • Dynamic infrastructure DOM (zones, devices, services, people, content) • updated by adapters and components • only topmost levels • Virtual models that extend infrastructure DOM • handled by adapters • state variables • Generic XML files hosted by DMC • eg. for RAD + testing Addressing • Dispatcher parses URI to model and resource reference (http://host:port/model#resource) • eg. /dmc/zones#bedroom/TV/channel infra adapter Mutation and Access • REST and/or REX • ui: GET + PUT, adapter: POST + DELETE + PUT • POST /dmc/devices body: <device name=”TV” id=”udn”>...</device> • PUT /dmc/zones#bedroom/tv/channel=5 5.6.2007 [email protected] Remote UIs UI Fragments • extend XHTML controls (currently scripted) • eg. bound list, slider, popup • patterns, microformats / RDF description ? • can be updated without loading entire page • XHR • binding to other controls • bound into micromodel using XPath • eg. bind=”/dmps/device[@id=$current]” micromodel (uDOM compliant) • state, maybe some frequently used items as well • mutated using REX Other things • On-the-fly interaction style change • eg. forms -> direct manipulation • IO decoupling • attach display from mobile phone to TV • continue using mobile keys as input device 5.6.2007 [email protected] Interaction Loop Browser micromodel ConcreteUI XML + DOM REST + XPointer Binding logic Adaptation XSLT + XProc Dispatcher Declarative • XHTML, SVG RemotePort (xhr, ajax, comet) • XSLT + XProc • XPath REX REST XPointer UI markup (mutation) REX (input) • REX Hub • XML fragments RemotePort (server, servlet, sockets) Procedural Dispatcher Adaptation • Java XSLT + XProc • ECMAScript XML + DOM REX AbstractUI Distributed Model REST XPointer, XML XSLT + XProc Adapters devices, services XML files + DBs content, communication 5.6.2007 [email protected] Sample Loop idGroup idDevice <ul id=”idGroup”> <li href=”/zones/home/livingroom” onclick=”httpGET(this)” xhrTarget=”idDevice”>Living Room</li> <li href=”/zones/home/library” onclick=”httpGET(this)” xhrTarget=“idDevice”>Library</li> ... </ul> 5.6.2007 [email protected] Sample Loop Browser 1. Addressing micromodel ConcreteUI - host / model # resource XML + DOM - GET http://dmc/zones#home/livingroom/ Binding logic Dispatcher - loop all models Dispatcher - registered for ”/zones” ? - yes: redirect ”/home/livingroom/” RemotePort (xhr, ajax, comet) REX REST XPointer UI markup 1 (mutation) REX (input) Hub RemotePort (server, servlet, sockets) Adaptation Dispatcher XSLT + XProc XML + DOM REX AbstractUI Distributed Model REST XPointer, XML XSLT + XProc Adapters devices, services XML files + DBs content, communication 5.6.2007 [email protected] Sample Loop Browser 1. Addressing micromodel ConcreteUI - host / model # resource XML + DOM - GET http://dmc/zones#home/livingroom/ Binding logic 2. Fetch XML fragment Dispatcher - from DOM or Adapter or file - select = ”/home/livingroom/” RemotePort (xhr, ajax, comet) <devices> <dev name=”TV” ref=”…”/> REX REST XPointer <dev name=”mediaServer” ref=“…”/> UI markup (mutation) REX (input) <dev name=”lights” ref=”…”/> Hub </devices> RemotePort (server, servlet, sockets) Adaptation Dispatcher XSLT + XProc XML + DOM 2 REX AbstractUI Distributed Model REST XPointer, XML XSLT + XProc Adapters devices, services XML files + DBs content, communication 5.6.2007 [email protected] Sample Loop Browser 1. Addressing micromodel ConcreteUI - host / model # resource XML + DOM - GET http://dmc/zones#home/livingroom/ Binding logic 2. Fetch XML fragment Dispatcher - from DOM or Adapter or file - select = ”/home/livingroom/” RemotePort (xhr, ajax, comet) 3. Transform into abstract UI REX REST XPointer - browserAbstr.xsl (or /session[@id]/abstract) UI markup (mutation) REX (input) - template match= ”devices” Hub <select1> RemotePort (server, servlet, sockets) <item label=”TV” ref=”…” … /> <item label=”mediaServer” ref=“…” … /> <item label=”lights” ref=”…” … /> Adaptation Dispatcher </select1> XSLT + XProc XML + DOM REX AbstractUI Distributed Model REST XPointer, XML XSLT + XProc 3 Adapters devices, services XML files + DBs content, communication 5.6.2007 [email protected] Sample Loop Browser 1. Addressing micromodel ConcreteUI - host / model # resource XML + DOM - GET http://dmc/zones#home/livingroom/ Binding logic 2. Fetch XML fragment Dispatcher - from DOM or Adapter or file - select = ”/home/livingroom/” RemotePort (xhr, ajax, comet) 3. Transform into abstract UI REX REST XPointer - browserAbstr.xsl (or /session[@id]/abstract) UI markup (mutation) REX (input) - <xsl:template match=”devices”/> Hub 4. Transform into concrete UI RemotePort (server, servlet, sockets) 4 - browserConcr.xsl (or /session[@id]/concrete) - <xsl:template match=”select1”/> Adaptation Dispatcher <ul> XSLT + XProc <li href=”…” … >TV</li> XML + DOM <li href=”…” … >Media Server</li> <li href=”…” … >Lights</li> REX AbstractUI Distributed Model </ul> REST XPointer, XML XSLT + XProc Adapters devices, services XML files + DBs content, communication 5.6.2007 [email protected] Result idGroup idDevice idControls <rex> <event target=”idDevice” name=”DOMNodeRemoved”> <ul id=”idDevice”> <li href=”…” onclick=”httpGET(this)” xhrTarget=”idControls”>TV</li> <li href=”…” onclick=”httpGET(this)” xhrTarget=“idControls”>Media Server A</li> ... </ul> </event> </rex> 5.6.2007 [email protected] XProc infra request stylesheets 1. Addressing - host / model # resource - GET http://dmc/zones#home/livingroom/ xinclude 2. Fetch XML fragment select = - from DOM or Adapter or file /model/fragment - select = ”/home/livingroom/” 3. Transform into abstract UI choose - browserAbstr.xsl (or /session[@id]/abstract) count(*) - template match= ”devices” >0 =0 4. Transform into concrete UI identity http-request - browserConcr.xsl (or /session[@id]/concrete) - template match= ”choice” xinclude • 58 lines of markup ! xslt • procedural: - server pipeline - uri + header parser 5.6.2007 [email protected] XProc W3C working draft (April 5 2007) • pipelines of (compound) steps . input and output ports . xinclude, xslt, validate, http-request . load, store, serialize, parse, identity, join, subsequence . xslt2, xquery, formatter • micro-operations . delete, insert, rename, replace, set, (un)wrap, label • conditionals, loops, exceptions • parameters, options • pipeline libraries for example • get UI components from web (eg. icons from Tango) • mashups: get metadata from IMDB or AMG 5.6.2007 [email protected] Techniques Summary XProc declarative glue XSLT to create ui pages / ui fragments / rex mutation, adapters also REX DOM mutation, user input, model state change notifications REST sync ui – model – target, access external devices/services XPointer XPath endian URIs to address resources XPath used in many places 5.6.2007 [email protected] Conclusion Observations • XProc seems to fit well into DMC scenario • usable techniques are available, but might be difficult to find • REST scales well, cleaner code than RPC, modeless, passes firewalls Some concerns • tools for design, debug etc. • security issues • performance issues 5.6.2007 [email protected] DMC Examples Targets Concretized UIs 5.6.2007 [email protected] Thank You ! Questions ? 5.6.2007 [email protected].