declarative techniques in Distributed Media Center

Jari Kleimola and Petri Vuorimaa

Telecommunications Software and Multimedia Lab Helsinki University of Technology

June 5, 2007

Outline DMC Compound Model Remote UIs Interaction Loop XProc Conclusion

W3C Workshop on Declarative Models of Distributed Web Applications June 5-6 2007, Dublin Application Scenario

Home UI devices Another Home

UI devices UI devices

DMC DMC

Target devices Target devices

Web content Target devices

Problem: Unified control and monitoring of networked  Targets  UI devices • media devices + apps • PCs + laptops • media services + content • phones, PDAs • home automation • TVs, legacy remotes

Possible solution:  DMC = Distributed Media Center • remote control hub • runs in a PC / STB / gateway 5.6.2007 [email protected] DMC Architecture

Browser Midlet thin client application user actions Browser HID UIs TV renderer decoupled IO Midlet IR Remote application Midi device

LAN / LAN / internet LAN / internet

markup RFB custom input - XHTML - XForms - JPG fragm. - eg. bytecode - HTTP REST / XPointer - CSS, SVG - MIDPML - markup (REX) - REX - JavaScript - OSC, MIDI, IR

Concrete UI

Localization, Personalization, Branding (XML), optional

DMC LT S

Abstract UI (XML) X

Models (XML + DOM, state variables)

UPnP oBIX Zeroconf MIDP Web other generics adapter adapter /Bonjour adapter adapter adapters adapter

CE devices home control other DMCs cell phones shares media apps XML files media apps automation Macs stores games XMLDB Targets home control VoD presentation automation live streams home studio 5.6.2007 videoVOIP [email protected] DMC Architecture

Browser Midlet thin client application user actions Browser HID UIs TV renderer decoupled IO IR Remote Midi device

LAN / internet LAN / internet LAN / internet

markup RFB custom input - XHTML - XForms - JPG fragm. - eg. bytecode - HTTP REST / XPointer - CSS, SVG - MIDPML - markup (REX) - REX - JavaScript

Concrete UI

Localization, Personalization, Branding (XML)

DMC LT S

Abstract UI (XML) X

Models (XML + DOM, state variables)

Adapters generics

CE devices home control other DMCs cell phones shares media apps XML files media apps automation Macs stores games XMLDB Targets home control VoD presentation automation live streams home studio 5.6.2007 videoVOIP [email protected] Compound Model

Distributed into • Dynamic infrastructure DOM (zones, devices, services, people, content) • updated by adapters and components • only topmost levels

• Virtual models that extend infrastructure DOM • handled by adapters • state variables

• Generic XML files hosted by DMC • eg. for RAD + testing

Addressing • Dispatcher parses URI to model and resource reference (http://host:port/model#resource) • eg. /dmc/zones#bedroom/TV/channel infra adapter

Mutation and Access • REST and/or REX • ui: GET + PUT, adapter: POST + DELETE + PUT • POST /dmc/devices body: ... • PUT /dmc/zones#bedroom/tv/channel=5 5.6.2007 [email protected] Remote UIs

UI Fragments • extend XHTML controls (currently scripted) • eg. bound list, slider, popup • patterns, microformats / RDF description ? • can be updated without loading entire page • XHR • binding to other controls

• bound into micromodel using XPath • eg. bind=”/dmps/device[@id=$current]” micromodel (uDOM compliant) • state, maybe some frequently used items as well • mutated using REX

Other things • On-the-fly interaction style change • eg. forms -> direct manipulation • IO decoupling • attach display from mobile phone to TV • continue using mobile keys as input device

5.6.2007 [email protected] Interaction Loop

Browser

micromodel ConcreteUI XML + DOM REST + XPointer Binding logic Adaptation XSLT + XProc Dispatcher Declarative • XHTML, SVG RemotePort (xhr, ajax, comet) • XSLT + XProc • XPath REX REST XPointer UI markup (mutation) REX (input) • REX Hub • XML fragments

RemotePort (server, servlet, sockets) Procedural Adaptation Dispatcher • Java XSLT + XProc • ECMAScript XML + DOM REX AbstractUI Distributed Model

REST XPointer, XML XSLT + XProc Adapters devices, services XML files + DBs content, communication 5.6.2007 [email protected] Sample Loop

idGroup idDevice

  • Living Room
  • Library
    • ...

5.6.2007 [email protected] Sample Loop

Browser 1. Addressing micromodel ConcreteUI - host / model # resource XML + DOM - GET http://dmc/zones#home/livingroom/ Binding logic Dispatcher - loop all models Dispatcher - registered for ”/zones” ? - yes: redirect ”/home/livingroom/”

RemotePort (xhr, ajax, comet)

REX REST XPointer UI markup 1 (mutation) REX (input) Hub

RemotePort (server, servlet, sockets)

Adaptation Dispatcher XSLT + XProc XML + DOM REX AbstractUI Distributed Model

REST XPointer, XML XSLT + XProc Adapters devices, services XML files + DBs content, communication 5.6.2007 [email protected] Sample Loop

Browser 1. Addressing micromodel ConcreteUI - host / model # resource XML + DOM - GET http://dmc/zones#home/livingroom/ Binding logic 2. Fetch XML fragment Dispatcher - from DOM or Adapter or file - select = ”/home/livingroom/” RemotePort (xhr, ajax, comet) REX REST XPointer UI markup (mutation) REX (input) Hub

RemotePort (server, servlet, sockets)

Adaptation Dispatcher XSLT + XProc XML + DOM 2 REX AbstractUI Distributed Model

REST XPointer, XML XSLT + XProc Adapters devices, services XML files + DBs content, communication 5.6.2007 [email protected] Sample Loop

Browser 1. Addressing micromodel ConcreteUI - host / model # resource XML + DOM - GET http://dmc/zones#home/livingroom/ Binding logic 2. Fetch XML fragment Dispatcher - from DOM or Adapter or file - select = ”/home/livingroom/” RemotePort (xhr, ajax, comet) 3. Transform into abstract UI

REX REST XPointer - browserAbstr. (or /session[@id]/abstract) UI markup (mutation) REX (input) - template match= ”devices” Hub RemotePort (server, servlet, sockets) Adaptation Dispatcher XSLT + XProc XML + DOM REX AbstractUI Distributed Model

REST XPointer, XML XSLT + XProc 3 Adapters devices, services XML files + DBs content, communication 5.6.2007 [email protected] Sample Loop

Browser 1. Addressing micromodel ConcreteUI - host / model # resource XML + DOM - GET http://dmc/zones#home/livingroom/ Binding logic 2. Fetch XML fragment Dispatcher - from DOM or Adapter or file - select = ”/home/livingroom/” RemotePort (xhr, ajax, comet) 3. Transform into abstract UI

REX REST XPointer - browserAbstr.xsl (or /session[@id]/abstract) UI markup (mutation) REX (input) - Hub 4. Transform into concrete UI RemotePort (server, servlet, sockets) 4 - browserConcr.xsl (or /session[@id]/concrete) - Adaptation Dispatcher

    XSLT + XProc
  • TV
    • XML + DOM
  • Media Server
  • Lights
    • REX AbstractUI Distributed Model

REST XPointer, XML XSLT + XProc Adapters devices, services XML files + DBs content, communication 5.6.2007 [email protected] Result

idGroup idDevice idControls

  • TV
  • Media Server A
    • ...

5.6.2007 [email protected] XProc

infra request stylesheets 1. Addressing - host / model # resource - GET http://dmc/zones#home/livingroom/

2. Fetch XML fragment select = - from DOM or Adapter or file /model/fragment - select = ”/home/livingroom/”

3. Transform into abstract UI choose - browserAbstr.xsl (or /session[@id]/abstract) count(*) - template match= ”devices” >0 =0 4. Transform into concrete UI identity http-request - browserConcr.xsl (or /session[@id]/concrete) - template match= ”choice” xinclude

• 58 lines of markup ! • procedural: - server pipeline - uri + header parser 5.6.2007 [email protected] XProc

W3C working draft (April 5 2007) • pipelines of (compound) steps . input and output ports . xinclude, xslt, validate, http-request . load, store, serialize, parse, identity, join, subsequence . xslt2, , formatter • micro-operations . delete, insert, rename, replace, set, (un)wrap, label • conditionals, loops, exceptions • parameters, options • pipeline libraries for example • get UI components from web (eg. icons from Tango) • mashups: get metadata from IMDB or AMG

5.6.2007 [email protected] Techniques Summary

XProc declarative glue XSLT to create ui pages / ui fragments / rex mutation, adapters also REX DOM mutation, user input, model state change notifications REST sync ui – model – target, access external devices/services XPointer XPath endian URIs to address resources XPath used in many places

5.6.2007 [email protected] Conclusion

Observations • XProc seems to fit well into DMC scenario • usable techniques are available, but might be difficult to find • REST scales well, cleaner code than RPC, modeless, passes firewalls

Some concerns • tools for design, debug etc. • security issues • performance issues

5.6.2007 [email protected] DMC Examples

Targets Concretized UIs

5.6.2007 [email protected] Thank You !

Questions ?

5.6.2007 [email protected]