Deutsche Telekom Laboratories The W3C Multimodal Architecture ETSI Workshop: Multimodal Interaction on Mobile Devices, Ingmar Kliche (Nov 19, 2008) Nov 19, 2008 1 W3C Multimodal Architecture. Agenda. The W3C W3C Multimodal Interaction Working Group W3C Multimodal Architecture (overview) Upcoming web standards SCXML (State Chart XML) VoiceXML 3.0 EMMA (Extensible Multimodal Annotation Markup Language) Context awareness in multimodal applications A practical implementation of the W3C Multimodal Architecture Conclusion Deutsche Telekom Laboratories Nov 19, 2008 2 The W3C. W3C Multimodal Architecture. The W3C – Mission and Goals. Mission: To lead the World Wide Web to its full potential by developing protocols and guidelines that ensure long-term growth for the Web. The main goal: To ensure interoperabilit yyy of web standards , irrespective of special interests. Deutsche Telekom Laboratories Nov 19, 2008 4 W3C Multimodal Architecture. The W3C – Governance and Statistics. Web standards creation Since 1994 some 110 Web standards have been published. Infrastructure W3C Hosts and offices map International representation (ERCIM*, MIT**, Keio Uni.***) with 400 member organizations from 40 countries Accountability Vendor-neutral forum Public discussion with full consideration of public feedback Participation in working groups , as member or invited expert Patent policy Strong patent policy to protect company and public interests * European Research Consortium for Informatics and Mathematics (ERCIM) ** MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) *** Keio University Deutsche Telekom Laboratories Nov 19, 2008 5 W3C MMI Working Group. W3C Multimodal Architecture. Upcoming web standards. W3C Multimodal Architecture. W3C Multimodal Interaction Working Group. Mission: “… to extend the Web to allow users to dynamically select the most appropriate mode of interaction for their current needs ...“ Participants 40 participants from 22 organizations More information: http://www.w3.org/2002/mmi Deutsche Telekom Laboratories Nov 19, 2008 7 W3C Multimodal Architecture. Multimodal Interaction Framework. W3C Multimodal Interaction Framework (published 2003) Deutsche Telekom Laboratories Nov 19, 2008 8 W3C Multimodal Architecture. “Multimodal Architecture and Interfaces”. “Multimodal Architecture and Interfaces “ (working draft status) Runtime Framework provides the basic Delivery Interaction Data Context Manager Component infrastructure and controls Component communication among the Runtime Framework constituents. Modality Component API Interaction Manager (IM) coordinates modality components (MCs) by life- Modality Modality cycle events and contains the shared Component 1 Component N data (context). Event-based communication between IM and MCs. Deutsche Telekom Laboratories Nov 19, 2008 9 W3C Multimodal Architecture. “Multimodal Architecture and Interfaces”. “Multimodal Architecture and Interfaces “ (working draft status) Delivery MMI lifecycle events Context Interaction Data Component Manager Component newContextRequest/Response startRequest/Response Runtime Framework cancelRequest/Response Modality Component API prepareRequest/Response Modality Modality statusRequest/Response Component 1 Component N pauseRequest/Response resumeRequest/Response Done extension Deutsche Telekom Laboratories Nov 19, 2008 10 W3C Multimodal Architecture. State Chart XML (SCXML). “State Chart XML (SCXML): State Machine Notation for Control Abstraction ” (working draft status) <scxml initial= "foo" ...> <datamodel> <data id= "x" expr= “’1’"/ > </datamodel> <state id= "foo"> <transition event= "foo" target= "bar"> <send event= "play " target= "VUI " targettype= "basichttp "/> </transition> </state> <state id= "bar"> ... </state> </scxml> Deutsche Telekom Laboratories Nov 19, 2008 11 W3C Multimodal Architecture. Voice Modality Component (VoiceXML). W3C VoiceBrowser Working Group Existing standards: VoiceXML 2.1 CCXML 1.0 <vxml version=“3.0” xmlns=“...” ...> SRGS 1.0 and SISR 1.0 <form id= "foo"> <field name= "foo "> PLS 1.0 and SSML 1.0 <grammar .../> <prompt .../> </field> Work in progress: <filled> <send event=.../> SCXML 1.0 </filled> VoiceXML 3.0 </form> <form id= "bar"> External eventing ... New features (e.g. speaker verification) </form> </vxml> Alignment with MMI architecture * Note: VoiceXML 3 syntax not yet finalized, example shows principle. Deutsche Telekom Laboratories Nov 19, 2008 12 W3C Multimodal Architecture. EMMA. “EMMA: Extensible MultiModal Annotation markup language” EMMA 1.0 ( www.w3.org/TR/emma ) currently in Candidate Rec. status Goal: Annotation/representation of user input Example: user utterance “flights from Boston to Denver“ <emma:emma version="1.0“ xmlns:emma=“ http://www.w3.org/2003/04/emma ” …> <emma:one-of id="r1" emma:start="1087995961542" emma:end="1087995963542“ emma:medium="acoustic" emma:mode="voice"> <emma:interpretation id="int1" emma:confidence="0.75“ emma:tokens="flights from boston to denver"> <origin>Boston</origin><origin> Boston </origin> <destination>Denver</destination><destination> Denver </destination> </emma:interpretation> <emma:interpretation id="int2" emma:confidence="0.68“ emma:tokens="flights from austin to denver"> <origin> Austin </origin> <destination> Denver </destination> </emma:interpretation> </emma:one-of> </emma:emma> Deutsche Telekom Laboratories Nov 19, 2008 13 W3C Multimodal Architecture. The Markup View. Implementation of the W3C Multimodal Architecture using W3C markup languages. Delivery Interaction Data Context Manager Component Component SCXML Runtime Framework EMMA Modality Component API Modality Component API to be use within MMI lifecycle events to represent HTMLModality VoiceXMLModality user input Componentfor GUI 1 Componentfor VUI N Deutsche Telekom Laboratories Nov 19, 2008 14 Context Awareness in multimodal applications. W3C Multimodal Architecture. Related W3C Standards. The W3C has developed other standards which are interesting in the context of multimodal applications. e.g.: DCCI (Delivery Context Client Interfaces) http://www.w3.org/TR/DPF Accessing Static and Dynamic Delivery Context Properties Device Description Repository http://www.w3.org/TR/DDR-Simple-API Simple API to access device information Deutsche Telekom Laboratories Nov 19, 2008 16 W3C Multimodal Architecture. Delivery Context Information. Delivery Context Client Interfaces (DCCI): Client interface to access (static and dynamic) device information, e.g. loudspeaker volume Delivery Interaction Data Context Manager Component Component SCXML Data model Runtime Framework Modality Component API Modality Component API HTMLModality VoiceXMLModality Componentfor GUI 1 Componentfor VUI N DCCI Deutsche Telekom Laboratories Nov 19, 2008 17 Related activities with regards to Multimodal Interaction. W3C Multimodal Architecture. Upcoming Activities. Emotion Incubator Group: Annotation of Emotions ( http://www.w3.org/2005/incubator/emotion ) Goal: Standardization of Emotion Markup Language (EmotionML) Model based User Interfaces Incubator Group: Evaluation of research on model-based user interface design (http://www.w3.org/2005/incubator/model-based-ui ) Source of further information: UsiXML ( http://www.usixml.org ) Deutsche Telekom Laboratories Nov 19, 2008 19 W3C Multimodal Architecture. Upcoming Activities. Geolocation Working Group: Provisioning of location information within web browser environments (http://www.w3.org/2008/geolocation ) Rich Web Application Backplane Incubator Group: Rich application backplane (http://www.w3.org/2005/incubator/app-backplane ) Definition of a set of common building blocks for web applications Deutsche Telekom Laboratories Nov 19, 2008 20 A practical implementation of the W3C Multimodal Architecture. W3C Multimodal Architecture. Proof of Concept Implementation. Distributed implementation using SCXML, HTML and CCXML/VXML Delivery Interaction Data Context Manager Component Component SCXML Runtime Framework Modality Component API Modality Component API HTMLModality VoiceXMLModality Componentfor GUI 1 Componentfor VUI N Deutsche Telekom Laboratories Nov 19, 2008 22 W3C Multimodal Architecture. Proof of Concept Implementation. Apache commons SCXML interpreter + ECMAScript interpreter. VoiceXML/CCMXML platform and HTML browser with AJAX support. Apache commons SCXML Server (Apache Tomcat) HTTP I/O Processor Modality Component API: HTTP + XML (using AJAX) Modality Component API: HTTP + XML (EMMA) CCXML/VoiceXML Platform Server HTML Browser Client (with AJAX support) Telephony interface (SIP) SIP Phone Client GUI modality component Voice modality component Deutsche Telekom Laboratories Nov 19, 2008 23 Conclusion. W3C Multimodal Architecture. Conclusion. Multimodal architecture specification (work in progress) Loosely coupled architecture Allows for co-resident and distributed implementations Leverages existing W3C standards (HTML, SVG, …) EMMA 1.0 specification (annotation of user input) completed SCXML will be a candidate for Interaction Manager implementations Deutsche Telekom Laboratories Nov 19, 2008 25 Thank you for your attention. Ingmar Kliche Deutsche Telekom Laboratories 10589 Berlin Goslarer Ufer 35 Tel.: +49 30 3497 2348 Email: [email protected] Deutsche Telekom Laboratories Nov 19, 2008 26.