Eye Tracking Data in Multimedia Containers for Instantaneous Visualizations
Julius Schoning¨ ,∗ Patrick Faion, Gunther Heidemann, and Ulf Krumnack Institute of Cognitive Science—Osnabruck¨ University, Germany
(a) (b) (c)
Figure 1: Instantaneous visualization of gaze data, show as yellow dots: (a) visualization of gaze data on a frame with VLC player — (b) like normal subtitles one can easily change between the gaze data of subjects — (c) exemplary gaze data on another video sequence.
ABSTRACT Thus, why not encapsulate gaze data alongside the associated Nowadays, the amount of gaze data records of subjects associ- video material in a joint container? This has become common prac- pdf ated with video sequences increases daily. These eye tracking data tice for storing text plus metadata, e.g., in the container. Nowa- ogg are unfortunately stored in separate files in custom-made data for- days video containers like the open container formats [20]( ), MPEG-4 Matroskaˇ MKV mats, which reduces accessibility even for experts and makes the [7], or the container format [9]( ) encap- data effectively inaccessible for non-experts. Consequently, we still sulate video and metadata like subtitles, audio comments, and in- lack interfaces for many common use cases, such as visualization, teractive features. These formats can be distributed as a single file, streaming, data analysis, high level understanding, and semantic played by standard multimedia players, and streamed via the Inter- web integration of eye tracking data. To overcome these shortcom- net. In this manner, the accessibility of gaze data will be increased substantially if they are encapsulated in a common container for- ings, we want to promote the use of existing multimedia container 1 formats to establish a standardized method of incorporating con- mat, cf demonstration video . tent videos with eye tracking metadata. This will facilitate instan- Thus, we argue that video eye tracking data sets should be stored taneous visualization in standard multimedia players, streaming via in a multimedia container, carrying the corresponding video, the the Internet, and easy usage without conversion. Using our proto- gaze trajectories of multiple subjects and other video related data. type software, we embed gaze data from eye tracking studies and We present a software and a multimedia container format allowing the corresponding video into a single multimedia container, which to combine gaze data of several subjects with the video material can be visualized by any media player. Based on this prototype in a single multimedia container. Current multimedia containers implementation, we discuss the benefit of our approach as a possi- already support a variety of video data formats and with our ap- ble standard for storing eye tracking metadata including the corre- proach add instantaneous visualization of gaze points in standard sponding video. media players or in slightly modified versions of them. Our long- term aim is to establish a standard format that facilitates applica- Index Terms: H.2.4 [Information Systems]: Systems — tion in various fields, ranging from annotation for computer vision Multimedia databases; I.2.10 [Computing Methodologies]: Vision learning algorithms over highlighting objects in movies for visually and Scene Understanding —Representations, data structures, and impaired people to creating auditory displays for blind people or transforms video analysts. Such a standard format can also boost accessibility and shareability of eye tracking data as well as combination with 1 INTRODUCTION other metadata. Gaze data information belonging to video files is still commonly Our contribution focuses on the instantaneous visualization of stored separately next to the video file in custom file formats. The eye tracking data, but we also try not to neglect other kinds of data structures within these files are mostly customized, sometimes metadata. The paper starts with an extensive review of available unique, and they are stored in a diversity of formats, e.g. plain text, data formats for video annotation, metadata description, timed data XML, Matlab mat format, or even binary. As a consequence, one formats, and multimedia containers. Based on a discussion of suit- needs special tools for accessing and visualizing this data, which able data representations for scientific metadata, we convert exist- leads to the fact that the use of this data is almost impossible for a ing eye tracking data sets [13,8,4] with its metadata—gaze data general audience. 1Demonstration video, software including all tools, and converted eye tracking data sets can be downloaded at https://ikw.uos.de/ ∗e-mail:{juschoening, pfaion, gheidema, krumnack}@uos.de %7Ecv/publications/ETVIS16 c 2016 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. of subjects, and annotations of objects—into multimedia contain- lows temporal interpolation using connected polynomials, of which ers with our software. These containers bundle all information and linear interpolation is a special case. On this basis, the SPATIO- provide an instantaneous visualization in standard video players. TEMPORAL LOCATOR describes spatio-temporal regions of an ob- We then compare the advantages and drawbacks of our approach ject of interest in a video sequence. and summarize the possible impact a standardized multimedia con- tainer will provide. 2.2 Timed text Timed text [16] refers to time-stamped text documents that allow 2 DATA FORMATS relating sections of the text to certain time intervals. Typical appli- Today’s video eye tracking data sets [17, cf. Chap. 2.2] are stored cations of timed text are subtitling of movies, captioning for hearing in a diversity of formats, e.g. plain text, XML, Matlab mat format, impaired or people lacking audio devices. The most simple form of or even binary. But hardly any these formats provides a method to timed text consists of units providing a time interval and a text to be support streaming of video data together with the metadata. This displayed during that interval. This is basically what is provided by is somewhat disappointing, considering the fact, that many stream- the popular SubRip format (SRT). able formats, in the domains of DVD, Blu-ray, or video compres- sion, exist. Some of these formats might be capable of carrying 2.3 Subtitle and Caption Formats these eye tracking metadata next to the video to fit the necessary The universal subtitle format (USF) was an open specification [11] requirements, give an easy to use visualization within a standard which aims at providing a modern format for encoding subtitles. multimedia player, and still provide the opportunity of expert visu- It tries to be comprehensive by providing features from different alization in specific tools. existing subtitle formats. An XML-based representation is chosen to gain flexibility, human readability, portability, Unicode support, 2.1 Metadata Formats hierarchical structure and an easier management of entries. The The content of video material is difficult to access by machines and subtitles are intended to be rendered by the multimedia player, al- e.g. eye tracking data may provide valuable hints for automatic pro- lowing to adapt the display style (color, size, position, etc.) to fit cessing. However, the lack of standardization makes it hard to com- the needs of the viewer. However, there are also tools to generate bine or exchange such eye tracking data, or to provide search across pre-rendered subtitles, e.g., in the VobSub format for players that do different repositories. not support USF. Standards for metadata abound and have become popular with Nowadays, the open source project USF [11] has become pri- the rise of the semantic web. The RDF standard [15] provides a vate, so the community does not have any influence on the develop- general format to make statements about resources, i.e., virtually ment. The latest version 1.1 has parts which are still under devel- everything that can be given a unique name. It comes with well- opment. Consequently, some parts—,e.g. the draw commands—are defined formal semantics and allows a distributed representation. incomplete. In addition to visualization of data on top of the video, However, it provides only a very limited predefined vocabulary, re- USF provides a comment tag which would allow storing additional quiring applications to extend the language for specific domains by information—not for display but for exchange. specifying schemes. By now, several of such schemes exist, but to Sub Station Alpha (SSA) is a file format for video subtitles, which our knowledge, no standard for the description of video material has been introduced with the subtitle editor Sub Station Alpha. It has been evolved. Videos feature a temporal and spatial structure, has been widely used by fansubbers and support has been imple- distinguishing them from most other types of data and requiring a mented for many multimedia players. The extended version V4.00+ special metadata framework. [14]—also known as Advanced SSA (ASS)—includes simple draw- rd The Continuous Media Markup Language (CMML)[12] is a for- ing commands, supporting straight lines, 3 degree bezier curves, rd mat for annotating time-continuous data with different types of in- and 3 degree uniform b-splines, which is probably sufficient to formation. It was developed to improve the integration of multi- roughly mark objects in a scene. Today’s video players usually sup- media content into the world wide web, providing markup facilities port the ASS drawing commands, but in older players, the drawing for timed objects in a way similar to what HTML provides for text commands are not implemented. documents. With CMML, temporal intervals (clips) can be marked and described using a predefined vocabulary, allowing to provide 2.4 Multimedia container textual descriptions, hyperlinks, images (e.g., a key frame), and not Multimedia objects consist of multiple parallel tracks, usually a further specified metadata in form of attribute – value pairs. While video track, one or more audio tracks, and some optional subtitle CMML is able to address the temporal structure of videos, it pro- tracks. These tracks are often combined into a single container for vides no specific means for referring to pixels, regions, or space- storage, distribution, or broadcasting. In contrast to classical data time volumes. archives, multimedia containers have to account for the temporal Probably the most advanced metadata framework for multime- nature of their payload, in order to support seeking and synchro- dia content is the Multimedia content description interface defined nized playback of the relevant tracks, as shown in Fig.2. To embed in the ISO/IEC 14496-3 standard, which has been developed by the data into a container, a packager needs some basic understanding of Moving Picture Experts Group and is also known as MPEG-7. It the data format, at least enough to understand its temporal structure. specifies a set of tools to describe various types of multimedia in- Further, certain aspects of the data or the container format may pre- formation. However, MPEG-7 has been criticized for lack of a for- vent a direct encapsulation. In brief: Not every payload is suited mal semantics, which causes ambiguity leading to interoperability for every container. problems and hinders a widespread application [10,1]. Common multimedia containers are the VOB and EVO formats MPEG-7 provides means to describe multimedia content at dif- used for DVDs that are based on the MPEG-PS standard. The more ferent degrees of abstraction. It is designed as an extensible for- modern MP4 format, specified in MPEG-4 (ISO/IEC 14496, Part mat, providing its own “Description definition language” (DDL – 14) [7], was established to hold video, audio and timed text data. basically an extended form of XML Schema). Its basic structure de- Though MP4 can not handle arbitrary video, audio and timed text fines some vocabulary, which can be used for different aims: GRID formats, it does conform to the formats introduced in the rest of the LAYOUT,TIME SERIES,MULTIPLE VIEW,SPATIAL 2D COORDI- standard. NATES, and TEMPORAL INTERPOLATION. Especially the TEMPO- The free ogg container format was originally put forward by RAL INTERPOLATION is quite interesting for object labeling. It al- the non-profit Xiph.Org [20] foundation for streaming VORBIS en- c 2016 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. coded audio files and is nowadays supported by many portable de- vices. With the introduction of Theora and Dirac video formats, Header ogg has also become a popular format for streaming multimedia
content on the web. The open Matroskaˇ container format [9](MKV) Static aims at flexibility to allow an easy inclusion of different types of General Metadata Payload payload. It serves as a basis for the webm format, which is pushed forward to establish a standard for multimedia content on the web. Beyond subtitling, the inclusion of timed metadata into multi- media containers seems to be discussed only sporadically. The em- bedding of CMML descriptions in an ogg container is one such ap- proach [19], but it seems quite limited due to the restricted data format, not providing direct support for spatial features like region Video Audio 1 Audio 2 Subtitle 1 Subtitle n markup. Subtitle 2 Probably the most matured specified approach is the embedding Temporal Payload (eye tracking data) of MPEG-7 [6] data into MP4 [7] containers. Up to now, we know (eye tracking data) (eye tracking data) of no software that is able to do this embedding. time t 3 SUITABLE DATA FORMAT What kind of data format would be best suited to store and share eye Figure 2: General data structure of a multimedia container, the tracking data of several subjects? Formats for metadata, as seen in header and metadata which have no temporal dependencies are Section2, abound and keep growing, fostered by the advance of stored before the temporal video, audio, subtitle, and metadata. For the semantic web and other digital technologies. While general for- streaming this video, only the non-temporal data have to be transmit- mats, like RDF [15], are well established, they are not geared to- ted before playing, the temporal data are transmitted during playing. wards the description of video material, and hence miss the required vocabulary. We suggest three approaches to reach a standard for- mat for video eye tracking data: i) extension of a general metadata ported by existing media players. Although these technologies are formalism like RDF with video and eye tracking vocabulary, ii) us- not triggered towards the storage and presentation of eye tracking ing a well-defined standard for video metadata like MPEG-7, and data, some important features can be realized based on these for- iii) an ad hoc approach that utilizes existing technologies, e.g., for mats. This kind of “hijacking” formats has the benefit that they will subtitles or captions, to store eye tracking data. be widely supported by current multimedia players. Even though The first approach, i.e. the development of a specialized eye there seems to be no general drawing framework, some subtitle for- tracking format on the basis of a well-established metadata frame- mats include drawing commands that allow highlighting regions in work like RDF has the obvious advantage that it can build on the a scene. These commands can also be used to visualize eye track- vast amount of software libraries and tools that support storage, ing data. Using a player’s standard methods for displaying subtitles management, exchange, and certain type of reasoning over such and switching between different languages, one can then display data. The main drawback is, that these formats lack any form of eye tracking data and switch between data of different subjects. support for the video domain, and do not even provide basic spatial When putting annotations into a hijacked format one has to be or temporal concepts. The development of a specialized vocabulary careful that no information is lost. Additionally, one should bear in to describe eye tracking scenarios would mean a huge effort, and it mind that the original format was designed for some other purpose, would have to include the implementation of tools for visualiza- so it may not support desired features, e.g., simultaneous display tion. Desirable features, like streamability, seem not to fit well with of gaze points of multiple subjects. We chose to use USF for three the originally static nature of these data formats. Furthermore, the main reasons: First, its specification considers possible methods specificity of the field of eye tracking would make a wide support for drawing shapes, a prerequisite for instantaneous visualization by common multimedia players unlikely. with a normal multimedia player. Second, it allows for storing ad- Using a specialized, well-established standard for video meta- ditional data, a necessity for carrying all eye tracking data so that data like MPEG-7 is the second possible approach. MPEG-7 has a expert visualization with specialized tools is possible from the same well defined and established vocabulary for various annotations but single file. Finally, and most important, USF is—like the preferable lacks on a vocabulary for eye tracking data. Nevertheless, MPEG-7 MPEG-7—an XML-based format and thereby capable to hold com- supports the description of points or regions of interest in space and plex data structures. However, although basic USF is supported by time and hence seems well suited to store the essential information some existing media players, the drawing commands are not imple- of eye tracking data. Unfortunately, no standard media player (like mented. We provide an extension for the VLC media player that VLC, MediaPlayer, and Windows Media Player) seems to currently implements some drawing commands. In addition, we provide a support the visualization of MPEG-7 video annotations. Hence, converter to the ASS format, which is widely supported, including one would have to extend these players to visualize embedded eye drawing commands, due to the free ASS library, thereby allowing tracking data—fortunately; one can build on existing MPEG-7 li- out of the box visualization of eye tracking data that works with braries [2]. We think that when implementing such multimedia many current multimedia players. However, its plain text format player extensions, one should aim at a generic solution that can is too restricted to hold all desirable information. Both approaches also be used to visualize other MPEG-7 annotations, as this would will be discussed in more detail in the next section. foster development, distribution, and support. Even though the first two approaches seem better suited in the 4 PROTOTYPES long run, they seem not realizable with overseeable effort and on Following the previous discussion to hijack, or more precisely, to a short time scale. Hence, in the remainder of this paper, we fo- modify an existing subtitle format for incorporating eye tracking cus on the third approach, which allows for the quick develop- data (cf. Section3), two kinds of multimedia container prototypes ment of a prototype, to demonstrate the idea and gain experience were implemented. The first one is based on USF and encapsulates in its application. The idea is to adopt existing technologies, like lossless the complete eye tracking metadata for visualization in a subtitles, captions, audio tracks, and online links [5], already sup- modified version of the VLC media player. The second one is based c 2016 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. on ASS and only able to carry selected metadata, but this data can the eye tracking data set without that player. Therefore, we provide be visualized by most current media players. a second prototype based on ASS, a subtitle format with drawing commands, that is widely supported due to the free ASS library. 4.1 Metadata as USF In contrast to USF, the ASS subtitle format can not carry all desired In order to use USF for encapsulating eye tracking data, we an- metadata as it is not capable of representing complex data structures alyzed which features of USF are available in the latest releases and it does not account for non-visualizable content. of common multimedia players. One of the most common me- From our USF files with eye tracking data of all subjects, created dia players is the VLC media player. The current version 3.0.0 for the first prototype, ASS files are generated using XSLT (Extensi- already supports a variety of USF attributes, which are text, image, ble Stylesheet Language Transformations) with a simple translation karaoke and comment. The latest USF specification introduces an stylesheet. After the conversion, a MKV container is created in- additional attribute shape that is still marked as under development, cluding the video and one ASS track for each subject. The resulting although this specification is already quite old. Since eye tracking container makes metadata accessible for a broad audience, as the data is commonly visualized with simple geometric shapes, like cir- ASS visualization can be displayed by many unmodified players. cles and ellipses, the use of the shape attributes for instantaneous Listing 1: Section of the USF specification [11], * marked attributes gaze visualization of subjects seems to be quite appropriate. are added to the specification and implemented in our altered VLC Since the exact specification of the shape attribute is, as men- player. tioned, not complete, we particularized it with respect to rectangles, ... polygons, and points, as illustrated in Listing1. These simple geo-
4.2 Metadata as ASS 6 CONCLUSION Since the USF based prototype requires a modified version of the The importance of eye tracking data in general and especially for VLC media player, a broad audience is still excluded from watching video analysis will significantly increase if they are provided in c 2016 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. multimedia containers as we suggest in this work. These contain- [15] W3C. RDF - semantic web standards https://www.w3.org/ ers can be interpreted by common video players in the same way rdf/, Feb. 2016. as today’s subtitles. In our opinion, the research community should [16] W3C. Timed text working group http://www.w3.org/ seek to established a standard based on MPEG-7 for recording, ex- audiovideo/tt/, Feb. 2016. changing and visualizing eye tracking data. Due to its proper spec- [17] S. Winkler and S. Ramanathan. Overview of eye tracking datasets. In ification [6], it provides a platform for consistent implementations Fourth International Workshop on Quality of Multimedia Experience, in all media players. A further advantage is that MPEG-7 can be QoMEX 2012, Melbourne, Australia, July 5-7, 2012, pages 212–217, encapsulated into a MP4 [7] container, so that both video and its 2013. related annotations are stored as one streamable file. Unfortunately, [18] C. Wylie, G. Romney, D. Evans, and A. Erdahl. Half-tone perspective we recognize a lack of MPEG-7 standard libraries for media play- drawings by computer. In Proceedings of the November 14-16, 1967, Fall Joint Computer Conference, AFIPS ’67 (Fall), pages 49–58, New ers. Therefore, we present an ad-hoc prototype allowing us to pro- York, NY, USA, 1967. ACM. mote the idea of embedding eye tracking data in multimedia con- [19] Xiph.org. CMML mapping into Ogg https://wiki.xiph. tainers. Our approach (mis)uses the USF subtitle format to encode org/index.php/cmml, Feb. 2016. eye tracking data, allowing to visualize them in a patched version of [20] Xiph.org. Ogg https://xiph.org/ogg/, Feb 2016. the popular VLC media player. We also provide a converter to gen- erate an ASS version of the data, which can be played by many other media players1. For the future, we plan to expand this approaches to support the embedding of almost every kind of scientific meta- data, so that annotation of objects, inter-object relations, conditions of objects, etc. can be stored in a single file. We thereby strive for a wider distribution and easier accessibility of such data.
ACKNOWLEDGEMENTS This work was funded by German Research Foundation (DFG) as part of the Priority Program “Scalable Visual Analytics” (SPP 1335).
REFERENCES [1] R. Arndt, S. Staab, R. Troncy, and L. Hardman. Adding formal se- mantics to MPEG-7. Arbeitsberichte des Fachbereichs Informatik 04/2007, Universitat¨ Koblenz-Landau, 2007. [2] W. Bailer, H. Frntratt, P. Schallauer, G. Thallinger, and W. Haas. A C++ library for handling MPEG-7 descriptions. Proceedings of the 19th ACM international conference on Multimedia - MM 11, pages 731–734, 2011. [3] J. E. Bresenham. Algorithm for computer control of a digital plotter. IBM Systems journal, 4(1):25–30, 1965. [4] A. Coutrot and N. Guyader. How saliency, faces, and sound influence gaze in dynamic social scenes. Journal of Vision, 14(8):55, Jul 2014. [5] J. R. Giorgio Bertellini. DVD supplements: A commentary on com- mentaries. Cinema Journal, 49(3):103–105, 2010. [6] ISO/IEC. Information technology—multimedia content description interface—Part 3: Visual (ISO/IEC 15938-3:2001), 2001. [7] ISO/IEC. Information technology—coding of audio-visual objects— Part 14: MP4 file format (ISO/IEC 14496-14:2003), 2003. [8] K. Kurzhals, C. F. Bopp, J. Bassler,¨ F. Ebinger, and D. Weiskopf. Benchmark data for evaluating visualization and analysis techniques for eye tracking for video stimuli. Workshop on Beyond Time and Errors Novel Evaluation Methods for Visualization (BELIV), pages 54–60, 2014. [9] Matroska. Matroska Media Container https://www. matroska.org/, Feb 2016. [10] F. Mokhtarian and M. Bober. Curvature Scale Space Representation: Theory, Representation: Applications, and MPEG-7 Standardization, volume 25 of Computational Imaging and Vision. Springer, 2003. [11] C. Paris, L. Vialle, and U. Hammer. TitleVision - USF specs http://register.titlevision.dk/files/ usf-specs-html.zip, Feb. 2016. [12] S. Pfeiffer, C. D. Parker, and A. Pang. The Continuous Media Markup Language (CMML), Version 2.1 https://www.ietf. org/archive/id/draft-pfeiffer-cmml-03.txt. Inter- net Engineering Task Force (IETF), March 2006. Draft! [13] N. Riche, M. Mancas, D. Culibrk, V. Crnojevic, B. Gosselin, and T. Dutoit. Dynamic saliency models and human attention: A compar- ative study on videos. Computer Vision ACCV 2012, pages 586–598, 2013. [14] Sub Station Alpha v4.00+ Script Format moodub.free.fr/ video/ass-specs.doc, Feb. 2016. c 2016 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.