Get Around 360º Hypervideo Luís A. R. Neng Teresa Chambel Faculty of Sciences LaSIGE, Faculty of Sciences University of Lisbon University of Lisbon 1749-016 Lisbon, Portugal 1749-016 Lisbon, Portugal +351217500087 +351217500087 [email protected] [email protected] ABSTRACT Traditionally, to view the content of a video, the user is locked to In traditional video, the user is locked to the angle where the the angle where the camera was pointing to during the capture, camera was pointing to during the capture of the video. With 360º and the resulting video has boundaries. But with 360º video video recording, there are no longer these boundaries, and 360º recording, there are no longer boundaries. A 360º hypervideo video capturing devices are becoming more common and player will let the users pan around to view the rest of the content affordable to the general public. Hypervideo stretches boundaries and quickly access related information provided by the even further, allowing to explore the video and to navigate to navigational links. Video is a very rich media type, presenting related information. In this paper, we describe an approach to the huge amounts of information that change along time. 360º video design and development of an immersive and interactive interface features even more information in the same time, and adds extra for the visualization and navigation of 360º hypervideos over the challenges when we cannot watch all around at the same time. But internet. Such videos allow users to pan around to view the it provides the whole picture all around the viewer, holding the contents in different angles and effectively access related potential to provide full immersive user experiences. information through the hyperlinks. Challenges for presenting this We are exploring navigation techniques to support users to get type of hypervideo include: providing users with an appropriate around a 360º hypervideo space, taking the 360º video watching interface capable to explore 360º contents, where the video should experience into a new level, through an immersive hypermedia change perspective so that the users actually get the feeling of space. The users can get around the current video and get around looking around; and providing the appropriate affordances to the hypermedia space, following navigational links available understand the hypervideo structure and to navigate it effectively during the hypervideo playback to extra or related information. in a 360º hypervideo space, even when link opportunities arise in The links take the user to other hypermedia contents such as text places outside the current viewport. and video. For example, when playing a video that contains the Liberty Statue, the users see an icon over the statue when a link Categories and Subject Descriptors may be followed, taking them to other hypermedia content, such H.5.1 [Information Interfaces and Presentation (I.7)]: as another video about the statue, some extra information on a text Multimedia Information Systems – video, hypertext navigation based page in this hyperdocument, or external information like a and maps; H.5.2 [Information Interfaces and Presentation Wikipedia page on a related topic. After the users finish watching (I.7)]: User Interfaces – interaction styles, screen design; H.5.4 the related information, they may return to the original video [Information Interfaces and Presentation (I.7)]: where they left off. In addition, by using summarization Hypertext/Hypermedia – navigation, user issues. techniques [8] an overview or outline of the content of the video can be provided to help the users get aware of the video content without watching it through, and navigate it in a more effective General Terms and flexible ways. Design, Experimentation, Human Factors. In the next sections, we describe 360º video formats and differences between hypervideo and 360º hypervideo. Next, we Keywords present the Get Around 360º Hypervideo player, focusing on Hypervideo, Hypermedia, 360º, Navigation, Video Streaming, design options for navigation support, followed by an overview of Hotspots, Link Awareness, Interactive Video, Immersive related work. Finally, the paper draws some conclusions and Interfaces. opens perspectives for future work.
1. INTRODUCTION 2. 360º VIDEOS With the 360º video capturing devices becoming more common Video has proven to be one of the most effective ways to and affordable to the general public, we are exploring navigation communicate, able to present in a rich cultural context a large techniques to view and interact with 360º hypervideos. quantity and diversity of information in a brief period of time [1]. With 360º videos, we are able to deliver an even richer and larger Permission to make digital or hard copies of all or part of this work for quantity of information than before, that can span all over the personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that viewer. 360º videos can be mapped onto cylindrical or spherical copies bear this notice and the full citation on the first page. To copy projections. A cylindrical projection is a video frame which looks otherwise, or republish, to post on servers or to redistribute to lists, perspective correct when mapped onto the inside of a cylinder. requires prior specific permission and/or a fee. Figure 1a is a sample of a video frame taken by a Sony Bloggie MindTrek 2010, October 6th-October 8th 2010, Tampere, FINLAND. PM5K Camera [9]. In Figure 1b, the same footage is projected Copyright 2010 ACM 978-1-4503-0011-7/10/10…$10.00. onto a long strip of video showing all 360º angles [10].
The example in Figure 2 is a table tennis game captured and interaction introduced by hypervideo allows the users to have projected with the same format as in Figure 1. We are able to see greater control over the contents they want to see. By extending both players in the projection [6]. The cylindrical projection was the hypervideo concept to hypervideo 360º, new challenges arise, actually invented in 1796 by the painter Robert Barker from especially because parts of the video may be out of sight. The Edinburgh who even took a patent on it [13]. 360º hypervideo player must provide to the users with the appropriate affordances to understand the hypervideo structure and to navigate it effectively in a 360º hypervideo space, while allowing them to have an immersive experience.
4. GET AROUND 360º HYPERVIDEO In order to be able to navigate in a 360º hypervideo space, we Figure 1: a) Cylindrical video before projection; b) Cylindrical need new navigation mechanisms. This section presents our main video after projection design options so far in the Get Around 360º Hypervideo Player, addressing the main challenges in hypermedia: disorientation and cognitive load, in this richer and more challenging 360º context. We focus in navigation mechanisms that help orientation with reduced cognitive load.
4.1 Video Navigation Drag Interface We developed a drag interface, illustrated in the sketches of Fig 4. This interface will let the users pan around to view the content in different angles. The users simply need to drag the content left or right to move the viewport. The user can move to one of the side continuously, to get around without any kind of boundaries. Location awareness became an issue due to the lack of Figure 2. Table tennis game in 360º video boundaries. During our tests, the users would often get lost in the 360º space without knowing in which angle they were looking at. A Spherical projection is a video frame which looks perspective correct when mapped onto the inside surface of a sphere [11]. Figure 3 shows this type of projection in a plane. 4.2 360º Hypervideo Navigation Two navigation mechanisms were developed to assist the users know where they are headed in 360º videos: • View area: is a circle similar to a pie graph where there is only one piece of pie that corresponds to the angle the user is looking at. It rotates to left or right when the user pans to the left or to the right. With this navigation mechanism, the user will be able to know the viewing angle. • Mini map: is the cylindrical projection video resized to a size where the users are able to see it all at once on the viewport. The mini map cursor (Figure 6) highlight which part of the video the users are currently watching.
Figure 3. Spherical video projection
Content outside viewport Viewport Content outside viewport 3. HYPERVIDEO & HYPERVIDEO 360º Content Content The use of video inside web pages consists mostly in the simple outside Viewport outside View area and orientation viewport viewport inclusion of this type of content, and the users are expected to Minimap watch it linearly. The interaction between the user and the video, in general, is limited to the actions like play and pause, fast Text box forward and reverse. In the last years, more and more researchers Content outside viewport Viewport Content outside viewport and companies are concerned with improving video functionali- ties and implementing higher interactivity over the video stream. Minimap Navigational Navigational The most promising one is likely to be the possibility of Links (hotspots ) Links (hotspots ) implementing hypervideo - providing true integration of video in hypermedia spaces, where it is not regarded as a mere illustration, Figure 4. Basic 360º Hypervideo Player Structure but can also be structured through links defined by spatial and Link awareness is vital for orientation. In hypervideo, link temporal dimensions [1]. It provides form of interactive video that awareness is more challenging than in traditional hypermedia greatly increases the video capabilities by providing flexible because links may have duration, several links can coexist in time interactive mechanisms that allow to navigate video and to and space and video changes in time. So, in order to be perceived integrate it with different types of media. The structure and by the users, more information has to be provided [1]. In 360º hypervideo, most of the content is invisible to the user because it that detects which are the coordinates of the thumbnail the user is not in the viewport, which means more work has to be done to clicked at. For example if the user clicks on the boy side of the provide new approaches for link awareness. third image map, it will link directly to the angle which is pointed to the boy and move the video play head to 00:00:09. A more We designed a hotspots availability and location indicators condensed version presents thumbnails of the central viewport in (Figure 5) to let the user know the availability and location of a every scene. It allows to present more scenes in the same space, hotspot outside the viewport. The hotspot availability indicators but limits awareness of the full surround video and flexibility to Y-axis position is used to locate them in the video, in terms of navigate to the desired viewport in one click. Instead, the users se- height. Their size is used to indicate how far away the hotspots are lect the scene and, on the scene, change the viewport as they wish. located. The closer the hotspot is located, the bigger is the hotspot availability indicator. The indicators are positioned at the edge of the screen and all the hotspots have a transparent design (not opaque) to minimize the impact over the video content. The Mini Map also provides information about the hotspot availability and location (Figure 5). We believe that a hotspot is not just a single link or button to other hypermedia contents. It must contain at least some information to give the users some expectation of the contents they are about to see after clicking on the hotspot (where to and when to link awareness). A comic balloon look-alike text- box (Figure 5) can accommodate a lot of important information. In the example of Figure 6 we place information like the name, description and the price on the text-box of the hotspot regarding the outfit of the girl.
When the users click on the hotspot, they may be redirected to a Figure 6. 360º Hypervideo Player Image Map Mode shopping website or the click information will be memorized in the system and displayed to the user on demand, later on. Some additional features are also provided in the traditional Depending on the characteristics of the video content, we can set timeline, or progress bar, to aid user orientation and control the 360º hypervideo player to open the link directly or memorize (Figure 7). Besides the traditional Play, Pause and Video Time it in the system. For example, if the purpose is learning, opening feature, we extended the timeline in response of some the link directly might make more sense because the users usually characteristics of a webpage. Since this is a web-based hypervideo watch that kind of video step by step. If the video is for player, we need to have in mind that the video requires time to entertainment, the users might not want to be interrupted by the load. A Bytes Loaded Bar gives the user the information on how link opening, so the link memorization system might come handy much content is loaded and prevents the user to access some in this kind of situation. The memorized links will be available in inaccessible area. For example, the video play head only can be the system and the users can access them whenever they want. moved within the Bytes Loaded Bar width. e.g.: at the end of the video.
Figure 7. 360º Hypervideo Progress Bar The hyperspace is navigated across spatial-temporal contexts that history recalls [1]. The Memory Bar provides information to the user about the parts of the video have been viewed. The Toggle Full Screen button switches the video between Full Screen or Standard Screen mode. The Full Screen Mode takes the user out of the frame of the browser and makes the video content fill the entire computer screen. This is a bridge to more immersive Figure 5. Hypervideo Player Mini Map Mode viewing modes, possible for example in 360º projection in a cave, The 360º Hypervideo Player also provides another detailed as we intend to explore in our future work. viewing mode (Figure 7). We could have the title and a description of the video. The Video Image Map Mode gives an 5. RELATED WORK overview of the content of the video. It is the type of Although the hypervideo concept exists since the early days of summarization technique described in [1,8] but with some extra hypertext, when Ted Nelson extended his hypermedia model to features for the 360º environment. include “branching movies” or “hyperfilms”, it has not yet been as Thumbnails refer to video scenes and are represented by widely adopted as it might have been expected, partially due to cylindrical projection in order to give, all at once, a full overview technological constraints in hardware and supporting tools [1,12]. of the video content over time, synchronously indicating the As video is becoming easier to process and transmit with good current scene, and offering the possibility to navigate where the quality, and faster through streaming over high bandwidth net- user chooses to go. The whole thumbnail is an area sensible link works, and we witness a growing popularity of public access to video on the Internet, it is becoming more relevant to adopt hy- problems as with early hypertext users becoming lost in pervideo and provide tools that support its authoring and access. hyperspace. As such, a 360º Hypervideo Player needs to be as clear as possible about how a user should interact with it. Video in hypermedia was discussed in late eighties and early nineties [1,12]. Since mid-nineties, a couple of works like Kon- Future work of this project will include more thorough usability Tiki and HyperCafe addressed aesthetical and rhetorical aspects of tests and development of different versions of the 360º Hyper- hyperlinking in video, others, like Hyper-Hitchcock [8], explored video Player in response to user feedback. The 360º Hypervideo "detail-on-demand video", focusing on summarization and buil- Player should not be locked on the web and it could be integrated ding on the concepts of spatial hypertext. Hypersoap addressed in kiosks or more immersive installations, for example in caves hypervideo in the context of entertainment in interactive TV, for with 360º projections. New or extended 360º hypervideo features example, delaying link following till the end of the programs. may need to be designed for these contexts. The access medium W3C SMIL language, based on the Amsterdam Hypermedia used to present the 360º Hypervideo Player is a web browser and Model, took an important step towards multimedia integration and the viewers use the mouse to interact with it. The rise of 3D video, synchronization on the web, but actual hypervideo support has multi-touch and eye-tracking technologies penetration could be been limited. Since late nineties, some commercial initiatives have also very interesting, allowing to build new kinds of visualization also taken place in the area of hypervideo, but with mild success. and interaction with 360º video spaces. These new ways of In the context of learning support, web lectures, more recent visualization and interaction come with new challenges but also works on computer-supported collaborative video analysis, and with potential for increased excitement to explore and experience. our own previous work [1,12] in the design and support of indivi- dual and collaborative learning with hypervideo, since late nine- 7. ACKNOWLEDGMENTS ties, have made some contribution to the field. However, visuali- We wish to thank the authors of the 360º videos in our examples. zing and interacting with 360º hypervideo is a quite new area of This work was partially supported by FCT through LaSIGE research. While there are a number of panorama photos viewers Multiannual Funding. and tools available, there is little work on 360º hypervideo. Google Street Maps [2] could be one of the widely known free 8. REFERENCES 360º photo viewer available on the market. It uses a spherical [1] Chambel, T., and Guimarães, N., 2002. Context Perception image projection to give the users an immersive feeling but it does in Video-Based Hypermedia Spaces. In Proc. of ACM not provide any kind of video. There is some very special Hypertext’02, College Park, MD, USA. navigation implemented due to its nature. The user can pan [2] Google Street Maps: around to see the contents in a 360º environment and “dive” into www.google.com/intl/en_us/help/maps/streetview/ the scene to follow the streets available. The YouTube website [3] iCinema Research Centre Sydney:www.icinema.unsw.edu.au could be one of the best web hypermedia players around when it started to support video annotations [14] that work like [4] Immersive Media: www.immersivemedia.com navigational links. It also allows the users to query databases for [5] Nelson, T., 1974. Branching presentational systems- specific topics and to obtain a list of video relevant to their search. Hypermedia, Dream Machines, 44-45. But it does not provide support for 360º video or hypervideo. [6] Ping Pong 360º:www.youtube.com/watch?v=3YKee3hxJwM Immersive Media [4] is a provider of 360º, full motion, and [7] Quake III Arena in a 360 Panoramic VR Installation with interactive videos. Their high-resolution camera is being used for PanoramaScreen, game demo: entertainment and advertising, street views, mapping and location- www.youtube.com/watch?v=Cqh_F4iN8pg based services, situational awareness and surveillance. They developed a 360º Video Player but it currently does not support [8] Shipman, F., Girgensohn, A., Wilcox, L. 2003. link navigations or annotations. The latest version of Sony Generation of Interactive Multi-Level Video Summaries. In Bloggie Handy Cam is one of the 360º capture devices available Proc. of ACM Multimedia 2003. pp.392-401. to the general consumer. It comes with a special lens that captures [9] Sony bloggie MHS PM5K 360º cylindrical video (before the video in 360º and with the Sony Picture Motion Browser we projection): www.youtube.com/watch?v=J9yJkRYeCGQ can convert the captured video into a cylindrical projection, but it does not support link navigations or annotations. Quake III Arena [10] Sony bloggie MHS PM5K 360º cylindrical video (after 360 Panoramic Virtual Reality Installation with PanoramaScreen projection): www.youtube.com/watch?v=LjVodf2Bnwg [7] is a cylindrical projection environment for 360º panoramic [11] Spherical video:www.youtube.com/watch?v=thDsCWQvzBI projections, which has been developed at the ZKM | Institute for Visual Media [15] in collaboration with the iCinema Research [12] Tiellet, C.A.B., Pereira, A.G., Reategui, E.B., Lima, J.V., Centre [3], Sydney, since 2005. It shares some features and Chambel, T. 2010. Design and Evaluation of a Hypervideo challenges with 360º hypervideo but it focuses in virtual reality Environment to Support Veterinary Surgery Learning. In and game interaction instead. Proc. of ACM Hypertext’2010, Toronto, Canada. [13] Yelick. S. 1980. Anamorphic Image Processing, MIT, May. 6. CONCLUSIONS AND FUTURE WORK http://dspace.mit.edu/bitstream/handle/1721.1/42957/089024 In this paper, we presented an approach for visualizing and 37.pdf?sequence=1 interacting with 360º hypervideos. Navigation in a 360º video [14] YouTube annotations: space and its navigational links presents a new experience for www.youtube.com/t/annotations_about most people and there are no consistent intuitions as to the [15] ZKM | Institute for Visual Media, Germany: behavior of these links. The users will likely experience similar http://on1.zkm.de/zkm/e/institute/bildmedien/