Dynamic Depth and Immersive 3D Interaction Sugon Kim, Min-Joon Kim, and Chieteuk Ahn School of Integrated Technology, Yonsei University, Incheon, South Korea Abstract— This paper focuses on a new immersive 3D adding some sort of feedback in the form of pseudo-haptics, interaction method where depth has more emphasis. We it is not easy to provide appropriate responses to the users first define what we consider dynamic and static depth relying solely on software. Another issue with 3D mice is the information. We propose a new user interface method with high learning curve the users go through to get comfortable emphasis on depth-based 3D environments. The main idea is with the device. While more degrees of freedom are added linking additional information on objects’ superficial depth for easier navigation, it may not be so in the beginning as that we know as closeness. On top of that we propose a the user is only able to achieve the ease of use through system where the user can easily interact with the 3D objects repeated trials and errors. For professionals this may not by controlling and defining the depth of the objects. We pose a problem, as they are required to use the device more envision an array of applications where dynamic depth and often than not. However, to a regular, non-professional user depth interaction can take place, despite the fact that our 3D mice may not suit them particularly well. Even with some current prototype demonstrates only basic interaction at the difficulties, it is more likely that the average user will be able moment. to navigate and manipulate 3D objects with a 2D mouse. 3D navigation devices should be accessible to a wide range of Keywords: Immersive 3D interaction, 3D interaction, depth ma- users, and to achieve that intuitiveness is key. nipulation, active depth, depth information Devices such as the Wii™ and Kinect™[4] have extended the ways a user interacts with a computer with their versa- 1. Introduction tility. Microsoft’s Kinect is widely used in many areas such In recent years we have seen a massive increase of as gesture recognition and depth mapping, and Nintendo’s products and research in 3D contents and its related fields of WiiMote has found usage as a device for 3D interaction study. Starting with the movie “Avatar” in 2009, more and [3][5]. Not only these devices are meant for games, but also more 3D movies have been released in succession. Wide they have been used for many research topics across the success in the movie industry has propelled 3D technology globe. While intuitive and easy for the user to “just use” to its current state [1]. With 3D movies and television shows the device, it has been observed that using these devices for becoming more and more widespread with the commercial- long periods of time cause fatigue to the user. ization of 3D television sets, it is possible to conclude that A common topic that the devices mentioned above do we will see more 3D contents in the foreseeable future. With not consider is the manipulation of depth, or rather the the rise of these 3D contents comes the question: how do proactivity of depth. In our own opinion, depth has been we interact with these 3D contents? a passive aspect in many 3D interfaces. Objects have been Traditional mice have been such a success in navigating given depth values in order to show the closeness of an 2D environments that we are so accustomed to using each object to the users. Zooming in or out and differing the day. But these tend to be limited in degrees of freedom views of the environment has long been used and is still for easy 3D navigation [2]. On the other hand advanced the same. It can be said that depth data has only been used devices developed in order to deal with specific tasks in in terms of the traditional meaning of depth: how close or 3D. Numerous products have been developed, starting from how far? In this research we give additional values to depth, familiar devices such as the traditional mouse to the more such as information or multimedia data, in order to make advanced 3D mouse, and even gesture recognizing devices depth a more active part of a 3D interface. Users will be such as Nintendo’s Wii™[3] and Microsoft’s Kinect™[4]. able to interact with an object’s depth and its additional Such devices have been developed for gaming purposes, but information associated with their depth values. This is an they also have contributed towards making 3D navigation attempt to extend the user’s control and allow 3D interaction and 3D interaction in general easier to the user. While these more intuitive and easier to engage. devices have greatly improved 3D user interactions it does First, we provide background information and related not mean that such devices do not have their shortcomings. work in section 2. In section 3 we discuss our approach 3D mice, although have the degrees of freedom to success- and concept as well as some example applications. We then fully navigate 3D scenes, tend to lack the haptic feedback. present our prototype in section 4 along with results of our The lack of feedback to the user tends to make a device more experiment demonstrations and conclude our paper in section difficult to use [2]. While the suggested solution would be 5. 2. Related Works In this research, we intend to combine elements of a 2D While there are many papers and research regarding vir- interface and that of a virtual reality interface in order to tual reality interfaces, we have not been able to find a paper efficiently and accurately control depth and also objects of that incorporates interaction with depth as in this paper. interests to the user. Regardless, our research is relatable to several research topics in regards to virtual reality and virtual interfaces. This 2.2 Varying Stereoscopic 3D Displays to the paper builds upon the following areas of previous research: User based on Head/Eye Tracking 1) gesture recognition and virtual interfaces; and 2) varying the stereoscopic 3d display to the user based on head/eye location. 2.1 Gesture Recognition and Virtual Interfaces Currently the way we interact with a computer is through a monitor and a set of input methods such as the keyboard and mouse combination or one’s finger to name a few. We have been using this idea since the first graphical user interface came to desktop. Here we have all our files in a Fig. 1: Concept diagram two-dimensional space where depth is nonexistent. We are familiar to this desktop concept and we have been using this Our intention is to combine 2D interface and 3D interface for quite some time. elements together, and this includes displaying the display Nowadays we have gone beyond the desktop concept, according to the user’s gaze or location of the head and its adding in three-dimensional information and trying to simu- features. Such research done by Kwon et al. [11], Newman, late not only one’s desk, but also one’s entire workspace. et al. [12], and Chen, et al. [13] have been tremendous help New interaction methods have been proposed for out-of- in terms of implementing our own prototype. While our screen objects and different input methods such as gesture initial solution does not include the use of eye and gaze in a virtual environment. Du et al. [6] use a 3D TV and tracking, we do use facial features in order to figure out the Wii remote devices to simulate interaction with out-of-screen viewing angle of the user so proper adjustments can be made 3D objects displayed on the television. They envision menu in the display. interactions, volume control, and browsing functions with out-of-screen interactions and gestures. Mine et al. [7] has provided a set of general rules to follow 3. Dynamic Depth and Depth Interaction in virtual-environment interactions. The problem is that in Before we further explain our ideas, we will illustrate virtual environments, the users lack haptic feedback in order what we define as “dynamic” depth information and "static" to be able to properly orient themselves to the objects they depth information. Static depth information is the depth are manipulating. They suggest the use of proprioception for value itself. Information such as how far an object is away the control of such objects. from the base of the screen is an example. Zooming in and Many studies do implement proprioception in some ways; out and changing the display also is considered static depth. as we can see in studies conducted use gestures, cameras, On the other hand, we consider dynamic depth infor- and other methods in order to correctly position the user mation to be adding information relative to the object’s to the relative material on the screen. Keefe et al. [8] have depth value. We have envisioned some examples that can developed an artistic medium called CavePainting, which is be implemented using our system. One example is a 3D file a fully immersive 3d interface. The users are able to paint on navigation system. One can imagine this as a 3D cabinet. a screen “directly” using a brush or various other methods A user can “press” on a folder and pull his/her finger away such as throwing paint on the canvas using buckets. from the monitor. The contents of that particular folder will Banerjee et al.