Eye tracking: the next natural step for VR

What is eye tracking?

Eye tracking technology has made leaps in the last 15 years, being successfully employed in a broad range of applications from medicine to marketing. In consumer devices, eye tracking is establishing its role as the next modality, making device interactions more intuitive and enhancing gaming by using the player’s gaze as an additional input. This document summarizes some of the possibilities eye tracking unlocks in VR, and how the technology contributes to the development of better devices and more powerful user experiences.

VR challenges

The VR race is on, and several of the world’s largest The key to success will be in striking the balance between consumer electronics brands are in it to win. Whoever attractive costs while providing outstanding user experience, succeeds in creating the most natural, immersive experiences graphic quality and sufficiently engaging content to entice will take the leading role in this exploding market. consumer demand. Despite the already significant market interest, mass-market One thing is certain: winning this race will not just come adoption of VR still faces a number of demanding challenges: down to speed. It requires significant innovation to create powerful, immersive virtual experiences. • Pricing • Image quality • Lack of attractive content

Better devices

Eye tracking is an essential component for building superior Eye tracking also allows devices to calculate inter-pupillary devices that are more power-efficient, mobile and adaptable. distance (IPD) – an important measurement to ensure HMDs are well positioned and suited to a person’s unique Foveated rendering, made possible by calculating the user’s eye positioning, ensuring a consistent experience across a gaze point on the screen, is the key to providing a high- broader range of the population. quality visual experience while optimizing performance. The capacity to measure the IPD lets the technology adjust By rendering only part of the screen in high definition, GPU to fit different individuals and ultimately create a more load is reduced dramatically. Since the human eye is only accurate 3D environment, improving immersion and reducing capable of processing a small area in crisp detail, the user nausea. gets the full benefit of a 4K resolution experience with standard GPU solutions. The reduced processing load ultimately results in lower system costs, lower power consumption and greater mobility for head-mounted displays (HMDs).

1 / 4 Better experiences Natural immersion

Eye tracking enables a unique toolset for creating a natural, immersive VR experience. When the VR HMD registers eye movement and positioning, the virtual environment can be responsive to the user’s attention and intent. Interactions become more intuitive by taking advantage of our natural hand-eye coordination and developers can artfully create intuitive, fluid and immersive experiences based on our gaze. From more realistic game mechanics to improved usability and enhanced immersion – or even personalization of content based on attention data – the possibilities are many.

Use cases

Here are a few examples of how eye tracking has been implemented to improve immersion and enhance gameplay:

Immersive Graphics Awareness Natural Targeting • Dynamic depth of field:The focus • Character awareness: NPCs • Throw at gaze: Hit your targets follows your gaze as in real life, (none playable character) can react more smoothly by aiming with the from the object in your hand to the to your gaze, just like humans would. aid of your gaze. scenery in the horizon. • Environment awareness: Objects, • Teleport/move to gaze: Look at • Directional sound: What you are lights and UI elements can respond a spot (or teleportation station) you looking at determines what you hear to your gaze. want to go to and press the button more clearly, or what sticks out in a to teleport. noisy environment.

When asking first-time users to compare their VR experience In many scenarios, eye tracking represents a paradigm with and without eye tracking, a common response is “it feels shift in VR. Rather than 1) looking, 2) turning your head or like I gained superpowers.” The interactions are reported as pointing with the hand controller and 3) selecting, you simply more fluid and the overall experience as more exciting when 1) look and 2) select. This means an entire step can be eye tracking is enabled. removed in just about every kind of interaction. This way, eye tracking can shorten the distance between intent and action, enabling users to perform the same action through fewer steps. You look, you click. The experience feels seamless and direct. It also removes the need for unnecessary and unnatural head movements, which most of today’s HMDs employ for certain applications. In real life, you don’t point with your forehead. You don’t navigate through a menu by moving your head. The emergence of eye tracking in VR could be considered similar to the implementation of touch in mobile phones. In both cases, the integration of sensors adapted to natural human biology allow for device interactions that are more intuitive, precise and direct.

2 / 4 Social interactions

The ability to turn VR into a more social and emotional experience is perhaps the most powerful form of interaction that eye tracking makes possible in VR. Our eyes don’t only reveal where our focus is – it’s also a significant way through which humans communicate and express emotions. Eye tracking opens the way for the accurate representation of human eye movements and one of the most significant ways through which all people connect beyond any cultural or language barriers: eye contact. Today, in VR multiplayer environments, the users are represented through avatars with motionless eyes that stare into the distance, giving them an unlifelike face. With eye tracking, the avatar’s eye movements follow that of the player, being better able to convey their emotions and thoughts.

A shared reality

The big promise of VR is in delivering the ultimate immersive experience – and it is commonly portrayed as an isolated activity. One user, living the otherwise impossible through a different reality presented right in front of their eyes. But with eye tracking, the immersion can be shared. People are able to socialize in virtual environments, communicate more naturally and ultimately get a stronger emotional experience.

The next step forward

A successful implementation and roll out of eye tracking in an HMD requires overcoming and piecing together three technical challenges.

• Optomechanics. Eye tracking begins with high frequency inter-pupillary distance measurements and pupil dilation series of pictures of your eyes. Integrated cameras deliver measurements. high quality images that enable the system to register your • UX design and easy-to-use Software Development gaze and your eye movement. Toolkits that enable developers to rely on eye tracking • Advanced computer vision algorithms, executed on a features to build their applications. standard SOC or Tobii EyeChip, transform the pictures into gaze x and y coordinates, eye open/close signals,

There are proven technical solutions for each of the challenges listed above. However companies that master all these technology areas are few.*

3 / 4 See the big picture

Eye tracking represents a paradigm shift in VR. By providing more natural, immersive and social interactions, it may well be a key factor propelling VR towards widespread market adoption. Fully functional eye tracking demos are available today**, and the forerunners in the VR industry are planning future product releases in which eye tracking is an indispensable component. The impact of making your VR HMD responsive to the user’s gaze is too great to be ignored. Eye tracking is the next step to taking VR further, building better devices and providing better user experiences.

About Tobii

• Tobii is the world leader in • Tobii is actively involved in the *Tobii has extensive experience **Working closely with leading eye tracking technology, development of the Khronos’ in the three critical knowledge game studios and hardware knowledge and services. The VR standard to be released areas highlighted above – suppliers around the world, company offers ET solutions in 2018. The standard will optomechanics, advanced the company offers unique ET for gaming, augmentative and include cross-platform APIs for computer vision algorithms, knowledge and resources, fully alternative communication, tracking HMDs, controllers and UX design and Software functional demos, a qualified behavioral research and other objects, and rendering Development Toolkits. development platform and technology integration. to a diverse set of display support, as well as business hardware. insight and opportunities for joint marketing activities.

What’s next?

Keep up to date with insights in the world of eye tracking and virtual reality at blog.tobii.com

4 / 4