Introduction to mixed realities Course 03 Content

● Recapitulation Course 2 ○ 3D Modeling, Blender, Unity ● Motion Tracking ○ How tracking works ○ Position Tracking ○ Tracking Systems ○ Motion Tracking in Blender ● Conclusions ● Bibliography

2 R - Basic Elements

● 3D Modelling - Polygon Modeling - Basic elements, Shaders, Textures, UV Mapping, Rigging

3 R - Basic Elements

● Blender ○ Scene Collection ○ Add ○ Shading

4 Interface to the Input

VR systems require not just a means for the user to tell the system what they want but a means to track at least some part of their body

User monitoring includes the continuous tracking of both user movements and user-initiated actions, such as pressing a button or issuing a voice command to the system

Continuous tracking of the user's movements is what allows the system to render and display the virtual world from a user-centric perspective-providing the effect of physical immersion

5 User Monitoring in VR/AR

In user monitoring there are active ways the user inputs information into the system; these can include the use of spoken commands, physical controls like wands, joysticks, steering wheels, dashboards, and keyboards, props, and platforms

In user monitoring, there are also passive ways in which information is supplied to the computer. Passive methods tell the computer how and where the participant is moving and where they are looking. These methods include tracking of the body (including hands, eyes, and feet) and position tracking that tells the computer the participant's location and orientation

6 Motion Tracking

Tracking or motion tracking can be defined as the process aiming to capture, follow, and get information about an object’s orientation and position, to be transferred to an application for further processing

This kind of tracking is used in VR to control the position of the user, joysticks and other important objects, whereas is used to track the object or market to be augmented and control the user’s position and orientation

7 Motion Tracking - History

Muybridge in 1878 used photographic techniques to quantify patterns of human movement

Motion tracking has been present in our lives since the 1970s, when they were used for the first time as a photogrammetric analysis tool in researches related to biomechanics

8 Position Tracking in VR

Being in a simulation is like entering another world, but true immersion comes when you are able to move within this world

Degrees of Freedom also known as DoF, which usually comes in 3DoF and 6DoF flavors

9 Degrees of Freedom (DoF)

DoF refers to the movement of a rigid body inside space. It could be explained as “different basic ways in which an object can move”

All objects around you that can move in 3D space move in 6 ways: 3 directional axes and 3 rotational axes. Each of these axes is a DoF. We live in a 3D world and interact it in all 6 DoF

10 DoF - Movements

Translation Movement - A body is free to translate in 3 degrees of freedom: forward/back, up/down, left/right

Rotation Movement - A body can also rotate with 3 degrees of freedom: pitch, yaw, and roll

11 DoF - Examples

Elevator is constrained to 1 DOF (a vertical translation), but inside this DOF it can freely move up or down

Ferris wheel is constrained to 1 DOF (a rotational DOF), where it can roll one way or the opposite way

A bumper car has 3 DOF in total: it can translate in only 2 of the 3 axis (it cannot move up and down like an elevator), and it can rotate in only one way (it cannot change its pitch or roll like an airplane). So 2 translations + 1 rotation makes 3 DOF

12 3 DoF in VR

In VR, DoF is used to describe axes that are being tracked. Tracking comes from the ability to monitor a change of angle or distance on the axes by using hardware

This exists in mobile and standalone VR headsets like the new Go. If you turn your head while wearing a headset, it is able to track the angle change of this axes and allows you to look around in the environment

13 Mobile VR Headsets

Oculus Go, Samsung Gear VR, and View only have rotational tracking (3DoF)

You are able to look up or down, to either side or tilt your head. But if you try to lean or actually move your head’s position, this is not tracked. The entire virtual world will move with you

14 IMUs and DOF

An Inertial Measurement Unit (also called an IMU or sometimes ‘tracker’), is an electronic device that measures and reports velocity, orientation, and gravitational forces, using a combination of sensors (, and magnetometers)

IMUs in the past was used as aircraft instrumentation, but nowadays they are used in all sorts of devices, including mobile phones

15 What is Positional Tracking and Why is It So Important for VR?

Positional tracking is a mix of hardware and software which is able to detect the absolute position of an object

In VR, in combination with orientation tracking it becomes possible to measure and report all the 6 DOF of real-life

The provides 3 DOF (rotational) head-tracking which is not positional. But, you won’t be able to lean down to inspect objects on the floor or lean around corners because there’s no way to tell exactly where you head is

16 Parallax

Positional tracking can also help improve the 3D perception of the virtual environment thanks to an effect known as ‘parallax’

Parallax is the way that objects that are closer to our eyes move faster than objects far from our eyes. This effect helps our brains to perceive distance in conjunction with stereoscopy (the difference between what our left and right eyes see)

Johnny Lee uses the wiimote to shows how parallax helps us to perceive 3D https://www.youtube.com/watch?v=Jd3-eiid-Uw

17 How do common virtual reality tracking systems work?

Tracking systems typically consist of a device capable of creating a signal and a sensor capable of reading it

These can take on various forms including optical, electromagnetic and acoustic signals, mechanical and inertial systems

18 Tracking Systems

In optical systems emitted light is captured by cameras in various formats

Electromagnetic tracking can be used where small electrified coils affect other electromagnetic sensors, and how their magnetic field affects others can position them in space

Acoustic systems use ultrasonic sound waves to identify the position and orientation of target objects

Mechanical tracking can use articulated arms/limbs/joysticks/sensors connected to headsets or inside them, much like the inertial tracking in phones often made possible by accelerometers and gyroscopes 19 VR Tracking with Optics

This method usually use cameras of one sort or another connected to a computer

Usually, the person being tracked has optical markers (dots of highly reflective material) on certain known points of their body or on the equipment such as the HMD or handheld controllers

When a camera installation capable of calculating depth sees a marker it can map it to 3D space. For example, two cameras at known angles that both see the same dot allow for this mathematical calculation

20 Passive and Active Markers

Reflective markers are known as passive markers as they reflect light

Active marker are computer controlled LEDs that allow for more accuracy and various workarounds for the weaknesses of passive marker methods. LEDs may be different colors or flash rapidly in sync with the capture system

Active marker systems perform better than passive ones, but the user has to wear a power supply or be tethered to the system somehow, which is clearly an encumbrance 21 Markerless Optical Systems

The principle of these markerless option is to avoid the use of extra elements in the detection of the object in motion, by means of an analysis based on image capture, segmentation and processing to extract the position of the object to be tracked

It is common to use techniques based on the background subtraction and multiple learning algorithms

22 Electromagnetic Systems for Tracking

These systems are based on electromagnetism theories, which describe the interaction between magnetic fields and electricity

The electromagnetic can calculate accurately and in real time the object position and orientation in the environment

23 Virtual Reality Glove as EMMT

Some modern version of the virtual reality glove and other haptic products also double as direct electro-mechanical motion trackers

When you flex your fingers sensors in the glove are activated and convert that movement into electrical signals for motion tracking (see GloveOne and the Salto)

24 Acoustic Tracking

An acoustic tracking system is copied from the nature systems of positioning and orientation. Bats or dolphins navigate in space using ultrasound. Acoustic tracking measures the time during which a particular acoustic signal reaches the receiver

There are two ways to determine the position of the object: to measure time-of-flight of the sound wave from the transmitter to the receivers or the phase coherence of the sinusoidal sound wave by receiving the transfer

25 Mechanical Systems for Tracking

They are based on the direct measurement of the object’s orientation with electromechanical potentiometers

These systems have no external forces or occlusion problems, so results effective in some cases because any of them cannot affect the measurement of the object’s orientation. In addition to this, their operation is quick, and they are portable

26 Myo Armband for Tracking

The Myo interprets electrical impulses from the muscles in the forearm allowing for gesture-based controls of computer software

27 as Tracking System

The Leap Motion is another example of a marker-free tracking system, but rather than full body tracking the Leap Motion creates high-resolution real-time scans of objects in close proximity to it

In virtual reality contexts, the device can be attached to the front of an HMD and provide precisely digitized versions of the user’s hands, which then allows for natural interaction

28 Inertial Tracking Devices

Inertial tracking devices are based on accelerometers and gyroscopes

The inertial sensor is placed in the object to be tracked to measure its orientation. With estimation techniques fusing angular rate and tilt (gravity vector) and, with some magnetometers, to get a better heading data

29 Inertial Tracking in Phones and Standalone Headsets

Phones are one of the more basic viewing options for VR content and have been providing a super cheap yet still interactive solution for several years now

The phone is put inside a headset, and the screen is split to give output for each eye, to give the illusion of depth

The and systems in the smartphones and headsets of the last two decades give virtual reality applications a true sense of motion

30 Simultaneous Localization and Mapping (SLAM)

SLAM is a method used for autonomous vehicles that lets you build a map and localize your vehicle in that map at the same time

SLAM algorithms allow the vehicle to map out unknown environments. Engineers use the map information to carry out tasks such as path planning and obstacle avoidance

SLAM is a chicken-or-egg problem:

a map is needed for localization and

a pose estimate is needed for mapping

31 SLAM Applications

SLAM is central to a range of indoor, outdoor, in-air and underwater applications for both manned and autonomous vehicles

Examples:

At home: vacuum cleaner, lawn mower

Air: surveillance with unmanned air vehicles

Underwater: reef monitoring

Underground: exploration of mines

Space: terrain mapping for localization 32 SLAM as Markerless Tracking Method

As a typical markerless tracking method, SLAM can perceive the 6-DoF pose of a moving camera within an unprepared environment, enabling mobile AR/VR applications without fiducial markers

Facebook detailed its first generation of the SLAM system with direct reference to ORB-SLAM, SVO, and LSD SLAM https://engineering.fb.com/virtual-reality/slam-bringing-art- to-life-through-technology/

Oculus solved the problem using Visual SLAM https://ai.facebook.com/blog/powered-by-ai-oculus-insight/ 33 SLAM in VR/AR

34 35 How is it that the VR environment being viewed moves when your head is moved? The accelerometer works by measuring the rate of change in your movement. Tiny sensors translate the forces into readable electrical signals. The accelerometer, however, cannot detect the phones position relative to everything else

The gyroscope can measure rotational movements relative to a neutral position. These more mechanically oriented systems are also used in the higher end headsets to improve the fluidity of the experience

36 HTC Lighthouse Systems

Ships used lighthouses or towers in order to judge their position relatively. This is the same principle for Valve’s Lighthouse Base stations used for the HTC Vive

The technology emits non-visible light to flood a room and using the reference points on the HMD’s and handhelds to give positional information back to the computer

This is done using both stationary LED’s and two active laser emitters spinning at 60Hz (32 photosensors in the HMD and 24 per controller). With multiple sensors it creates a “pose” (with an exact position and direction its facing) 37 Oculus Tracking - Oculus Rift

The Oculus Rift and Rift S, Nintendo Wii Controllers, use a similar “pose” recognition principle

The headset for the Rift uses constellations of infrared LED’s built into the HMD and controllers. These are subsequently picked up by the two desktop sensors and convert their placement into positional data

There is also a magnetometer, gyroscope and accelerometer in the headset. Combined these allow for accurate tracking across all three dimensions of the 3D world

38 Oculus Tracking - Quest

This wireless and computer-less system uses a radically different tracking method in the form of scanning a room for any objects within your space (curtains, windows, coffee tables etc.), and creating a 3D map to play in

It combines data from a gyroscope and accelerometer with the map to give the HMDs position at 1000Hz (once every millisecond)

Similarly, the Microsoft HoloLens headset uses two front mounted cameras to scan the surrounding area and in conjunction with IMU gives precise positions 39 Mechatech Tracking

Mechatech has developed a tracking system based on direct measurement of the human body

An exoskeleton frame is worn over the body, the AgileVR product is worn over the knee only, and direct measurements are taken

It doesn’t need a camera! It’s not trapped in , and it also does not suffer from occlusion – imagine putting your arm behind your back – the camera cannot see it

The pose data does not have to be calculated, it can be read straight from the device in real-time, reducing lag 40 ARKit Face Tracking https://docs.unity3d.com/Manual/com.unity.xr.arkit-face-tracking.html https://docs.unity3d.com/Packages/[email protected]/man ual/index.html

Support for face tracking on ARKit. Includes support for:

Face pose tracking

Blendshapes

Works on iOS device with front-facing TrueDepth camera (minimum iPhone X) https://www.oxfordechoes.com/ios-arkit-face-tracking-vertices/ 41 The Future of VR Tracking

The future of tracking has already begun developing with being taken from the medical and aerospace fields and being applied within HMDs for determining what you are focusing on. This could provide amazing updates to peoples set up such as retinal scanning for security or eye strain measurement for game time efficiencies

Hand scanning has also been talked about and Facebook have stated “the future for this technology is in all-day wearable AR glasses that are spatially aware”

42 Hand Tracking in Unity

The hand tracking feature enables the use of hands as an input method for the device

It delivers a new sense of presence, enhances social engagement, and delivers more natural interactions with fully tracked hands and articulated fingers. Integrated hands can perform object interactions by using simple hand gestures such as pinch, unpinch, and pinch and hold.

43 Motion Tracking in Blender

MT is used to track the motion of objects and/or a camera and, through the constraints, to apply this tracking data to 3D objects

Blender’s MT supports a couple of very powerful tools for 2D tracking and 3D motion reconstruction, including camera tracking and object tracking, as well as some special features like the plane track for compositing

Tracks can also be used to move and deform masks for rotoscoping in the Mask Editor, which is available as a special mode in the Movie Clip Editor

Extension vfx means visual effects

44 Blender - Motion Tracking

Views

Manual Lens Calibration

Camera and Object Motion Solving

Tools for Scene Orientation and Stabilization

Clip View

Graph View

Dope Sheet View 45 Blender - MT - Views

In Tracking Mode there are three different views available. You can toggle between view modes using the View dropdown, which is located in the header. When you selected a view in the whole area of the Movie Clip editor will change. Hence, to display a curve or dope sheet view, the editor must be split into two, with one switched to the curve or dope sheet view.

46 Blender - MT - Manual Lens Calibration

All cameras record distorted video (because this is the manner in which optical lenses work). For accurate camera motion, the exact value of the focal length and the “strength” of distortion are needed

Currently, focal length can be automatically obtained only from the camera’s settings or from the EXIF information. Within Blender you can use the Annotation tool for this

To calibrate your camera more accurately, use the Grid calibration tool from OpenCV 47 Blender - MT - Camera and Object Motion Solving

Blender not only supports the solving of camera motion, including tripod shots, but also the solving of object motion in relation to the motion of the camera. In addition to that there is the Plane Track, which solves the motion of all markers on one plane https://vimeo.com/79032879

48 Blender - MT - Tools for Scene Orientation and Stabilization

After solve, you need to orient the real scene in the 3D scene for more convenient compositing. There are tools to define the floor, the scene origin, and the X/Y axes to perform scene orientation

Sometimes, the video footage includes spurious jumps and tilting movements, like e.g. when using a hand-held camera. Based on some tracked image elements, the 2D Stabilization is able to detect and compensate such movements to improve the quality of the final result https://www.youtube.com/watch?v=x6hAIG49_zY&app=desktop

49 Blender - MT - Clip View

The Clip View is used is the main part of the of the Movie Clip editor. Almost all motion tracking tools are concentrated in the Movie Clip Editor

The camera solver consists of three quite separate steps:

1. 2D tracking of footage 2. Camera intrinsics (focal length, distortion coefficients) specification/estimation/calibration 3. Solving camera, scene orientation, and scene reconstruction

50 Blender - Clip View - Tracking Marker

Point - S scales the whole marker. The whole pattern area only will be scaled by pressing S twice; The Pattern can also be rotated using the R key which, depending on the used pivot point

Plane - The left bottom corner of the plane does have X/Y axis (X is red, Y is green) to help distinguishing orientation of the plane in space

51 Blender - MT - Graph View

The graph or curves view has numerous purposes based on the color of the lines. The red and green lines on the graph show you the speed of the trackers at a given frame

Green is vertical movement, Red is horizontal. Therefore the first frames will always be at zero

The blue line is the line that comes out when you click on the film strip is the average per-frame error. This curve is available only after pressing camera solve and is not editable. This is the one line that you want to be as flat as possible and as closer to zero as you can. The high points will show you where in your shot you are having inaccurate tracking 52 Blender - MT - Dope Sheet View

The Dope Sheet View is used to visualize motion tracking data, it is implemented as separate view of the Movie Clip editor just like the Graph View

It displays channels for selected tracks and each channel visualizes tracked segments of tracks as dark bars and keyframed positions of tracks as small diamonds

53 Blender - Tracking & Masking

Masks can be used in a motion tracking workflow to mask out, or influence a particular object in the footage https://www.youtube.com/watch?v=EusYVH5RRss

Mask data-block containing multiple mask layers and splines. They are the most high-level entities used for masking purposes

54 Blender - Movie Clip Editor

The Movie Clip Editor has two main purposes, it can be used for tracking or masking movies

55 3D SLAM Reconstruction in Blender

3D SLAM is a technique for scanning 3D environments in real-time and combining the data with GPS or other location info to build up a 3D map of an area

Google Code page for open source 3DSLAM library: http://code.google.com/p/3dslam/

Blender SLAM Models: https://www.turbosquid.com/Search/3D-Models/slam/blend

56 Conclusions

Motion Tracking deals with Position Tracking, Degree of Freedom (3DoF, 6DoF)

Tracking Systems include optical, mechanical, magnetic, acoustic systems (eye and face recognition and hand recognition)

Movie Clip Editor from Blender allows us to track and to mask the movies

57 Questions

What represents 3DoF and 6DoF?

Give examples of Tracking Systems.

What we can do with masking in Blender?

58 Bibliography

● Gumster, v. J. (2015) Blender for Dummies, 3rd Edition http://2.droppdf.com/files/5IzWw/blender-for-dummies-3rd-edition.pdf ● Burgard, W., Stachniss, C., Arras, K., Bennewitz, M. (2020) Introduction to Mobile Robotics. SLAM: Simultaneous Localization and Mapping. http://ais.informatik.uni-freiburg.de/teaching/ss12/robotics/slides/12-sl am.pdf ● Huang, Y. (2014) SLAM, Visual Odometry, Structure from Motion, Multiple View Stereo https://www.slideshare.net/yuhuang/visual-slam-structure-from-motio n-multiple-view-stereo

59 Links

● An Introduction to Positional Tracking and Degrees of Freedom (DOF) https://www.roadtovr.com/introduction-positional-tracking-degrees-freedom-dof/ ● About creating budget stereoscopic images on fingers (stereogram, anaglyph, stereoscope) https://sudonull.com/post/1156-About-creating-budget-stereoscopic-images-on-fingers-stereogram-anaglyph-stere oscope ● How do common virtual reality tracking systems work? https://www.mechatech.co.uk/journal/how-do-common-virtual-reality-tracking-systems-work ● 3 Dimensional Space and Virtual Reality, Understanding X, Y, Z Coordinates https://learnvr.org/3-dimensional-space-and-virtual-reality-understanding-x-y-z-coordinates/ ● AgileVR https://www.mechatech.co.uk/agilevr ● Tracking Systems in Virtual Reality: which is the best choice? https://3dcoil.grupopremo.com/blog/tracking-systems-virtual-reality-the-best-choice/ ● Unity Hand Tracking: https://developer.oculus.com/documentation/unity/unity-handtracking/ ● Position Tracking in VR https://4experience.co/vr-tracking-meet-degrees-of-freedom/ ● VR positional tracking https://en.wikipedia.org/wiki/VR_positional_tracking#Acoustic_Tracking ● Blender - Motion Tracking https://docs.blender.org/manual/en/latest/movie_clip/tracking/index.html

● What Is SLAM? 3 things you need to know https://www.mathworks.com/discovery/slam.html 60 ●