Introduction to mixed realities Course 03 Content
● Recapitulation Course 2 ○ 3D Modeling, Blender, Unity ● Motion Tracking ○ How tracking works ○ Position Tracking ○ Tracking Systems ○ Motion Tracking in Blender ● Conclusions ● Bibliography
2 R - Basic Elements
● 3D Modelling - Polygon Modeling - Basic elements, Shaders, Textures, UV Mapping, Rigging
3 R - Basic Elements
● Blender ○ Scene Collection ○ Add ○ Shading
4 Interface to the Virtual World Input
VR systems require not just a means for the user to tell the system what they want but a means to track at least some part of their body
User monitoring includes the continuous tracking of both user movements and user-initiated actions, such as pressing a button or issuing a voice command to the system
Continuous tracking of the user's movements is what allows the system to render and display the virtual world from a user-centric perspective-providing the effect of physical immersion
5 User Monitoring in VR/AR
In user monitoring there are active ways the user inputs information into the system; these can include the use of spoken commands, physical controls like wands, joysticks, steering wheels, dashboards, and keyboards, props, and platforms
In user monitoring, there are also passive ways in which information is supplied to the computer. Passive methods tell the computer how and where the participant is moving and where they are looking. These methods include tracking of the body (including hands, eyes, and feet) and position tracking that tells the computer the participant's location and orientation
6 Motion Tracking
Tracking or motion tracking can be defined as the process aiming to capture, follow, and get information about an object’s orientation and position, to be transferred to an application for further processing
This kind of tracking is used in VR to control the position of the user, joysticks and other important objects, whereas Augmented reality is used to track the object or market to be augmented and control the user’s position and orientation
7 Motion Tracking - History
Muybridge in 1878 used photographic techniques to quantify patterns of human movement
Motion tracking has been present in our lives since the 1970s, when they were used for the first time as a photogrammetric analysis tool in researches related to biomechanics
8 Position Tracking in VR
Being in a virtual reality simulation is like entering another world, but true immersion comes when you are able to move within this world
Degrees of Freedom also known as DoF, which usually comes in 3DoF and 6DoF flavors
9 Degrees of Freedom (DoF)
DoF refers to the movement of a rigid body inside space. It could be explained as “different basic ways in which an object can move”
All objects around you that can move in 3D space move in 6 ways: 3 directional axes and 3 rotational axes. Each of these axes is a DoF. We live in a 3D world and interact it in all 6 DoF
10 DoF - Movements
Translation Movement - A body is free to translate in 3 degrees of freedom: forward/back, up/down, left/right
Rotation Movement - A body can also rotate with 3 degrees of freedom: pitch, yaw, and roll
11 DoF - Examples
Elevator is constrained to 1 DOF (a vertical translation), but inside this DOF it can freely move up or down
Ferris wheel is constrained to 1 DOF (a rotational DOF), where it can roll one way or the opposite way
A bumper car has 3 DOF in total: it can translate in only 2 of the 3 axis (it cannot move up and down like an elevator), and it can rotate in only one way (it cannot change its pitch or roll like an airplane). So 2 translations + 1 rotation makes 3 DOF
12 3 DoF in VR
In VR, DoF is used to describe axes that are being tracked. Tracking comes from the ability to monitor a change of angle or distance on the axes by using hardware
This exists in mobile and standalone VR headsets like the new Oculus Go. If you turn your head while wearing a headset, it is able to track the angle change of this axes and allows you to look around in the environment
13 Mobile VR Headsets
Oculus Go, Samsung Gear VR, and Google Daydream View only have rotational tracking (3DoF)
You are able to look up or down, to either side or tilt your head. But if you try to lean or actually move your head’s position, this is not tracked. The entire virtual world will move with you
14 IMUs and DOF
An Inertial Measurement Unit (also called an IMU or sometimes ‘tracker’), is an electronic device that measures and reports velocity, orientation, and gravitational forces, using a combination of sensors (accelerometers, gyroscopes and magnetometers)
IMUs in the past was used as aircraft instrumentation, but nowadays they are used in all sorts of devices, including mobile phones
15 What is Positional Tracking and Why is It So Important for VR?
Positional tracking is a mix of hardware and software which is able to detect the absolute position of an object
In VR, in combination with orientation tracking it becomes possible to measure and report all the 6 DOF of real-life
The Oculus Rift provides 3 DOF (rotational) head-tracking which is not positional. But, you won’t be able to lean down to inspect objects on the floor or lean around corners because there’s no way to tell exactly where you head is
16 Parallax
Positional tracking can also help improve the 3D perception of the virtual environment thanks to an effect known as ‘parallax’
Parallax is the way that objects that are closer to our eyes move faster than objects far from our eyes. This effect helps our brains to perceive distance in conjunction with stereoscopy (the difference between what our left and right eyes see)
Johnny Lee uses the wiimote to shows how parallax helps us to perceive 3D https://www.youtube.com/watch?v=Jd3-eiid-Uw
17 How do common virtual reality tracking systems work?
Tracking systems typically consist of a device capable of creating a signal and a sensor capable of reading it
These can take on various forms including optical, electromagnetic and acoustic signals, mechanical and inertial systems
18 Tracking Systems
In optical systems emitted light is captured by cameras in various formats
Electromagnetic tracking can be used where small electrified coils affect other electromagnetic sensors, and how their magnetic field affects others can position them in space
Acoustic systems use ultrasonic sound waves to identify the position and orientation of target objects
Mechanical tracking can use articulated arms/limbs/joysticks/sensors connected to headsets or inside them, much like the inertial tracking in phones often made possible by accelerometers and gyroscopes 19 VR Tracking with Optics
This method usually use cameras of one sort or another connected to a computer
Usually, the person being tracked has optical markers (dots of highly reflective material) on certain known points of their body or on the equipment such as the HMD or handheld controllers
When a camera installation capable of calculating depth sees a marker it can map it to 3D space. For example, two cameras at known angles that both see the same dot allow for this mathematical calculation
20 Passive and Active Markers
Reflective markers are known as passive markers as they reflect light
Active marker are computer controlled LEDs that allow for more accuracy and various workarounds for the weaknesses of passive marker methods. LEDs may be different colors or flash rapidly in sync with the capture system
Active marker systems perform better than passive ones, but the user has to wear a power supply or be tethered to the system somehow, which is clearly an encumbrance 21 Markerless Optical Systems
The principle of these markerless option is to avoid the use of extra elements in the detection of the object in motion, by means of an analysis based on image capture, segmentation and processing to extract the position of the object to be tracked
It is common to use techniques based on the background subtraction and multiple learning algorithms
22 Electromagnetic Systems for Tracking
These systems are based on electromagnetism theories, which describe the interaction between magnetic fields and electricity
The electromagnetic tracking system can calculate accurately and in real time the object position and orientation in the environment
23 Virtual Reality Glove as EMMT
Some modern version of the virtual reality glove and other haptic products also double as direct electro-mechanical motion trackers
When you flex your fingers sensors in the glove are activated and convert that movement into electrical signals for motion tracking (see GloveOne and the Salto)
24 Acoustic Tracking
An acoustic tracking system is copied from the nature systems of positioning and orientation. Bats or dolphins navigate in space using ultrasound. Acoustic tracking measures the time during which a particular acoustic signal reaches the receiver
There are two ways to determine the position of the object: to measure time-of-flight of the sound wave from the transmitter to the receivers or the phase coherence of the sinusoidal sound wave by receiving the transfer
25 Mechanical Systems for Tracking
They are based on the direct measurement of the object’s orientation with electromechanical potentiometers
These systems have no external forces or occlusion problems, so results effective in some cases because any of them cannot affect the measurement of the object’s orientation. In addition to this, their operation is quick, and they are portable
26 Myo Armband for Tracking
The Myo interprets electrical impulses from the muscles in the forearm allowing for gesture-based controls of computer software
27 Leap Motion as Tracking System
The Leap Motion is another example of a marker-free tracking system, but rather than full body tracking the Leap Motion creates high-resolution real-time scans of objects in close proximity to it
In virtual reality contexts, the device can be attached to the front of an HMD and provide precisely digitized versions of the user’s hands, which then allows for natural interaction
28 Inertial Tracking Devices
Inertial tracking devices are based on accelerometers and gyroscopes
The inertial sensor is placed in the object to be tracked to measure its orientation. With estimation techniques fusing angular rate and tilt (gravity vector) and, with some magnetometers, to get a better heading data
29 Inertial Tracking in Phones and Standalone Headsets
Phones are one of the more basic viewing options for VR content and have been providing a super cheap yet still interactive solution for several years now
The phone is put inside a headset, and the screen is split to give output for each eye, to give the illusion of depth
The accelerometer and gyroscope systems in the smartphones and headsets of the last two decades give virtual reality applications a true sense of motion
30 Simultaneous Localization and Mapping (SLAM)
SLAM is a method used for autonomous vehicles that lets you build a map and localize your vehicle in that map at the same time
SLAM algorithms allow the vehicle to map out unknown environments. Engineers use the map information to carry out tasks such as path planning and obstacle avoidance
SLAM is a chicken-or-egg problem:
a map is needed for localization and
a pose estimate is needed for mapping
31 SLAM Applications
SLAM is central to a range of indoor, outdoor, in-air and underwater applications for both manned and autonomous vehicles
Examples:
At home: vacuum cleaner, lawn mower
Air: surveillance with unmanned air vehicles
Underwater: reef monitoring
Underground: exploration of mines
Space: terrain mapping for localization 32 SLAM as Markerless Tracking Method
As a typical markerless tracking method, SLAM can perceive the 6-DoF pose of a moving camera within an unprepared environment, enabling mobile AR/VR applications without fiducial markers
Facebook detailed its first generation of the SLAM system with direct reference to ORB-SLAM, SVO, and LSD SLAM https://engineering.fb.com/virtual-reality/slam-bringing-art- to-life-through-technology/
Oculus solved the problem using Visual SLAM https://ai.facebook.com/blog/powered-by-ai-oculus-insight/ 33 SLAM in VR/AR
34 35 How is it that the VR environment being viewed moves when your head is moved? The accelerometer works by measuring the rate of change in your movement. Tiny sensors translate the forces into readable electrical signals. The accelerometer, however, cannot detect the phones position relative to everything else
The gyroscope can measure rotational movements relative to a neutral position. These more mechanically oriented systems are also used in the higher end headsets to improve the fluidity of the experience
36 HTC Lighthouse Systems
Ships used lighthouses or towers in order to judge their position relatively. This is the same principle for Valve’s Lighthouse Base stations used for the HTC Vive
The technology emits non-visible light to flood a room and using the reference points on the HMD’s and handhelds to give positional information back to the computer
This is done using both stationary LED’s and two active laser emitters spinning at 60Hz (32 photosensors in the HMD and 24 per controller). With multiple sensors it creates a “pose” (with an exact position and direction its facing) 37 Oculus Tracking - Oculus Rift
The Oculus Rift and Rift S, Nintendo Wii Controllers, use a similar “pose” recognition principle
The headset for the Rift uses constellations of infrared LED’s built into the HMD and controllers. These are subsequently picked up by the two desktop sensors and convert their placement into positional data
There is also a magnetometer, gyroscope and accelerometer in the headset. Combined these allow for accurate tracking across all three dimensions of the 3D world
38 Oculus Tracking - Quest
This wireless and computer-less system uses a radically different tracking method in the form of scanning a room for any objects within your space (curtains, windows, coffee tables etc.), and creating a 3D map to play in
It combines data from a gyroscope and accelerometer with the map to give the HMDs position at 1000Hz (once every millisecond)
Similarly, the Microsoft HoloLens headset uses two front mounted cameras to scan the surrounding area and in conjunction with IMU gives precise positions 39 Mechatech Tracking
Mechatech has developed a tracking system based on direct measurement of the human body
An exoskeleton frame is worn over the body, the AgileVR product is worn over the knee only, and direct measurements are taken
It doesn’t need a camera! It’s not trapped in room scale, and it also does not suffer from occlusion – imagine putting your arm behind your back – the camera cannot see it
The pose data does not have to be calculated, it can be read straight from the device in real-time, reducing lag 40 ARKit Face Tracking https://docs.unity3d.com/Manual/com.unity.xr.arkit-face-tracking.html https://docs.unity3d.com/Packages/[email protected]/man ual/index.html
Support for face tracking on ARKit. Includes support for:
Face pose tracking
Blendshapes
Works on iOS device with front-facing TrueDepth camera (minimum iPhone X) https://www.oxfordechoes.com/ios-arkit-face-tracking-vertices/ 41 The Future of VR Tracking
The future of tracking has already begun developing with eye tracking being taken from the medical and aerospace fields and being applied within HMDs for determining what you are focusing on. This could provide amazing updates to peoples set up such as retinal scanning for security or eye strain measurement for game time efficiencies
Hand scanning has also been talked about and Facebook have stated “the future for this technology is in all-day wearable AR glasses that are spatially aware”
42 Hand Tracking in Unity
The hand tracking feature enables the use of hands as an input method for the Oculus Quest device
It delivers a new sense of presence, enhances social engagement, and delivers more natural interactions with fully tracked hands and articulated fingers. Integrated hands can perform object interactions by using simple hand gestures such as pinch, unpinch, and pinch and hold.
43 Motion Tracking in Blender
MT is used to track the motion of objects and/or a camera and, through the constraints, to apply this tracking data to 3D objects
Blender’s MT supports a couple of very powerful tools for 2D tracking and 3D motion reconstruction, including camera tracking and object tracking, as well as some special features like the plane track for compositing
Tracks can also be used to move and deform masks for rotoscoping in the Mask Editor, which is available as a special mode in the Movie Clip Editor
Extension vfx means visual effects
44 Blender - Motion Tracking
Views
Manual Lens Calibration
Camera and Object Motion Solving
Tools for Scene Orientation and Stabilization
Clip View
Graph View
Dope Sheet View 45 Blender - MT - Views
In Tracking Mode there are three different views available. You can toggle between view modes using the View dropdown, which is located in the header. When you selected a view in the whole area of the Movie Clip editor will change. Hence, to display a curve or dope sheet view, the editor must be split into two, with one switched to the curve or dope sheet view.
46 Blender - MT - Manual Lens Calibration
All cameras record distorted video (because this is the manner in which optical lenses work). For accurate camera motion, the exact value of the focal length and the “strength” of distortion are needed
Currently, focal length can be automatically obtained only from the camera’s settings or from the EXIF information. Within Blender you can use the Annotation tool for this
To calibrate your camera more accurately, use the Grid calibration tool from OpenCV 47 Blender - MT - Camera and Object Motion Solving
Blender not only supports the solving of camera motion, including tripod shots, but also the solving of object motion in relation to the motion of the camera. In addition to that there is the Plane Track, which solves the motion of all markers on one plane https://vimeo.com/79032879
48 Blender - MT - Tools for Scene Orientation and Stabilization
After solve, you need to orient the real scene in the 3D scene for more convenient compositing. There are tools to define the floor, the scene origin, and the X/Y axes to perform scene orientation
Sometimes, the video footage includes spurious jumps and tilting movements, like e.g. when using a hand-held camera. Based on some tracked image elements, the 2D Stabilization is able to detect and compensate such movements to improve the quality of the final result https://www.youtube.com/watch?v=x6hAIG49_zY&app=desktop
49 Blender - MT - Clip View
The Clip View is used is the main part of the of the Movie Clip editor. Almost all motion tracking tools are concentrated in the Movie Clip Editor
The camera solver consists of three quite separate steps:
1. 2D tracking of footage 2. Camera intrinsics (focal length, distortion coefficients) specification/estimation/calibration 3. Solving camera, scene orientation, and scene reconstruction
50 Blender - Clip View - Tracking Marker
Point - S scales the whole marker. The whole pattern area only will be scaled by pressing S twice; The Pattern can also be rotated using the R key which, depending on the used pivot point
Plane - The left bottom corner of the plane does have X/Y axis (X is red, Y is green) to help distinguishing orientation of the plane in space
51 Blender - MT - Graph View
The graph or curves view has numerous purposes based on the color of the lines. The red and green lines on the graph show you the speed of the trackers at a given frame
Green is vertical movement, Red is horizontal. Therefore the first frames will always be at zero
The blue line is the line that comes out when you click on the film strip is the average per-frame error. This curve is available only after pressing camera solve and is not editable. This is the one line that you want to be as flat as possible and as closer to zero as you can. The high points will show you where in your shot you are having inaccurate tracking 52 Blender - MT - Dope Sheet View
The Dope Sheet View is used to visualize motion tracking data, it is implemented as separate view of the Movie Clip editor just like the Graph View
It displays channels for selected tracks and each channel visualizes tracked segments of tracks as dark bars and keyframed positions of tracks as small diamonds
53 Blender - Tracking & Masking
Masks can be used in a motion tracking workflow to mask out, or influence a particular object in the footage https://www.youtube.com/watch?v=EusYVH5RRss
Mask data-block containing multiple mask layers and splines. They are the most high-level entities used for masking purposes
54 Blender - Movie Clip Editor
The Movie Clip Editor has two main purposes, it can be used for tracking or masking movies
55 3D SLAM Reconstruction in Blender
3D SLAM is a technique for scanning 3D environments in real-time and combining the data with GPS or other location info to build up a 3D map of an area
Google Code page for open source 3DSLAM library: http://code.google.com/p/3dslam/
Blender SLAM Models: https://www.turbosquid.com/Search/3D-Models/slam/blend
56 Conclusions
Motion Tracking deals with Position Tracking, Degree of Freedom (3DoF, 6DoF)
Tracking Systems include optical, mechanical, magnetic, acoustic systems (eye and face recognition and hand recognition)
Movie Clip Editor from Blender allows us to track and to mask the movies
57 Questions
What represents 3DoF and 6DoF?
Give examples of Tracking Systems.
What we can do with masking in Blender?
58 Bibliography
● Gumster, v. J. (2015) Blender for Dummies, 3rd Edition http://2.droppdf.com/files/5IzWw/blender-for-dummies-3rd-edition.pdf ● Burgard, W., Stachniss, C., Arras, K., Bennewitz, M. (2020) Introduction to Mobile Robotics. SLAM: Simultaneous Localization and Mapping. http://ais.informatik.uni-freiburg.de/teaching/ss12/robotics/slides/12-sl am.pdf ● Huang, Y. (2014) SLAM, Visual Odometry, Structure from Motion, Multiple View Stereo https://www.slideshare.net/yuhuang/visual-slam-structure-from-motio n-multiple-view-stereo
59 Links
● An Introduction to Positional Tracking and Degrees of Freedom (DOF) https://www.roadtovr.com/introduction-positional-tracking-degrees-freedom-dof/ ● About creating budget stereoscopic images on fingers (stereogram, anaglyph, stereoscope) https://sudonull.com/post/1156-About-creating-budget-stereoscopic-images-on-fingers-stereogram-anaglyph-stere oscope ● How do common virtual reality tracking systems work? https://www.mechatech.co.uk/journal/how-do-common-virtual-reality-tracking-systems-work ● 3 Dimensional Space and Virtual Reality, Understanding X, Y, Z Coordinates https://learnvr.org/3-dimensional-space-and-virtual-reality-understanding-x-y-z-coordinates/ ● AgileVR https://www.mechatech.co.uk/agilevr ● Tracking Systems in Virtual Reality: which is the best choice? https://3dcoil.grupopremo.com/blog/tracking-systems-virtual-reality-the-best-choice/ ● Unity Hand Tracking: https://developer.oculus.com/documentation/unity/unity-handtracking/ ● Position Tracking in VR https://4experience.co/vr-tracking-meet-degrees-of-freedom/ ● VR positional tracking https://en.wikipedia.org/wiki/VR_positional_tracking#Acoustic_Tracking ● Blender - Motion Tracking https://docs.blender.org/manual/en/latest/movie_clip/tracking/index.html
● What Is SLAM? 3 things you need to know https://www.mathworks.com/discovery/slam.html 60 ●