Introduction to mixed realities Course 03 Content ● Recapitulation Course 2 ○ 3D Modeling, Blender, Unity ● Motion Tracking ○ How tracking works ○ Position Tracking ○ Tracking Systems ○ Motion Tracking in Blender ● Conclusions ● Bibliography 2 R - Basic Elements ● 3D Modelling - Polygon Modeling - Basic elements, Shaders, Textures, UV Mapping, Rigging 3 R - Basic Elements ● Blender ○ Scene Collection ○ Add ○ Shading 4 Interface to the Virtual World Input VR systems require not just a means for the user to tell the system what they want but a means to track at least some part of their body User monitoring includes the continuous tracking of both user movements and user-initiated actions, such as pressing a button or issuing a voice command to the system Continuous tracking of the user's movements is what allows the system to render and display the virtual world from a user-centric perspective-providing the effect of physical immersion 5 User Monitoring in VR/AR In user monitoring there are active ways the user inputs information into the system; these can include the use of spoken commands, physical controls like wands, joysticks, steering wheels, dashboards, and keyboards, props, and platforms In user monitoring, there are also passive ways in which information is supplied to the computer. Passive methods tell the computer how and where the participant is moving and where they are looking. These methods include tracking of the body (including hands, eyes, and feet) and position tracking that tells the computer the participant's location and orientation 6 Motion Tracking Tracking or motion tracking can be defined as the process aiming to capture, follow, and get information about an object’s orientation and position, to be transferred to an application for further processing This kind of tracking is used in VR to control the position of the user, joysticks and other important objects, whereas Augmented reality is used to track the object or market to be augmented and control the user’s position and orientation 7 Motion Tracking - History Muybridge in 1878 used photographic techniques to quantify patterns of human movement Motion tracking has been present in our lives since the 1970s, when they were used for the first time as a photogrammetric analysis tool in researches related to biomechanics 8 Position Tracking in VR Being in a virtual reality simulation is like entering another world, but true immersion comes when you are able to move within this world Degrees of Freedom also known as DoF, which usually comes in 3DoF and 6DoF flavors 9 Degrees of Freedom (DoF) DoF refers to the movement of a rigid body inside space. It could be explained as “different basic ways in which an object can move” All objects around you that can move in 3D space move in 6 ways: 3 directional axes and 3 rotational axes. Each of these axes is a DoF. We live in a 3D world and interact it in all 6 DoF 10 DoF - Movements Translation Movement - A body is free to translate in 3 degrees of freedom: forward/back, up/down, left/right Rotation Movement - A body can also rotate with 3 degrees of freedom: pitch, yaw, and roll 11 DoF - Examples Elevator is constrained to 1 DOF (a vertical translation), but inside this DOF it can freely move up or down Ferris wheel is constrained to 1 DOF (a rotational DOF), where it can roll one way or the opposite way A bumper car has 3 DOF in total: it can translate in only 2 of the 3 axis (it cannot move up and down like an elevator), and it can rotate in only one way (it cannot change its pitch or roll like an airplane). So 2 translations + 1 rotation makes 3 DOF 12 3 DoF in VR In VR, DoF is used to describe axes that are being tracked. Tracking comes from the ability to monitor a change of angle or distance on the axes by using hardware This exists in mobile and standalone VR headsets like the new Oculus Go. If you turn your head while wearing a headset, it is able to track the angle change of this axes and allows you to look around in the environment 13 Mobile VR Headsets Oculus Go, Samsung Gear VR, and Google Daydream View only have rotational tracking (3DoF) You are able to look up or down, to either side or tilt your head. But if you try to lean or actually move your head’s position, this is not tracked. The entire virtual world will move with you 14 IMUs and DOF An Inertial Measurement Unit (also called an IMU or sometimes ‘tracker’), is an electronic device that measures and reports velocity, orientation, and gravitational forces, using a combination of sensors (accelerometers, gyroscopes and magnetometers) IMUs in the past was used as aircraft instrumentation, but nowadays they are used in all sorts of devices, including mobile phones 15 What is Positional Tracking and Why is It So Important for VR? Positional tracking is a mix of hardware and software which is able to detect the absolute position of an object In VR, in combination with orientation tracking it becomes possible to measure and report all the 6 DOF of real-life The Oculus Rift provides 3 DOF (rotational) head-tracking which is not positional. But, you won’t be able to lean down to inspect objects on the floor or lean around corners because there’s no way to tell exactly where you head is 16 Parallax Positional tracking can also help improve the 3D perception of the virtual environment thanks to an effect known as ‘parallax’ Parallax is the way that objects that are closer to our eyes move faster than objects far from our eyes. This effect helps our brains to perceive distance in conjunction with stereoscopy (the difference between what our left and right eyes see) Johnny Lee uses the wiimote to shows how parallax helps us to perceive 3D https://www.youtube.com/watch?v=Jd3-eiid-Uw 17 How do common virtual reality tracking systems work? Tracking systems typically consist of a device capable of creating a signal and a sensor capable of reading it These can take on various forms including optical, electromagnetic and acoustic signals, mechanical and inertial systems 18 Tracking Systems In optical systems emitted light is captured by cameras in various formats Electromagnetic tracking can be used where small electrified coils affect other electromagnetic sensors, and how their magnetic field affects others can position them in space Acoustic systems use ultrasonic sound waves to identify the position and orientation of target objects Mechanical tracking can use articulated arms/limbs/joysticks/sensors connected to headsets or inside them, much like the inertial tracking in phones often made possible by accelerometers and gyroscopes 19 VR Tracking with Optics This method usually use cameras of one sort or another connected to a computer Usually, the person being tracked has optical markers (dots of highly reflective material) on certain known points of their body or on the equipment such as the HMD or handheld controllers When a camera installation capable of calculating depth sees a marker it can map it to 3D space. For example, two cameras at known angles that both see the same dot allow for this mathematical calculation 20 Passive and Active Markers Reflective markers are known as passive markers as they reflect light Active marker are computer controlled LEDs that allow for more accuracy and various workarounds for the weaknesses of passive marker methods. LEDs may be different colors or flash rapidly in sync with the capture system Active marker systems perform better than passive ones, but the user has to wear a power supply or be tethered to the system somehow, which is clearly an encumbrance 21 Markerless Optical Systems The principle of these markerless option is to avoid the use of extra elements in the detection of the object in motion, by means of an analysis based on image capture, segmentation and processing to extract the position of the object to be tracked It is common to use techniques based on the background subtraction and multiple learning algorithms 22 Electromagnetic Systems for Tracking These systems are based on electromagnetism theories, which describe the interaction between magnetic fields and electricity The electromagnetic tracking system can calculate accurately and in real time the object position and orientation in the environment 23 Virtual Reality Glove as EMMT Some modern version of the virtual reality glove and other haptic products also double as direct electro-mechanical motion trackers When you flex your fingers sensors in the glove are activated and convert that movement into electrical signals for motion tracking (see GloveOne and the Salto) 24 Acoustic Tracking An acoustic tracking system is copied from the nature systems of positioning and orientation. Bats or dolphins navigate in space using ultrasound. Acoustic tracking measures the time during which a particular acoustic signal reaches the receiver There are two ways to determine the position of the object: to measure time-of-flight of the sound wave from the transmitter to the receivers or the phase coherence of the sinusoidal sound wave by receiving the transfer 25 Mechanical Systems for Tracking They are based on the direct measurement of the object’s orientation with electromechanical potentiometers These systems have no external forces or occlusion problems, so results effective in some cases because any of them cannot affect the measurement of the object’s orientation. In addition to this, their operation is quick, and they are portable 26 Myo Armband for Tracking The Myo interprets electrical impulses from the muscles in the forearm allowing for gesture-based controls of computer software 27 Leap Motion as Tracking System The Leap Motion is another example of a marker-free tracking system, but rather than full body tracking the Leap Motion creates high-resolution real-time scans of objects in close proximity to it In virtual reality contexts, the device can be attached to the front of an HMD and provide precisely digitized versions of the user’s hands, which then allows for natural interaction 28 Inertial Tracking Devices Inertial tracking devices are based on accelerometers and gyroscopes The inertial sensor is placed in the object to be tracked to measure its orientation.