Microsoft 2

Renato Mainetti Alessandro Tironi Jacopo Essenziale

[email protected] [email protected] [email protected] Lab 05 Outline: • Introduction and Theory • Kinect vs others NUI devices • Kinect History and how does it works? • Avateering (Rotations vs Positions) • Quaternions • Kinect Skeleton • Let’s CODE! • Kinect SDK • Kinect Unity Integration • Avateering in Unity • Kinect as NUI for our AAA game (Assembly line) • Kinect & VR

Kinect 2 - VR 2 NUI: In computing, a natural user interface is a user interface that is effectively invisible, and remains invisible as the user continuously learns increasingly complex interactions. The word natural is used because most computer interfaces use artificial control devices whose operation has to be learned.

(examples are: multitouch, speech recognition, gesture motion recognition)

Kinect 2 - VR 3 Kinect vs …

Obtrusiveness, precision, usability, etc… Kinect 1 – Hardware (Primesense)

● RGB: 640x480 @ 30fps, 1280x960 @ 12fps ● Depth: 320x240 @ 30fps ● 4 mic array (16 KHz) ● Servo for Tilt regulation

5 Kinect 1 ( 360) vs Kinect 1 (for Windows) The windows version has introduced: • Near mode • Sitting position • RGB cam exposure etc… • Different use licenses

6 How does it work? V1 Proprietary technology:

• Structured light • Depth from focus • Stereo principles

7 Kinect 2 – Hardware (3DV Systems, Canesta)

● RGB: 1920x1080 @ 30fps ● Depth & IR cam: 512x424 @ 30fps ● 4 mic array (48 KHz) ● USB 3.0

8 How does it work? V2 TOF:

9 Kinect 1 Skeleton:

20 joints 2 bodies

10 Kinect 2 Skeleton:

11 Skeleton - demo

12 Colour and Depth demo (alignment)

13 Face tracking demo

Orientations Expression etc

14 Kinect HeartRate:

15 Audio direction demo

16 Speech recognition demo

SIT - ROLL - BARK

17 3D scanning (and retopolgy)

18 Kinect Studio

19 Mocap / Avateering

• Brekel PRO body V2

20 Avateering (Why rotation are so important)

The right approach is to apply the rotation evaluated to the avatar skeleton.

If we try to translate the avatar’s joints to the Kinect’s joint position then the mesh could get strechtched, depending how the skeleton was made and skinned to the mesh.

21 Quaternion:

• Generalization of complex numbers • Very powerful to describe rotation and orientation • It doesn’t suffer of the gimbal lock problem • Excellent for interpolation

22 Kinect’s rotations

For each frame, Kinect 2 returns the orientation quaternion of each joint (leaf joints excluded).

Each quaternion represents the absolute rotation of the parent bone.(e.g: the ElbowRight quaternion represents the right arm rotation)

The Y axis is always parallel to the bone

23 Microsoft Kinect SDK 2.0

The SDK gives you the ability to access all the Kinect’s features (frame by frame):

• Color sensor (RGB image)

• Depth sensor (Float depth image)

• Audio samples

• Body tracking (Joint Pos (X,Y,Z), Bone Quaternion (X,Y,Z,W))

Other features:

• Face tracking

• Gesture Recognition

24 Sensor initialization and joint evaluation

Let’s try showing the movements of the joints in a unity scene.

Remember to deallocate each frame after using it. (in c# use the using structure)

It is possible to retrieve the 3D coordinates using the Body.Joints dictionary:

Pay attention to the reference axis system! (Kinect is a “right-handed”)

25 Sensor initialization and shutdown

26 From positions to rotations

We show in another scene how Kinect rotations change in real time.

We switch from Kinect’s reference system to Unity’s system by reverting Z and W values for each quaternion.

If we want to apply additional rotations, we multiply the retrieved quaternion by another.

27 Avateering

Let’s import an already skinned avatar from Blender and apply rotations to joints.

If we correctly polish rotation offsets then we will see the mesh move without any deformation.

28 How to use the Kinect package inside our scene

There is a singleton KinectManager object which constantly retrieves a new frame and sends it to a set of handles (named KinectServices). Each KinectService will elaborate the new frame independently.

The avateering-related KinectService will apply received rotations to the avatar, after the required elaborations.

29 Avateering algorithm (simplified)

30 Hitting cubes

We want the avatar to prevent the red cubes from arriving to the end of the belt by punching or kicking them, so we apply a rigidbody and a sphere collider to hands and feet.

31 Hitting cubes

In order to compute the punch/kick direction we buffer the hands and feet positions from the previous frame. Whenever a collision occours, we compute the direction vector with the current position and the buffered one.

32 Suggestion for future work

KinectManager calls AcquireLatestFrame once for each game frame. This means that the game frame rate will be forced to the Kinect’s frame rate!

Possible solution: multi-threading!

33 Kinect and VR

• You can try this combination for your final project in the lab!

Kinect 2 - VR 34 What’s the future?

Orbbec: https://www.youtube.com/watch?v=RWA4cHWxTN0

RealSense: https://www.youtube.com/watch?v=r42z259-HHE

Kinect 2 - VR 35