POSTER 2016, PRAGUE MAY 24 1

Kinect: From Entertainment to Scientific Research on Virtual Movement

Juraj MIHAL’OV and Marek DUFALA

Department of Computers and Informatics, Technical University of Košice, Letná 9, 042 00 Košice, Slovak Republic

[email protected], [email protected]

Abstract. A few years ago the world has experienced which allows image-based 3D reconstruction. Altogether, the birth of a new approach to the game controlling. Al- they enable their users to achieve expected results. though the classic mouse & keyboard configuration will not be beaten so easily, the new motion sensing controllers bring the gaming to a whole another level. The purpose of this publication is to bring a general information about , 2. Kinect V1 its history and future as well as its bright and dark sides. As Clayman remembers, Kinect was presented for the first time in June 2009 at "Electronic Entertainment Expo" under the title "Project Natal" [3]. came up with Keywords the name Kinect a year later. Kinect (Fig. 1) is compatible with game consoles Kinect, motion sensors, RGB camera, depth scanning, so it was not a surprise that the first presentations regarding Xbox, Project Natal. its utilization were mainly focused on games.

1. Introduction As Luo and Yang state, Kinect is a line of motion sens- ing input devices by Microsoft for and consoles and Windows PCs [1]. It utilizes device similar to a web camera which is designed in such a way that its users are able to control this device through various ges- tures or even speech. Fig. 1. Typical Kinect V1 graphics After its release, the first three months were extremely successful for Xbox 360. Zhang states that in that period Based on Sector Portal [4], the opening variety of Kinect’s sales climbed up to more than 10 million [2] which games was focused mainly at nonplayers, who Microsoft granted Kinect a record for the Fastest-Selling Consumer wanted to lure to Xbox 360. So the audience, which had Electronics Device in the Guinness World Records. not played so far, because the controller was too difficult for them. Thanks to Kinect they just step in front of the TV and The software for Kinect was developed by , owned they can start playing without any learning. They just con- by Microsoft. The hardware consists of three main parts [1]: trol some character on the screen with their body (Fig. 2) and do whatever they want their character to do so. In the • RGB camera, beginning, the games were as simple as sports, minigames, dancing or physical exercises. The first bigger game with a • Depth sensor, story was the Harry Potter game.

• Multi-array microphone. As you can read at Microsoft’s News [5], in the begin- ning of the year 2011 a non-commercial set of tools was pub- lished (SDK – Software Development Kit). It was supposed The camera with the sensor were produced by an Israeli to bring the increasing community of scientists, developers developer at PrimeSense. They both provide Kinect with and others to Kinect for Windows to experience human mo- the ability to interpret gestures in three dimensions. This is tion tracking, depth sensing or voice and object recognition. possible because of an infrared projector, camera and a spe- cial microchip, which brings the movement tracking into the As Zhang wrote, Microsoft calls this the Kinect Effect game. Moreover, it also includes Light Coding, a 3D scanner [2]. The new Kinect technology, launched on 4 November 2 J. MIHAL’OV & M. DUFALA, KINECT: FROM ENTERTAINMENT TO SCIENTIFIC RESEARCH

2010, made such a boom, that just a month later there were Kinect observes human body as a number of joints rep- already nine pages containing brief descriptions of approx- resenting body parts such as head, neck, shoulders and arms imately 90 projects, and the number of projects posted on (Fig. 3). Every joint is defined by its 3D coordinates. Kinect KinectHacks.net has grown steadily. There were 24 pages is scanning all the joints at once in order to provide fluent on 10 February 2011, 55 pages on 2 August 2011, 63 pages interactivity which results in a good gaming experience. Mi- on 12 January 2012, and 65 pages on 18 February. crosoft has chosen the way of recognition per-pixel for dif- ferent body parts, rather than observing the whole pose in 3D at once. Microsoft is also the developer of a new DMF (De- formable Model Fitting) algorithm for 3D face recognition, which uses Kinect to regularize the mimic with the maximal probability.

Fig. 2. Example of Kinect remote control A former news site and discussion community Kinec- tHacks.net commented on Kinect’s success: Every few hours new applications are emerging for the Kinect and creating new phenomenon that is nothing short of revolutionary [2]. The first scientific contribution regarding Kinect [6] de- scribed the way of how different engineers broke into its Fig. 4. Example of face sensing functionality or gained control over its motors and sensors. Microsoft Kinect is using this technology, so one may do whatever he wants to in front of the Kinect and his avatar on the screen will imitate everything he carries out from talking or smiling to voice expressions. Using Kinect one can connect with other friends online, within a chosen environment they can meet together in order to perform ac- tivities they could do in a real life even if they are not able to meet physically. This way, one may actually see his friends’ reactions in a real time.

3. Kinect V2 As stated in [7], the fact that newer versions are pro- duced periodically indicates that the Kinect as well as SDK development still proceeds. Today, the most actual Kinect is of version 2 (V2), also called Kinect for Xbox One (Fig. 5).

According to Dado [8], with the arrival of Xbox One Fig. 3. Kinect skeletal tracking in 2014 the motion sensor started to play a lot more impor- As it continues in [2], skeletal tracking is one of the tant role. It not only serves for playing games, but one may innovations Kinect has brought to market. The demands on control the entire Xbox with it. By just saying Xbox one the such systems are relatively high. Not only they must fit the console turns on. Same happens with turning off. Kinect 2 way of how they are used, they must recognise a person from got other improvements as well, concerning the speed of the any distance or any angle. It has to take into account the moves. Thus, it recognises particular moves faster, making shape of the person, size, hair and clothing as well as the en- the response shorter, of course. Moreover, now it can sense vironment in the household such as lights, furniture or pets. up to six users at once. It can also record one’s heart-rhythm. POSTER 2016, PRAGUE MAY 24 3

As Henry et al. state in [17], Kinect uses RGB-D cam- era (Fig. 6) which captures both visual (RGB) and depth (D) information. There have been custom-build devices based on this system years ago, but just now after Kinect’s release it became popular in research areas. Now it is used mainly in gaming and entertainment, but hopefully it will be used more frequently in other areas as well. The publication [18] of Khoshelham et al. discusses more about depth sensing. Depth and colour images are cap- tured at a framerate of up to 30 fps. About 300,000 pixels in every frame provide a complete point cloud of an indoor environment in a real time. Fig. 5. Typical Kinect V2 graphics Numerous publications point out that Kinect is not a flawless depth sensor [19]. There are noises (shadow-like er- As for the Xbox One games, there is no shortage of rors) produced because of different reasons like objects too them. Right after the launch there were many titles available close or too far from Kinect, mirror-like surfaces or in con- like Battlefield 4, FIFA 14, : Ghosts, Exclusive trast the surfaces that absorb the light. Another problem oc- Ryse: Son of Rome and 3. Motorsport curs when multiple Kinects are used together. Their infrared 5 is worth mentioning as well, since it is probably the best sensors tend to intersect causing more noise. example of what Xbox One is capable of. There are a lot of applications too, e.g. YouTube, , Xbox Video, Blu-ray Kinect has already found its utilization in health-care. Player, U-play or . There was a study [20] which used Microsoft’s Kinect mo- tion sensor to develop an intelligent rehabilitation system. After many conversations with some therapists they realized that people with different physical disabilities tend to lack 4. Scientific Contributions by Kinect enthusiasm for physical exercise prescribed to them by doc- First scientific publications mentioning Kinect analyze tors. This is why they came up with an idea of using Kinect its effects [9], [10], but mainly they describe its potential through different enjoying games and activities to help these applications in augmented reality [11], [12] or they analyze people move more. The result was that people started to ex- human motion in a specific environment [13], [14]. These ercise more and to be more motivated about recovery. publications prove Kinect has had a great potential since its very beginning not only in the gaming area but also in the field of virtual reality, environment analysis or various sensor fusions based on the newest technologies. Other frequent problematics discussed publications have an evaluation character, regarding several comparisons between various sensors, including Kinect [15]. In [16] the authors performed an experiment based on a set of 13 measurements describing selected problems in gesticulation recognition. It nicely categorises gestures by visible representation into static (poses) and dynamic ges- tures. By interpretation it divides the gestures on emblems (thumb up), illustrators (pointing), regulators (hand raising Fig. 7. Surgeon interacting with computer through hand ges- at discussion), affect displays (facial expressions in combi- tures in a sterile environment nation with gestures), adaptors (adjusting glasses) etc. Another great utilization is Adora, Doctor’s Opera- tional Research Assistant created by Slovenian students [21]. The assistant is controlled by voice and hand gestures what enables surgeons to access patient data in a computer touch- less, i.e. no additional device has to be present within the sterile operating field (Fig. 7). Such a system saves a lot of time, previously spent by leaving surgery room, working on a computer but mainly spent by hygienic process when enter- ing the surgery room back again. The project was successful in 2013 during world finale of Microsoft Imagine Cup [22]. Fig. 6. Example of Kinect RGB-D image 4 J. MIHAL’OV & M. DUFALA, KINECT: FROM ENTERTAINMENT TO SCIENTIFIC RESEARCH

5. Conclusion Cited: March 5, 2016. Available at: http://news.microsoft. com/2011/06/16/microsoft-releases-kinect-for- The purpose of this publication was to analyze Kinect windows-sdk-beta-for-academics-and-enthusiasts by Microsoft, starting from its history, through current scien- [6] GILES, J. Inside the Race to Hack Microsoft’s Kinect. New Scientist, tific topics to its potentials in the future. We have skimmed 2010, vol. 208, no. 2789, p. 22–23 through the development of Kinect from the beginning to [7] PTERNEAS, V. Kinect for Windows version 2: Overview. Pub- lished: February 9, 2014. Cited: March 5, 2016. Available the present. Microsoft has improved it since the launch of at: http://www.codeproject.com/Articles/723665/ the Kinect V1 in 2009, so it is no more yet another gaming Kinect-for-Windows-version-overview console. Kinect has a huge influence on the world whether [8] Dado, M. Xbox ONE – recenzia (Xbox ONE – Overview). Pub- we take games into account or science. lished: December 18, 2013. Cited: March 5, 2016. Available at: https://www.gamesite.sk/xbox-one/recenzie/ Many scientific contributions have been published re- 17245-xbox-one-recenzia garding Kinect and we are sure there will be more of them. [9] CHIANG, O. The Xbox Moves In. Forbes, 2010, vol. 186, no. 11, p. Kinect V2 took the depth mapping or motion sensing to an 34–34 [10] DONG, Y., CONRAD, D., DeSouza, G. "Wii Using only We": Us- entirely other level. We balieve new version of Kinect will ing Background Subtraction and Human Pose Recognition to Eliminate be presented soon. The public is already curious whether Game Controllers. In IEEE International Conference on Robotics and to expect an improvement in depth sensing, useful e.g. in Automation (ICRA). Shanghai (China), 2011, p. 3887–3892 an environment modeling or motion sensing, so it will soon [11] NEWCOMBE, R. et al. KinectFusion: Real-Time Dense recognize well even the fingers or mimic features of the face. Mapping and Tracking. In IEEE/ACM International Symposium on Mixed and Augmented Reality (ISMAR). Basel (Switzerland), 2011, p. It is probable robotics will depend on Kinect-like sen- 127–136 sors in the future. Kinect-based online wardrobe [23] is al- [12] MATUSZEK, C. et al. Gambit: An Autonomous Chess-Playing Robotic System. In IEEE International Conference on Robotics and ready under development. The aim is to try clothing virtually Automation (ICRA). Shanghai (China), 2011, p. 4291–4297 outside stone shops. It is also very probable that Kinect-like [13] DUBOIS, A., DIB, A., CHARPILLET, F. Using HMMs for Discrim- technologies will be used in education as well. inating Mobile from Static Objects in a 3D Occupancy Grid. In IEEE International Conference on Tools with Artificial Intelligence (ICTAI). Streaming is a well-known technology. It is common Boca Raton (USA), 2011, p. 170–176 that one may watch a lecture from the other side of the world. [14] MRAZOVAC, B., BJELICA, M. PAPP, I., TESLIC, N. Smart Au- However, these new technologies can help to be actually in dio/Video Playback Control Based on Presence Detection and User Lo- the lecture room, e.g. lifting a hand to ask a question. Holo- calization in Home Environment. In IEEE Eastern European Regional Conference on the Engineering of Computer Based Systems (ECBS- grams from Star Trek do not look so sci-fi anymore. Curious EERC). Budapest (Hungary), 2011, p. 44–53 about the future? We are too. Let us wait together! [15] MAIMONE, A., FUCHS, H. Encumbrance-free telepresence system with real-time 3D capture and display using commodity depth cameras. In IEEE/ACM International Symposium on Mixed and Augmented Re- ality (ISMAR). Basel (Switzerland), 2011, p. 137–146 Acknowledgements [16] PISHARADY, P., KUMAR, P., SAERBECK, M. Recent methods and databases in vision-based hand gesture recognition: A review. Com- Study described in the paper was supervised by Prof. puter Vision and Image Understanding, 2015, vol. 141, p. 152–165 L. Vokorokos, Technical University of Košice, Slovakia. [17] HENRY, P. et al. RGB-D mapping: Using Kinect-style depth cam- eras for dense 3D modeling of indoor environments. The International This work was supported by the Slovak Research and Journal of Robotics Research, 2012, vol. 31, no. 5, p. 647–663 Development Agency under the contract No. APVV-0008- [18] KHOSHELHAM, K., ELBERINK, S.O. Accuracy and Resolution of 10. Kinect Depth Data for Indoor Mapping Applications. Sensors, 2012, vol. 12, no. 2, p. 1437–1454 [19] Mallick, T., DAS, P.P., MAJUMDAR, A.K. Characterizations of Noise in Kinect Depth Images: A Review. IEEE Sensors Journal, 2014, References vol. 14, no. 6, p. 1731–1740 [20] HUANG, J.D. Kinerehab: A Kinect-based System for Physical Reha- [1] LUO, Q., YANG, G. Research and Simulation on Virtual Movement bilitation: A Pilot Study for Young Adults with Motor Disabilities. In Based on Kinect. Lecture Notes in Computer Science. Virtual, Aug- 13thth International ACM SIGACCESS Conference on Computers and mented and Mixed Reality: Designing and Developing Virtual and Aug- Accessibility (ASSETS). Dundee (Scotland), 2011, p. 319–320 mented Environments, Part I., 2014, vol. 8525, p. 85–92 [21] ADORA-MED. Home page. Cited: March 6, 2016. Available at: [2] ZHANG, Z. Microsoft Kinect Sensor and Its Effect. IEEE Multimedia, http://adora-med.com/ 2012, vol. 19, no. 2, p. 4–10 [22] SVENSEK, A. Dora targets the world market with their ’surgical as- [3] CLAYMAN, D. 2010: Project Natal is "Kinect". Pub- sistant’. Published: October 14, 2013. Cited: March 6, 2016. Available lished: June 13, 2010. Cited: March 5, 2016. Available at: at: http://www.rtvslo.si/news-in-english/dora- http://www.ign.com/articles/2010/06/13/e3- targets-the-world-market-with-their-surgical- 2010-project-natal-is-kinect assistant/320097 [4] SECTOR NET. Predstavujeme: Xbox360 Kinect (We Introduce: [23] ISIDOGAN, F., GOKCEHAN, K. A Real Time Virtual Dress- Xbox360 Kinect). Published: November 10, 2010. Cited: March 5, ing Room Application Using Kinect. In Computer Vision Course 2016. Available at: http://www.sector.sk/clanok/29161/ Project (CMPE537). Published: January 20, 2012. Cited: March predstavujeme-xbox360-kinect.htm 5, 2016. Available at: http://www.isikdogan.com/files/ [5] Microsoft News Center. Microsoft Releases Kinect for Windows isikdogan\_kinect.pdf SDK Beta for Academics and Enthusiasts. Published: June 16, 2011.