A Virtual Dressing Room based on Depth Data DIPLOMARBEIT zur Erlangung des akademischen Grades Diplom-Ingenieur im Rahmen des Studiums Medieninformatik eingereicht von Philipp Presle Matrikelnummer 0526431 an der Fakultät für Informatik der Technischen Universität Wien Betreuung: Priv.-Doz. Mag. Dr. Hannes Kaufmann Mitwirkung: - Wien, 07.06.2012 (Unterschrift Verfasser) (Unterschrift Betreuung) Technische Universität Wien A-1040 Wien Karlsplatz 13 Tel. +43-1-58801-0 www.tuwien.ac.at A Virtual Dressing Room based on Depth Data MASTER’S THESIS submitted in partial fulfillment of the requirements for the degree of Diplom-Ingenieur in Media Informatics by Philipp Presle Registration Number 0526431 to the Faculty of Informatics at the Vienna University of Technology Advisor: Priv.-Doz. Mag. Dr. Hannes Kaufmann Assistance: - Vienna, 07.06.2012 (Signature of Author) (Signature of Advisor) Technische Universität Wien A-1040 Wien Karlsplatz 13 Tel. +43-1-58801-0 www.tuwien.ac.at Erklärung zur Verfassung der Arbeit Philipp Presle Gschwendt 2B/1, 3400 Klosterneuburg Hiermit erkläre ich, dass ich diese Arbeit selbständig verfasst habe, dass ich die verwende- ten Quellen und Hilfsmittel vollständig angegeben habe und dass ich die Stellen der Arbeit - einschließlich Tabellen, Karten und Abbildungen -, die anderen Werken oder dem Internet im Wortlaut oder dem Sinn nach entnommen sind, auf jeden Fall unter Angabe der Quelle als Ent- lehnung kenntlich gemacht habe. (Ort, Datum) (Unterschrift Verfasser) i Abstract Many of the currently existing Virtual Dressing Rooms are based on diverse approaches. Those are mostly virtual avatars or fiducial markers, and they are mainly enhancing the experience of online shops. The main drawbacks are the inaccurate capturing process and the missing possibility of a virtual mirror, which limits the presentation to a virtual avatar. By utilizing depth cameras like the Microsoft Kinect it becomes possible to track the movements of a body, extract body measurements and furthermore create a virtual mirror with the corresponding video stream. The video image can be merged with a piece of clothing frame by frame. The project is implemented in Unity, a programming environment for 3D applications. OpenNI and the NITE middleware are used for various fundamental functions and for the track- ing process in combination with the Microsoft Kinect. Taking a closer look at the results, several 3D cloth models were created and textured. The pieces of garment are based on a female 3D model. The clothes are adapted to the body of the user in front of the Kinect during runtime. In addition, cloth physics are taking care of a realistic representation. Furthermore trying on clothes in front of different backgrounds and surroundings (e.g. at night) shall be possible. Also a lot of value is placed on the interaction aspects of the Virtual Dressing Room. iii Kurzfassung Viele der momentan existierenden virtuellen Umkleidekabinen basieren auf unterschiedlichen Lösungsansätzen wie beispielsweise virtuellen Avataren oder aber auch fiducial Markern. Vor allem in Webshops kommen solche Lösungen zum Einsatz. Diese haben den Nachteil, dass sie öfters ungenau sind und keinerlei Möglichkeit eines virtuellen Spiegels bieten, also lediglich einen Avatar darstellen. Durch die Verwendung von Tiefenkameras wie der Microsoft Kinect wird es nun möglich, die Körperbewegungen und Körpermaße einer Person zu ermitteln und weiters mit Hilfe des zugehörigen Videostreams eine Art virtuellen Spiegel zu erstellen. Frame für Frame soll so das Videobild mit einem Kleidungsstück ’verschmolzen’ werden. Implementiert wird dieses Projekt in Unity, einer Entwicklungsumgebung für 3D Anwen- dungen. Um diverse grundlegende Funktionen und auch das Tracking mittels der Microsoft Ki- nect durchzuführen, wird OpenNI und die NITE Middleware verwendet. Für das Endresultat wurden 3D-Kleidungsmodels erstellt und texturiert. Die Kleidungs- stücke basieren auf einem weiblichen 3D-Modell. Diese werden während der Laufzeit auf den jeweiligen Benutzer vor der Kinect angepasst. Zusätzlich sollen Cloth Physics für eine realis- tischere Darstellung sorgen. Auch ein Ankleidevorgang vor verschiedenen Umgebungen und Hintergründen (beispielsweise bei Nacht) soll möglich sein. Weiters wurde großer Wert auf eine unkomplizierte und schnelle Interaktion gelegt. v Contents 1 Introduction 1 1.1 Overview & Motivation . 1 1.2 Goals & Challenges . 1 2 Related Work 3 2.1 Avatar generation . 3 2.2 Virtual cloth generation and simulation . 5 2.3 Real time tracking technologies . 7 2.3.1 Markers . 8 2.3.2 Optical tracking . 9 2.3.3 Depth cameras . 10 2.4 Example systems . 12 3 Fundamentals 17 3.1 Microsoft Kinect . 17 3.1.1 Skeleton tracking . 17 3.1.2 Software modules . 18 3.2 Unity . 21 3.3 Cinema 4D . 24 4 Design 25 4.1 Features . 25 4.2 System description . 26 4.3 Program flow . 26 4.4 Garment . 28 4.4.1 Cloth generation . 29 4.4.2 Garment adaption . 31 4.5 User interface . 32 4.5.1 Gesture recognition . 33 4.5.2 2D graphical user interface . 33 4.5.3 3D interaction elements . 33 4.6 Debug functions . 35 vii 5 Implementation 37 5.1 Class structure . 37 5.2 Unity scene configuration . 39 5.3 Controlling the Kinect . 40 5.4 Reading and outputting streams . 41 5.5 Tracking . 42 5.6 User interfaces . 42 5.6.1 3D interaction elements . 43 5.6.2 2D user interface . 43 5.7 Garment . 43 5.7.1 Garment adaption . 43 5.7.2 Body measurements . 44 5.7.3 Loading procedure . 45 5.7.4 Cloth parameters and XML structure . 46 6 Tests and Results 49 6.1 Results . 49 6.2 Feedback . 55 7 Conclusion 59 7.1 Future work . 59 Bibliography 61 viii CHAPTER 1 Introduction 1.1 Overview & Motivation Trying on clothes in stores today is one of the most time-consuming tasks. Usually long waiting periods have to be taken into account, for example when standing in front of full fitting rooms. Furthermore, additional time is lost when taking clothes on and off. Reducing this time and helping people to put on a large collection of garment in reduced time was a relevant motivation for this thesis. Using modern technology - hardware as well as software - the try-on experience can be drastically improved. Even in web shops people are very sceptic buying clothes because a try-on of clothes is not possible. The techniques discussed in this paper can enhance the shopping experience. It even offers customers a more precise representation than 2D images of the cloth they are willing to buy and therefore this may also reduce the amount of goods the buyers return. In this thesis I will introduce a Virtual Dressing Room, which offers a solution for the men- tioned aspects. The application is based on a mirror, represented by a display that outputs the image of the camera. If a person is standing in front of this virtual mirror, the person will be able to select desired clothes. The selected garment is then virtually superimposed with the image recorded by the camera. In general, this technique can be categorized under augmented reality (AR), where a real-time view of the reality is extended and furthermore overlaid with additional information. This paper mainly focuses on the applications in cloth stores although a home setup is possible as well. 1.2 Goals & Challenges The aim of the thesis is to create a Virtual Dressing Room that realistically reflects the appear- ance and the behavior of garment. It should further adapt to specific bodies of different persons depending on their body measurements. This will be one of the main challenges since the pieces 1 of cloth should correctly fit to as many persons as possible independent of their individual di- mensions. Technically speaking, the fitting room will be based on the Microsoft Kinect, an innovative technology which provides a new way of interaction between humans and the computer. From the depth image of the Kinect the skeleton is extracted and the position and orientation of the cloth are adapted in regard to the joint positions and body measurements. 3D models of the cloth will be overlaid with the color image from the RGB camera to obtain the function of a virtual mirror. To achieve a realistic simulation of the cloth, a physical simulation is performed on the garment. Providing a useful connection between the Microsoft Kinect and all the other parts of the application will be another ambition. 2 CHAPTER 2 Related Work The following four subsections are focusing on related work. First of all, the generation of avatars is discussed. These avatars are used as a three dimensional representation of the human body. The second subsection is dealing with the generation and the simulation of the garment. The simulation part depicts different methods that will enhance the try-on experience such as physical simulations of the garment or realistic cloth textures. The next section is dealing with real time tracking technologies. Most of the existing real time virtual fitting rooms are based upon marker based approaches, optical tracking methods or depth cameras like the Microsoft Kinect. The last section is describing example systems that are comprehending the presented techniques. 2.1 Avatar generation This section will deal with the generation of avatars that are representing the human body in many virtual fitting room applications. The measurements can be entered manually, which is usually done when using web based applications. Beyond that they can also be extracted by performing various measurement techniques, for example laser scanning or using a multi-camera capturing setup. An example of avatar generation is presented in [6, 7]. The system is based on a web appli- cation that adjusts the mannequin according to the measurements of the human body, resizes the cloth and in addition allows a realistic simulation due to physics. In order to generate an avatar, the eight length measurements have to be entered by the user.