EUROGRAPHICS 2005 / M. Alexa and J. Marks Volume 24 (2005), Number 3 (Guest Editors) Garment Motion Capture Using Color-Coded Patterns Submission id: paper1243 Abstract In this paper we present an image-based algorithm for surface reconstruction of moving garments from multiple calibrated video cameras. Using a color-coded cloth texture allows reliable matching of circular features between different camera views. We use an a priori known triangle mesh as surface model. By identifying the mesh vertices with texture elements we obtain a consistent parameterization of the surface over time without further processing. Missing data points resulting from self-shadowing are plausibly interpolated by minimizing a thin-plate functional. The deforming geometry can be used for different graphics applications, e.g. for realistic retexturing. We show results for real garments demonstrating the accuracy of the recovered flexible shape. Categories and Subject Descriptors (according to ACM CCS): I.4.1 [Image Processing and Computer Vision]: Dig- itization and Image Capture I.3.7 [Computer Graphics]: Animation 1. Introduction imations in real-time. In order to achieve real time simu- Correct realistic simulation and visualization of textiles has lations many simplifications, heuristics, and precalculation been at the focus of many research fields such as math- have to be done which produce physically reasonable results ematics, physics, materials science, and computer graph- but the calculated movements still do not look naturally and ics. A wide area of applications ranging from virtual actors lack the correct physical behavior ([VMT01]). for the film industry to virtual prototyping of cloth design An alternative to physical simulation is motion capture. has made this research field especially attractive. Physically Correct physics is not an issue in this case. Rather than using based approaches aim at modeling the behavior of textiles complex models for the human body and the garment, the based on physical models of textile dynamics. The meth- motion is measured directly excluding numerical or physical ods used vary from linear elastic to nonlinear visco-elastic inaccuracies. To close the gap between model and experi- approaches. Much research is concentrating on finding the ment, an automatic measurement technique is needed. Much right model to capture the physical behavior and represent research has been devoted to skeletal and facial motion cap- ∗ it [CK02, EKS03, RBF03, MTCK 04]. Since the work of ture, but cloth capture still remains an open problem. Baraff and Witkin [BW98], implicit numerical time integra- This paper is organized as follows. In Section 2 we men- tion has proved to be a powerful numerical method to solve tion related work regarding the acquisition of non-rigid sur- the stiff ordinary differential equations that arise in cloth faces. Section 3 explains the process of garment produc- simulations. tion. We then move on to describe our method for shape re- The obtained results have to be rendered photo- construction in Section 4. Section 5 explains our rendering realistically (Figure 1) to obtain a good look and feel of method and Section 6 presents the obtained results. Finally, the simulated result. Here, research aims at modeling the we summarize our findings in Section 7 and conclude with visual behavior of cloth to enhance the realistic impression ∗ an outlook on potentially fruitful future work. of the animation ([SSK03, MMS 04]). Nowadays, current cloth simulation engines have become powerful enough to 2. Related work be used in quality demanding applications relevant to tex- tile industries. Moreover, animations produced by film in- The problem of capturing non-rigid motion has been ad- dustry include very realistically looking cloth which cannot dressed by a number of researchers. Carceroni and Kutu- be distinguished from real captured cloth. Despite these ef- lakos [CK01] obtain shape, reflectance and non-rigid motion forts and faster computers coming up every year, the known of a dynamic 3D scene by an algorithm called surfel sam- methods are still far from generating realistic detailed an- pling. Experimental results for complex real scenes (a wav- submitted to EUROGRAPHICS 2005. 2 Submission id: paper1243 / Garment Motion Capture Using Color-Coded Patterns repeated over the whole fabric. In this method the color code is only used for the parameterization of the surface. The work by Guskov et al. [GKB03] is closest to our work. They use color-coded quad markers for the acquisi- tion of non-rigid surfaces. Results for different surface types, including a T-shirt are presented. The used color code has also a limited size of codewords so that a tracking method based on Markov random fields is employed. The system achieves real-time performance. Tracking performance dete- riorates for fast motion and the quads have to be quite large which limits surface resolution. In contrast, our method uses a color code with more codewords and can cope with fast motion. It makes use of the a priori knowledge of surface connectivity and the color-coded pattern. In our work we present the first results for complex motion of real garments. 3. Preliminary work Our approach requires a costum-printed cloth pattern. We describe its production in the following. Additionally, a tri- Figure 1: Simulation and visualization of virtual cloth with measured physical data on a captured person (taken from angle mesh for the garment is constructed as input for the ∗ [MMS 04]). acquisition algorithm. 3.1. Color-coded patterns Color codes are well-known in the context of structured ing flag, skin, shiny objects) are shown. The reconstructed light reconstruction techniques [ZCS02]. In [PSGM03], a surfels are quite large which gives a coarse sampling of the good overview of projection patterns including color codes surface. A flow-based tracking method which does not re- is given. For two-dimensional cloth textures we need a pat- quire prior shape models is described in [TYAB01]. The tern which encodes both axes. On the one hand, the pat- method produces 3D reconstructions from single-view video tern should be large enough for manufacturing garments. by exploiting rank constraints on optical flow. Results for a On the other hand, each point in the pattern must be iden- shoe and a T-shirt tracking sequence are shown. tifiable and thus have a different codeword. We have cho- ∗ ∗ Bhat et al. [BTH 03] estimate the parameters for a cloth sen M-arrays [MOC 98], a color code which encodes each simulation by adjusting the simulation results to real world point in the pattern by its spatial neighborhood (Figure 2). × footage. This is an elegant way to avoid parameter tuning by In this code, each 3 3 neighborhood of a point is unique hand. Results for fabrics with different material properties and can be used for point identification. The number of pos- are shown. By reducing non-rigid motion to several mate- sible codewords depends on the number of colors c and is rial parameters, this method is suitable mainly for qualitative reproduction. Pritchard and Heidrich [PH03] use an image- based approach to cloth motion. They use a calibrated stereo camera pair and obtain the surface parameterization by using SIFT feature matching. Motion blur caused by fast motion reduces the accuracy of the matching. Lobay and Forsyth [LF04] show that shape-from-texture techniques can be applied to cloth reconstruction. The results are based on still images and a surface model with irradiance maps is reconstructed. Their shape from texture approach derives surface normals from the shape of the individual tex- ture elements which requires a regular texture pattern. The results look smooth but lack detail. Ebert et al. [AED03] use color-coded cloth textures for retexturing virtual clothing. Figure 2: The pseudo-random color pattern used for our Together with range scans of the garment a parameteriza- garments contains five colors: cyan, magenta, yellow, or- tion of the mesh is obtained. The authors use a color code ange and green. which has a limited size of codewords so that the pattern is submitted to EUROGRAPHICS 2005. Submission id: paper1243 / Garment Motion Capture Using Color-Coded Patterns 3 given by c9 (1.9 million for c = 5). By choosing five well the cloth patterns are specified, and the complete piece of distinguishable colors we are able to construct a pattern with cloth is constructed. In this step, we assert that the patterns a reasonable size for textile printing (76 × 251 points). For are triangulated in such a way that two corresponding seam pseudo-random code generation we adopt an incremental al- lines have the same number of vertices. More precisely, each ∗ gorithm described in [MOC 98]. It begins by seeding the seam is given by a list of pairs of vertices being the cor- top-left 3 × 3 window of the pattern matrix with a random responding vertices on two not necessarily distinct planar color assignment and fills up the matrix by incrementally patterns. Afterwards, the integrated cloth simulation based adding random colors. In each step, the window property on [EKS03] is used to achieve a smooth triangle mesh as is verified. In our case, the windows may be rotated in the input for the reconstruction algorithm. A more detailed de- camera images. In order to make point identification invari- scription of the mesh construction process can be found in ∗ ant to rotations, all windows are also verified against rotated [GMP 04, KFW04]. The correspondence between the mesh versions in 45 degree steps. This reduces the number of pos- vertices inside the borders and the colored pattern dots is sible codewords but still allows patterns of reasonable size. also established during mesh construction. The uv texture The output of the algorithm is a pattern matrix M with en- coordinates of a vertex correspond to its index in the pattern tries for the five colors.