Real-Time Range Imaging for Human-Machine Interfaces Frank Forster Lehrstuhl für Mensch-Maschine-Kommunikation Technische Universität München Real-Time Range Imaging for Human-Machine Interfaces Frank Forster Vollständiger Abdruck der von der Fakultät für Elektrotechnik und Informationstechnik der Technischen Universität München zur Erlangung des akademischen Grades eines Doktors der Naturwissenschaften genehmigten Dissertation. Vorsitzender: Univ.-Prof. Dr. techn. Josef A. Nossek Prüfer der Dissertation: 1. Univ.-Prof. Dr. rer. nat. Manfred K. Lang 2. Univ.-Prof. Dr. rer. nat. Bernd Radig 3. Univ.-Prof. Dr.-Ing. Gert Hauske Die Dissertation wurde am 8.11.2004 bei der Technischen Universität München eingereicht und durch die Fakultät für Elektrotechnik und Informationstechnik am 20.06.2005 angenommen. Lehrstuhl für Mensch-Maschine-Kommunikation der Technischen Universität München Vorwort Die vorliegende Arbeit ist das Ergebnis meiner Forschungstätigkeit als externer wissenschaftlicher Mitarbeiter am Lehrstuhl für Mensch-Maschine-Kommunikation der Technischen Universität München und als Mitarbeiter der Abteilung CT PS 9 der Siemens AG. Mein ganz besonderer Dank gilt meinem Doktorvater Professor Manfred Lang, der sich freundli- cherweise bereit erklärt hat, die wissenschaftliche Betreuung zu übernehmen. Er war jederzeit für wertvolle Diskussionen und fachlichen Rat verfügbar und schuf damit eine wichtige Grundvoraus- setzung für den Erfolg dieser Arbeit. Als sehr fruchtbar erwies sich, dass er mir im Rahmen des Themas viel Freiraum bei der Gestaltung der Dissertation ließ. Auch von seiner Unterstützung beim Besuch internationaler Fachtagungen, teilweise durch seine persönliche Anwesenheit, profi- tierte diese Arbeit. Desgleichen möchte ich Herrn Professor Radig danken, der diese Dissertation ebenfalls wissen- schaftlich betreute. Trotz seiner vielen Verpflichtungen fand er Zeit, mir beratend zur Seite zu ste- hen, und trug somit viel zum Gelingen dieser Arbeit bei. Ein herzliches Dankeschön geht an die Mitarbeiter des Fachzentrums CT PS 9 der Siemens AG, insbesondere an Herrn Rummel, Dr. Doemens und Dr. Laloni, die mich, soweit der stressige Be- rufsalltag es zuließ, bei meinem Promotionsvorhaben stets unterstützten. Weiter danke ich allen an dieser Arbeit beteiligten Diplomanden, Praktikanten und Werkstudenten für ihr Engagement. Abschließend danke ich allen Personen aus meinem persönlichen Umfeld, die mich bei dieser Ar- beit unterstützt haben, vor allem natürlich meiner Monika, die leider oft hinter der Promotionsar- beit zurückstehen musste. Vielen Dank für das Verständnis und die Unterstützung! München, im Juli 2004 Frank Forster Frank Forster – Real-Time Range Imaging for Human-Machine Interfaces V Lehrstuhl für Mensch-Maschine-Kommunikation der Technischen Universität München VI Frank Forster – Real-Time Range Imaging for Human-Machine Interfaces Lehrstuhl für Mensch-Maschine-Kommunikation der Technischen Universität München Abstract This thesis presents a new approach to automatically acquire accurate high-resolution range images in real-time. While this work focuses on scenes relevant for human-machine communication such as human faces or hands, the proposed technique can be used with arbitrary close-range scenes. Moreover, it is well suited for an integrated 2D-3D vision approach as it provides a color image of the scene along with the range data. The central part of the presented technique is a color coded light approach with a single static pro- jection pattern. Existing methods using this technique are limited to scenes with neutral or uniform reflectance. This work concludes from an abstract scene model that employing local color edge patterns for encoding is a way to overcome this limitation. It furthermore establishes the properties a coded light projection pattern should ideally have so that an algorithm is able to reliably demodu- late and decode it from an image. From these theoretical considerations it derives a corresponding new type of projection pattern that also permits a high lateral resolution of the range data. It proves that this pattern type permits detecting if the reflection of a local color edge pattern is disrupted in virtually all practically relevant cases. Next, it introduces an algorithm that exploits the properties of the projection pattern to robustly convert a color image of a scene illuminated by such a pattern into a range image. It finally describes a pseudo-random approach to generate the necessary com- plex color edge patterns. The proposed coded light approach works well with most scenes, but has certain intrinsic weak- nesses, e.g. at surface singularities. The thesis shows that stereo algorithms are typically well suited for obtaining range values for the parts of the scene where the coded light step fails; also that such algorithms are capable of operating in real-time in this case because these problematic regions tend to make up only a small percentage of the scene. A corresponding stereo algorithm that comple- ments the coded light step is presented, yielding a two-stage ranging technique suited for arbitrary scenes. It is a precondition for range image acquisition that both camera and projector are calibrated. This work introduces an approach to camera calibration that extends Tsai’s well-known monoview cali- bration method to one based on several views of a planar calibration target. It describes how the task of projector calibration can be solved with this approach. The experimental results given indi- cate the technique has certain advantages over comparable state-of-the-art calibration methods and permits the accurate calibration of a coded light, respectively stereo system on the basis of a simple planar target. The thesis further performs a range error analysis based on a parameterized model of a triangula- tion-system, including the complex case of a convergent geometric set-up. It develops exact as well as approximate formulas for the error in the measured coordinates as function of these parameters. A prototype system based on the presented ranging approach, integrated using low-cost off-the- shelf components, is evaluated. Experiments show that it is able to acquire range maps of resolu- tion 780 by 580 at up to 25 frames per second on a standard PC with an exemplary measurement accuracy of 0.2 mm standard deviation over a cubical working space of about 0.5 m side length; also that the method is robust against background illumination and works well with scenes that are strongly colored and textured. Finally, this thesis describes a face recognition system based on embedded hidden Markov models that works with color/gray level and range data; also a database of 2700 color and corresponding range images of a test population of 20 people acquired with a prototype system of the proposed approach. The evaluation of the face recognition system on the database demonstrates that using range data improves the performance of a standard face recognition technique significantly over its color/gray level only version. Frank Forster – Real-Time Range Imaging for Human-Machine Interfaces VII Lehrstuhl für Mensch-Maschine-Kommunikation der Technischen Universität München VIII Frank Forster – Real-Time Range Imaging for Human-Machine Interfaces Lehrstuhl für Mensch-Maschine-Kommunikation der Technischen Universität München Table of Contents 1 INTRODUCTION.....................................................................................................................1 1.1 MOTIVATION AND PURPOSE OF THIS WORK .......................................................................1 1.2 ORGANIZATION OF THIS WORK...........................................................................................2 2 FUNDAMENTALS OF RANGE IMAGE ACQUISITION..................................................3 2.1 PHYSICS OF RADIATION ......................................................................................................3 2.1.1 Basic Concepts and Quantities of Radiometry............................................................4 2.1.2 The Propagation of Radiation.....................................................................................6 2.2 GEOMETRIC CAMERA MODELS...........................................................................................9 2.2.1 The Pinhole Camera Model ........................................................................................9 2.2.2 The Lens Camera Model...........................................................................................10 2.3 THE CAMERA SENSOR MODEL..........................................................................................12 2.3.1 Formation and Description of Digital Images..........................................................12 2.3.2 The Concept of Color, Color Vision and Color Images............................................14 2.4 SUMMARY .........................................................................................................................16 3 STATE OF THE ART IN RANGE IMAGING ...................................................................17 3.1 BASIC RANGING TERMS....................................................................................................17 3.2 ASPECTS OF RANGE ACQUISITION SYSTEMS ....................................................................19 3.3 RANGE IMAGING METHODS ..............................................................................................19 3.3.1 Time-of-Flight Ranging.............................................................................................20