Learn Microsoft® Kinect API Rob Miles Copyright © 2012 by Rob Miles All rights reserved. No part of the contents of this book may be reproduced or transmitted in any form or by any means without the written permission of the publisher. ISBN: 978-0-735-66396-1 1 2 3 4 5 6 7 8 9 LSI 7 6 5 4 3 2 Printed and bound in the United States of America. Microsoft Press books are available through booksellers and distributors worldwide. If you need support related to this book, email Microsoft Press Book Support at [email protected]. Please tell us what you think of this book at http://www.microsoft.com/learning/booksurvey. Microsoft and the trademarks listed at http://www.microsoft.com/about/legal/en/us/IntellectualProperty/ Trademarks/EN-US.aspx are trademarks of the Microsoft group of companies. All other marks are property of their respective owners. The example companies, organizations, products, domain names, email addresses, logos, people, places, and events depicted herein are fictitious. No association with any real company, organization, product, domain name, email address, logo, person, place, or event is intended or should be inferred. This book expresses the author’s views and opinions. The information contained in this book is provided without any express, statutory, or implied warranties. Neither the authors, Microsoft Corporation, nor its resellers, or distributors will be held liable for any damages caused or alleged to be caused either directly or indirectly by this book. Acquisitions and Developmental Editor: Russell Jones Production Editor: Kristen Borg Editorial Production: Tiffany Rupp, S4Carlisle Publishing Services Technical Reviewer: Peter Robinson Copyeditor: Heath Lynn Silberfeld Indexer: WordCo Indexing Services, Inc. Cover Design: Jake Rae Cover Composition: Karen Montgomery Illustrator: S4Carlisle Publishing Services To Gus Contents at a Glance Introduction xiii Part I GettiNG Started CHAPTER 1 An Introduction to Kinect 3 CHAPTER 2 Getting Started with Kinect 13 CHAPTER 3 Writing Software for Kinect 25 Part II UsiNG THE KINect SENSOR CHAPTER 4 Your First Kinect Application—Video Snapshots 43 CHAPTER 5 Moving Pictures 67 CHAPTER 6 Fun with the Depth Sensor 81 CHAPTER 7 Fun with the Sound Sensor 103 Part III CreatiNG ADVANced User INterFaces CHAPTER 8 Body Tracking with Kinect 123 CHAPTER 9 Voice Control with Kinect 145 CHAPTER 10 Augmented Reality with Kinect 165 Part IV KINect IN THE ReaL WORLD CHAPTER 11 Real-World Control with Kinect 201 CHAPTER 12 Taking Kinect Further 229 Index 241 Contents Introduction ....................................................xiii Part I GettiNG Started Chapter 1 An Introduction to Kinect 3 The Kinect Sensor .................................................3 Getting Inside a Kinect Sensor .................................4 Recognizing People with Kinect .....................................9 Programming the Kinect ..........................................10 Kinect for Xbox and Kinect for Windows ............................10 Summary. .11 Chapter 2 Getting Started with Kinect 13 Kinect for Windows SDK Prerequisites ..............................13 Kinect Device ..............................................13 Visual Studio ...............................................14 DirectX Studio. .14 Installing the Kinect for Windows SDK ..............................14 Connecting the Kinect Sensor Bar ..................................17 Powering the Kinect Sensor. .17 Installing the Kinect Sensor USB Drivers .......................18 Testing the Kinect Sensor Bar. .18 The Kinect SDK Sample Browser. .18 What do you think of this book? We want to hear from you! Microsoft is interested in hearing your feedback so we can continually improve our books and learning resources for you. To participate in a brief online survey, please visit: microsoft.com/learning/booksurvey vii Troubleshooting Your Kinect Installation ............................21 Remove Old SDK Installations ................................21 Ensure That Visual Studio 2010 Is Installed but Not Running During Installation ..........................................22 Ensure That There Are No Windows Updates in Progress ........22 Ensure That the Kinect Is Powered Correctly ...................22 Remove Any Old USB Drivers. .22 Summary. .24 Chapter 3 Writing Software for Kinect 25 Making a Kinect Video Camera ....................................25 Creating a New Visual Studio Project for Kinect ................25 Getting the Kinect Sensor Working ...........................28 Displaying a Video Frame ....................................33 Adding Error Handling ......................................38 Summary. .39 Part II UsiNG THE KINect SENSOR Chapter 4 Your First Kinect Application—Video Snapshots 43 Image Storage in Computers ......................................43 Getting the Kinect Image Data onto the Screen ................44 Controlling the Color of the Pixels ............................47 Creating a Color Adjustment Program ..............................49 Improving the Speed by Writing Unsafe Code ..................50 Saving the Image to a File ...................................57 Improving Video Quality ....................................59 Improving Performance by Waiting for Each Kinect Frame ............61 Creating a Video Display Thread ..............................62 Updating the Image from a Different Thread ...................63 Stopping the Background Thread .............................65 Summary. .66 viii Contents Chapter 5 Moving Pictures 67 Detecting Movement in Video Images ..............................67 Storing a Video Image in Program Memory. .68 Detecting Changes in Video Images ..........................69 Sounding the Alarm .........................................72 A Complete Alarm Program ..................................74 Switching to Black and White ................................77 Summary. .80 Chapter 6 Fun with the Depth Sensor 81 Visualizing Kinect Depth Information ...............................81 The Kinect Depth Sensor ....................................81 Obtaining Depth Information from the Sensor .................82 Visualizing Depth Information ................................84 Using the Depth Information to Detect Intruders. .89 Using the Depth and Video Sensors at the Same Time ...........89 Drawing in the Air ................................................90 Detecting Objects. .92 Counting Depth Values ......................................93 Making You into the Controller ...............................96 Using the Kinect Sensor with an XNA Game. .97 Summary. .101 Chapter 7 Fun with the Sound Sensor 103 Capturing Sound Using Kinect ....................................103 Sound and Computers .....................................103 Receiving Sound Signals from Kinect .........................106 Playing Sound Using XNA ..................................108 Sound Signals and Latency ..................................111 Visualizing a Sound Signal in XNA .................................112 Storing Sound Data in a File and Replaying It .......................115 Creating a WAV File. .116 Playing a Recorded Sound ..................................118 Summary. .120 Contents ix Part III CreatiNG ADVANced User INterFaces Chapter 8 Body Tracking with Kinect 123 Kinect Body Tracking ............................................123 Kinect Skeleton Information. 124 A Head Tracking Program ........................................126 The Joints Collection and C# Dictionaries .....................128 Using Format Strings to Build a Message .....................130 Skeleton Information Quality. .131 Joint Tracking State ........................................132 Drawing a Skeleton ..............................................133 Drawing Lines in WPF ......................................133 Converting Joint Positions to Image Coordinates ..............134 Clearing the Canvas ........................................136 Drawing a Complete Skeleton ...............................136 Detecting Gestures ..............................................139 Calculating the Distance Between Two Points in Space .........139 Using a Gesture to Trigger an Action. 140 Biometric Recognition with Kinect ...........................141 Creating a “Kiss-Detecting” Program ..............................141 Finding Two Skeletons That Are Being Tracked ................141 Summary. .143 Chapter 9 Voice Control with Kinect 145 Using the Microsoft Speech Platform ..............................145 Testing Voice Recognition ..................................146 Creating a Program That Recognizes Color Names. 147 Adding the Speech Platform SDK Assemblies to a Project .......147 Creating a Speech Recognition Engine .......................147 Building the Commands. .151 Creating a Grammar .......................................151 Getting Audio into the Speech Recognizer ....................152 Responding to Recognized Words ...........................153 x Contents Creating a Voice-Controlled Painting Program ......................154 Speech Commands ........................................155 Drawing a Skeleton Cursor ..................................157 Drawing Using the Artist’s Hand. 158 Saving the Drawing Canvas to a File. .158 Tidying Up When the Program Ends .........................160 Improving the Drawing Program ............................162 Adding Speech Output to Programs ...............................162 Feedback Problems. .163 Summary. .164 Chapter 10 Augmented Reality with Kinect