Applying for a CTE Teaching Enhancement Award

Kinect for Education: Making the Senior Design More Relevant, More Cutting-Edge, and More Fun

Wenbing Zhao Department: Electrical and Computer Engineering Email address: [email protected]

I. Please describe the project for which you are seeking funding

Kinect for Xbox 360 (or Kinect for short), is a hands-free game controlling device that revolutionized the gaming industry. It is equipped with an RGB camera, a depth sensor, and a multi-array microphone. Unlike Wii and PlayStation 3, which require the players to carry a game controller physically, Kinect enables Xbox to offer a natural user interface to players so that they could use gestures and spoken commands to control the game console and play games (i.e., the player’s body is the controller). While it is great to play games with Kinect, what is more exciting perhaps is the Kinect’s potential to be used in areas far beyond its designated use, particularly in medicine and health care1. The recent release of a software development kit for Kinect by Microsoft paved the way for much widespread adoption of this revolutionary technology.

The objective of this project is to identify and develop a number of interesting Kinect applications that are both educational and practical, for the Senior Design Course (EEC490), which I will be teaching in spring 2012 (for Computer Engineering students). Traditionally, the applications that Computer Engineering students developed in the Senior Design Course are limited to the scope of computing systems. While such applications are often interesting, they are not meant to solve real-world problems. It is much more beneficial to students if they have the opportunity to discuss the needs and problems facing researchers and engineers in domains other than Computer Engineering, such as health science and civil engineering, and use their expertise to help solve their problems. This is exactly what I am planning to facilitate as part of this project.

I plan to target the Kinect use in the following two main areas: (1) outpatient physical/occupational therapy, and (2) drive safety study, because I am relatively familiar with these two areas through previous collaborated research with Dr. Ann Reinthal in Health Science Department and Dr. Jacqueline Jenkins in Civil Engineering, and also because I am sure my collaborators will be happy to assist in this project. Currently, devices that are equipped with accelerometers and gyroscopes are used to capture the motions of the subjects under tests in both areas. While interesting results have been obtained, the fidelity of the motion data captured is low because the actual motions have to be inferred, which is prone to errors. By using Kinect, which could provide raw sensor streams including video frames of the subject and sophisticated skeletal tracking, I anticipate that the accuracy of the research results could be drastically improved. Furthermore, a huge benefit of using Kinect is that the test subjects do not need to attach any sensor to their body, and particularly in the physical therapy study, as shown in Figure 1, all motions can be captured accurately by placing a second Kinect sensor on the

1 For example, Kinect has been used to measure a range of disorder symptoms in children (http://news.softpedia.com/news/Minnesota-University-Team-Adapts-Kinect-for-Medical-Use-189553.shtml); and Kinect has also been used to capture the hand motions of surgeons to direct the imaging, freeing their hands from having to touch computer keyboards (http://www.qj.net/qjnet/xbox-360/medical-practice-finds-use-for-kinect-hack.html). 1 side of the Kinect sensor that is being used by the subject to play games (i.e., all motion data that are supplied to the game itself are captured and provided to the Kinect application, which is impossible to accomplish in other gaming systems such as Wii and PlayStation3).

Figure 1. A configuration for capturing all motions for the Kinect application while a subject is playing an Xbox game.

In this project, I plan to engage students directly in ongoing research projects and encourage students to find novel uses of Kinect in ways beyond what have been described above. I am also going to utilize my contacts at Great Lakes NeuroTech Inc., a local company that develops medical devices. I will meet them regularly, and arrange direct meetings between them and the students, to learns their needs and problems that could potentially be solved by using Kinect in the beginning of spring 2012, and to calibrate the design and revise the implementation of the applications throughout the semester, and to demonstrate the students’ final products towards the end of the semester.

This project will result in the following deliverables:  D-1. A set of lecture notes (boot camp style) on Kinect application development  D-2. A software library that simplifies Kinect application development for physical therapy and drive safety study.  D-3. Senior design project formation, consultation, and evaluation.  D-4. Assessment of teaching and student learning.

Project Plan

Task 1: Preparation for lecture notes on Kinect application development (D-1). Prior to the start of the spring semester, I am going to prepare lecture nodes for about 4 lectures on Kinect application development. The lectures will be given to students within the first 2 weeks (the class will meet three times a week) so that the students can be prepared for their projects. The lecture notes will cover the following topics: getting familiar with Microsoft Visual Studio, how to set up Kinect programming environment, loading the Microsoft Kinect drivers, controlling Kinect sensors, Kinect for Windows architecture, the Kinect Natural User Interface (NUI), NUI image data streams interpretation, NUI skeletal tracking, NUI transformations. I do not plan to cover the audio application programming interface at this stage because audio will not be a focus in the proposed project.

2 Task 2: Design and implementation of a software library that simplifies Kinect application development for physical therapy and drive safety study (D-2) There are specific needs for physical therapy research and drive safety study. One project I was involved with in physical therapy research is to count the repeated movements made by a post stroke patient when he/she is playing games as part of the physical therapy. Instead of asking the students to come up with a solution on how to derive the count based on the NUI skeletal tracking, which might be very challenging for them, I plan to design and implement a solution myself, as part of a higher level library so that students could use the functionality directly in their project. For the drive safety study, on the other hand, the most interesting motion is head movement (e.g., is the driver looking at the road, or getting distracted by the passenger on the side or children at the rear row?). I will study the Kinect NUI and design a high level interface for head movement (instead of full body movement), as part of the library.

Task 3: Senior design project formation, consultation, and evaluation (D-3) The students in the class will be divided into four groups. Each group will be working on a distinct project. To help students form concrete ideas on what kind of applications they want to build, I will invite three guest speakers (Dr. Reinthal, and Dr. Jenkins, and the vice president of Great Lakes NeuroTech) to the class introducing their research/engineering projects and their needs during the first week. In the beginning of week 3, I will help each group come up with a project outline. Once all groups have determined their project outlines, I will schedule a presentation session for each group to present their ideas and plans to go forward. The session will be video-taped and will be compared with their final presentation. Subsequently, I will meet each group weekly to advise and to check on their progress. I will collect a written progress report from each group every other week. During the final week, I will schedule another presentation for each group to present and demonstrate their final application. The guest speakers will be invited back to observe and critique in the session and the session will also be video-taped for comparison. The final submission of the senior design project (from each group) will consist of a project report and the source code of the application.

Task 4: Assessment of teaching and student learning (D-4) The teaching and student learning will be evaluated both subjectively and objectively. Subjective evaluation will primarily be based on surveys (designed with the help of an educational specialist from the College of Education). We will also interview selected students for immediate feedback throughout the semester. Objective assessment will be based on the quality of the final applications students submitted.

II. What are the specific goals of this project? That is, what student outcomes do you expect to result?

The main goal of the project is to offer students an opportunity to learn Kinect programming, and gain valuable experiences with real-world, multi-disciplinary application development. More specifically, I expect that the proposed project will result in following student outcomes:  Students have a strong understanding of how to perform skeletal tracking for the test subject using the Natural User Interface (NUI) provided by the Kinect software development kit.  Students have good knowledge of how to use the depth sensor data to determine the subject’s position.  Students have good knowledge of how to control the Kinect sensors programmatically.  Students have the ability to use the Microsoft Visual Studio 2010 tool (as the development environment for Kinect applications), and select the appropriate APIs.

3  Students have the ability to design and implement useful multi-disciplinary Kinect applications in C++ or C#.

III. How will you determine whether or not the goals of the project have been accomplished?

We plan to enlist the assistance of an educational specialist from College of Education to help with the assessment. The teaching and student learning will be evaluated both subjectively and objectively. Subjective evaluation will primarily be based on surveys (designed with the help of an assessment expert from the College of Education). Objective assessment will be based on the quality of the final applications students submitted and whether or not they are instrumental to the research projects.

IV. Provide a detailed budget, with brief justification for each item. If you are seeking funds to pay for a student assistant, specify what the students would do to help achieve your proposed goals, why the amount of time requested is appropriate, and where you got the wage figure you are using in the budget. Remember to include fringe benefits in your calculations.

 Expense of 2 Xbox 360 with Kinect bundle, 2-3 Kinect games, 1 PC monitor with speaker (or a small HDTV) for game playing display, and 4 standalone Kinects. In the project, students will form four groups. Two groups will be working in the area of physical therapy, which requires 2 sets of game consoles with appropriate games suitable for physical therapy use. Only 1 monitor is included here because the monitor requested as part of the desktop PC below can be used for game playing display too. Two other groups will be working in the area of drive safety study. Each group will need one standalone Kinect for motion capturing and for Kinect application development. 2 Xbox 360 with Kinect bundle (unit price about $400) = $800 4 Kinect (unit price about $150) = $600 2-3 Kinect games = $100 1 PC Monitor with speakers or a small HDTV = $425 Total cost = $1,925  Expense of a research-grade desktop PC (Z210 CMT in CSU’s standard list) to be used for development of Kinect applications. The PC will be installed in my lab to be used primarily by students in the Senior Design class. 1 Desktop PC (Z210 CMT) = $1,129 1 Monitor (LP2475) = $460 Total cost = $1,589  Expense of a research-grade Windows laptop to be used by myself for the development of Kinect libraries and demo applications for students, for preparation of instructional materials, and for teaching the course in classroom 1 Windows Laptop (HP 8560p – Research) =$1,486  Total budget = $5,000

V. Map out a timeline for the project, with major benchmarks.

Time Work Assessment December, 2011  Complete Task 1: Preparation for  Solicit feedback from lecture notes on Kinect application interested students and development. Set up a course Web site, colleagues and post all lecture notes.

4 January, 2012  Complete Task 2: Design and  Solicit feedback from implementation of a software library interested students and that simplifies Kinect application colleagues development for physical therapy and  Feed back of the library drive safety study. The library will be from the research emailed to students as soon as it is community and industry ready.  The source code of the library will be made publicly available at a later date January-May,  Complete Task 3: Senior design  Student survey on the 2012 project formation, consultation, and quality of the applications evaluation May, 2012  Complete Task 4: Assessment of  Pretest and posttest teaching and student learning surveys  Quality of the final applications